X-LoRA

LoRA 专家混合模型 (X-LoRA) 是一种 PEFT 方法，支持基于高粒度（token、层、序列）缩放矩阵的稀疏或密集 LoRA 专家混合。这利用冻结的 LoRA 适配器和冻结的基础模型，大幅减少了需要微调的参数数量。

X-LoRA 的一个独特之处在于其通用性：它可以应用于任何带有 LoRA 适配器的 transformers 基础模型。这意味着，尽管采用了专家混合策略，但无需对模型代码进行任何更改。

下图展示了针对每个 token 的不同提示，缩放如何变化。这突显了随着生成过程的进行以及序列创建新上下文，不同适配器的激活情况。

Token-by-token scalings

论文摘要如下：

我们报告了一种专家混合策略，旨在使用基于低秩自适应 (LoRA) 的深度分层 token 级别方法创建微调的大型语言模型。从一组预训练的 LoRA 适配器开始，我们的门控策略使用隐藏状态来动态混合适配层，从而使生成的 X-LoRA 模型能够利用不同的能力，并创建前所未有的深度分层组合来解决任务。该设计灵感来源于普遍性和多样性的生物学原理，其中神经网络构建块在不同的分层表现形式中被重用。因此，X-LoRA 模型可以轻松地为任何现有的大型语言模型 (LLM) 实施，而无需修改底层结构。我们开发了一种定制的 X-LoRA 模型，该模型提供科学能力，包括正向/逆向分析任务和增强的推理能力，专注于生物材料分析、蛋白质力学和设计。这项工作的影响包括访问易于扩展和适应的模型，这些模型具有强大的领域知识和跨知识领域整合的能力。我们进行了一系列以物理为重点的案例研究，专家领域包括生物学、数学、推理、生物启发材料、力学和材料、化学、蛋白质生物物理学、力学和基于量子力学的分子特性。该模型不仅能够对蛋白质的纳米力学特性或量子力学分子特性进行定量预测，而且还能对结果进行推理，并正确预测可能解释不同分子行为的机制。.

请引用 X-LoRA，如下：

@article{10.1063/5.0203126,
    author = {Buehler, Eric L. and Buehler, Markus J.},
    title = "{X-LoRA: Mixture of low-rank adapter experts, a flexible framework for large language models with applications in protein mechanics and molecular design}",
    journal = {APL Machine Learning},
    volume = {2},
    number = {2},
    pages = {026119},
    year = {2024},
    month = {05},
    abstract = "{We report a mixture of expert strategy to create fine-tuned large language models using a deep layer-wise token-level approach based on low-rank adaptation (LoRA). Starting with a set of pre-trained LoRA adapters, our gating strategy uses the hidden states to dynamically mix adapted layers, allowing the resulting X-LoRA model to draw upon different capabilities and create never-before-used deep layer-wise combinations to solve tasks. The design is inspired by the biological principles of universality and diversity, where neural network building blocks are reused in different hierarchical manifestations. Hence, the X-LoRA model can be easily implemented for any existing large language model without a need for modifications of the underlying structure. We develop a tailored X-LoRA model that offers scientific capabilities, including forward/inverse analysis tasks and enhanced reasoning capability, focused on biomaterial analysis, protein mechanics, and design. The impact of this work includes access to readily expandable and adaptable models with strong domain knowledge and the capability to integrate across areas of knowledge. Featuring experts in biology, mathematics, reasoning, bio-inspired materials, mechanics and materials, chemistry, protein biophysics, mechanics, and quantum-mechanics based molecular properties, we conduct a series of physics-focused case studies. We examine knowledge recall, protein mechanics forward/inverse tasks, protein design, adversarial agentic modeling including ontological knowledge graph construction, and molecular design. The model is capable not only of making quantitative predictions of nanomechanical properties of proteins or quantum mechanical molecular properties but also reasoning over the results and correctly predicting likely mechanisms that explain distinct molecular behaviors.}",
    issn = {2770-9019},
    doi = {10.1063/5.0203126},
    url = {https://doi.org/10.1063/5.0203126},
    eprint = {https://pubs.aip.org/aip/aml/article-pdf/doi/10.1063/5.0203126/19964043/026119\_1\_5.0203126.pdf},
}

XLoraConfig

class peft.XLoraConfig

< 源代码 >

( task_type: typing.Union[str, peft.utils.peft_types.TaskType, NoneType] = None peft_type: typing.Union[str, peft.utils.peft_types.PeftType, NoneType] = None auto_mapping: typing.Optional[dict] = None base_model_name_or_path: typing.Optional[str] = None revision: typing.Optional[str] = None inference_mode: bool = False hidden_size: int = None adapters: dict[str, str] = None enable_softmax: bool = True enable_softmax_topk: bool = False layerwise_scalings: bool = False xlora_depth: int = 1 xlora_size: int = 2048 xlora_dropout_p: float = 0.2 use_trainable_adapters: bool = False softmax_temperature: float = 1.0 top_k_lora: Optional[int] = None scaling_pass_value: float = 0.0 global_scaling_weight: float = 1.0 )

参数

hidden_size (int) — 基础模型的隐藏层大小。
adapters (dict) — 适配器名称到 LoRA 适配器 ID 的映射，按照 PeftModel.load_adapter。它们将被自动加载，用作 LoRA 专家。当使用 from_pretrained 时，将新的适配器字典作为关键字参数传递。
enable_softmax (bool, 可选, 默认为 True) — 启用 X-LoRA 分类器的 softmax 应用。
enable_softmax_topk (bool, 可选, 默认为 False) — 为 top-k LoRA 适配器启用 softmax 应用。与 enable_softmax 互斥，且仅当设置了 top_k_lora 时才可设置。
softmax_temperature (float, 可选, 默认为 1.0) — Softmax 温度，值越低预测结果越清晰
layerwise_scalings (bool, 可选, 默认为 False) — 如果为 True，则为每个 LoRA 适配器（每层）生成缩放。如果为 False，则缩放将被广播，每层都相同。
top_k_lora (int, 可选, 默认为 None) — 稀疏地选择 top_k LoRA 专家，而不是默认的密集方法。
xlora_depth (int, 可选, 默认为 1) — X-LoRA 分类器的深度。
xlora_size (int, 可选, 默认为 2048) — X-LoRA 分类器的隐藏层大小，如果 xlora_depth=1，则无关紧要。
xlora_dropout_p (float, 可选, 默认为 0.2) — X-LoRA 分类器的 Dropout 概率，如果 xlora_depth=1，则无关紧要。
use_trainable_adapters (bool, 可选, 默认为 False) — 使适配器可训练。
scaling_pass_value (float, 可选, 默认为 0) — 缩放传递值。
global_scaling_weight (float, 可选, 默认为 1) — 用于乘以每个 LoRA 适配器输出的权重。

这是用于存储 XLoraModel 配置的配置类。当重新加载配置时，adapters 字段的路径将被忽略，而优先使用已保存的适配器。因此，在加载过程中只有键是重要的。

XLoraModel

类 peft.XLoraModel

< source >

( model: nn.Module config: Union[dict[str, XLoraConfig], XLoraConfig] adapter_name: str torch_device: Optional[str] = None ephemeral_gpu_offload: bool = False autocast_adapter_dtype: bool = True **kwargs ) → torch.nn.Module

参数

model (torch.nn.Module) — 要适配的模型。
config (XLoraConfig) — Lora 模型的配置。
adapter_name (str) — 适配器的名称，不影响 LoRA 适配器名称。

torch.nn.Module

X-LoRA 模型。

从预训练的 transformers 模型创建 X-LoRA (LoRA 专家混合) 模型。目前，此 X-LoRA 实现仅适用于具有 transformer 架构的模型。

该方法在 https://arxiv.org/abs/2402.07148 中有详细描述。

示例

>>> from transformers import AutoModelForCausalLM, AutoConfig, BitsAndBytesConfig
>>> from peft import LoraConfig, PeftModel, get_peft_model, prepare_model_for_kbit_training

>>> model_config = AutoConfig.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")
>>> config = XLoraConfig(
...     task_type="CAUSAL_LM",
...     hidden_size=model_config.hidden_size,
...     xlora_depth=4,
...     adapters={
...         "adapter_1": "./path/to/the/checkpoint/",
...         "adapter_2": "./path/to/the/checkpoint/",
...         "adapter_n": "./path/to/the/checkpoint/",
...     },
... )
>>> int8_config = BitsAndBytesConfig(load_in_8bit=True)
>>> model = AutoModelForCausalLM.from_pretrained(
...     "mistralai/Mistral-7B-Instruct-v0.1",
...     trust_remote_code=True,
...     attn_implementation="flash_attention_2",
...     device_map="cuda:0",
...     torch_dtype=torch.bfloat16,
...     quantization_config=int8_config,
... )
>>> model = prepare_model_for_kbit_training(4)
>>> xlora_model = get_peft_model(model, config)

< > 在 GitHub 上更新