X-LoRA

混合 LoRA 专家（X-LoRA）是一种 PEFT 方法，它基于高粒度（词元、层、序列）的缩放矩阵，实现了 LoRA 专家的稀疏或密集混合。该方法利用冻结的 LoRA 适配器和冻结的基础模型，从而大大减少了需要微调的参数数量。

X-LoRA 的一个独特之处在于其通用性：它可以应用于任何带有 LoRA 适配器的 transformers 基础模型。这意味着，尽管采用了混合专家策略，但无需对模型代码进行任何更改。

下图展示了对于不同的提示，每个词元的缩放值如何变化。这突显了随着生成过程的推进和序列创建新上下文时，不同适配器被激活的情况。

Token-by-token scalings

论文摘要如下：

我们提出了一种混合专家策略，使用基于低秩适应（LoRA）的深度逐层词元级方法来创建微调后的大型语言模型。从一组预训练的 LoRA 适配器开始，我们的门控策略利用隐藏状态动态混合已适应的层，使得最终的 X-LoRA 模型能够利用不同的能力，并创建前所未有的深度逐层组合来解决任务。其设计灵感来源于普遍性和多样性的生物学原理，即神经网络的构建模块在不同的层级表现形式中被重复使用。因此，X-LoRA 模型可以轻松地应用于任何现有的大型语言模型（LLM），而无需修改其底层结构。我们开发了一个定制的 X-LoRA 模型，该模型提供了科学能力，包括正向/逆向分析任务和增强的推理能力，专注于生物材料分析、蛋白质力学和设计。这项工作的影响包括可以获得易于扩展和适应、具有强大领域知识并能整合跨知识领域知识的模型。我们整合了生物学、数学、推理、仿生材料、力学与材料、化学、蛋白质生物物理学、力学以及基于量子力学的分子特性等领域的专家，进行了一系列以物理学为重点的案例研究。我们检验了知识召回、蛋白质力学正向/逆向任务、蛋白质设计、包括本体知识图谱构建在内的对抗性智能体建模以及分子设计。该模型不仅能够对蛋白质的纳米力学性能或量子力学分子特性进行定量预测，还能对结果进行推理，并正确预测解释不同分子行为的可能机制。.

引用 X-LoRA 时，请使用

@article{10.1063/5.0203126,
    author = {Buehler, Eric L. and Buehler, Markus J.},
    title = "{X-LoRA: Mixture of low-rank adapter experts, a flexible framework for large language models with applications in protein mechanics and molecular design}",
    journal = {APL Machine Learning},
    volume = {2},
    number = {2},
    pages = {026119},
    year = {2024},
    month = {05},
    abstract = "{We report a mixture of expert strategy to create fine-tuned large language models using a deep layer-wise token-level approach based on low-rank adaptation (LoRA). Starting with a set of pre-trained LoRA adapters, our gating strategy uses the hidden states to dynamically mix adapted layers, allowing the resulting X-LoRA model to draw upon different capabilities and create never-before-used deep layer-wise combinations to solve tasks. The design is inspired by the biological principles of universality and diversity, where neural network building blocks are reused in different hierarchical manifestations. Hence, the X-LoRA model can be easily implemented for any existing large language model without a need for modifications of the underlying structure. We develop a tailored X-LoRA model that offers scientific capabilities, including forward/inverse analysis tasks and enhanced reasoning capability, focused on biomaterial analysis, protein mechanics, and design. The impact of this work includes access to readily expandable and adaptable models with strong domain knowledge and the capability to integrate across areas of knowledge. Featuring experts in biology, mathematics, reasoning, bio-inspired materials, mechanics and materials, chemistry, protein biophysics, mechanics, and quantum-mechanics based molecular properties, we conduct a series of physics-focused case studies. We examine knowledge recall, protein mechanics forward/inverse tasks, protein design, adversarial agentic modeling including ontological knowledge graph construction, and molecular design. The model is capable not only of making quantitative predictions of nanomechanical properties of proteins or quantum mechanical molecular properties but also reasoning over the results and correctly predicting likely mechanisms that explain distinct molecular behaviors.}",
    issn = {2770-9019},
    doi = {10.1063/5.0203126},
    url = {https://doi.org/10.1063/5.0203126},
    eprint = {https://pubs.aip.org/aip/aml/article-pdf/doi/10.1063/5.0203126/19964043/026119\_1\_5.0203126.pdf},
}

XLoraConfig

class peft.XLoraConfig

< 源码 >

( task_type: typing.Union[str, peft.utils.peft_types.TaskType, NoneType] = None peft_type: typing.Union[str, peft.utils.peft_types.PeftType, NoneType] = None auto_mapping: typing.Optional[dict] = None base_model_name_or_path: typing.Optional[str] = None revision: typing.Optional[str] = None inference_mode: bool = False hidden_size: int = None adapters: dict[str, str] = None enable_softmax: bool = True enable_softmax_topk: bool = False layerwise_scalings: bool = False xlora_depth: int = 1 xlora_size: int = 2048 xlora_dropout_p: float = 0.2 use_trainable_adapters: bool = False softmax_temperature: float = 1.0 top_k_lora: Optional[int] = None scaling_pass_value: float = 0.0 global_scaling_weight: float = 1.0 )

参数

hidden_size (int) — 基础模型的隐藏层大小。
adapters (dict) — 适配器名称到 LoRA 适配器 ID 的映射，遵循 PeftModel.load_adapter 的规范。*它们将被自动加载*，用作 LoRA 专家。当使用 from_pretrained 时，将新的 adapters 字典作为关键字参数传入。
enable_softmax (bool, *可选*, 默认为 True) — 为 X-LoRA 分类器启用 softmax 应用。
enable_softmax_topk (bool, *可选*, 默认为 False) — 为 top-k LoRA 适配器启用 softmax 应用。与 `enable_softmax` 互斥，并且只有在设置了 `top_k_lora` 时才能设置。
softmax_temperature (float, *可选*, 默认为 1.0) — Softmax 温度，值越低，预测结果越尖锐。
layerwise_scalings (bool, *可选*, 默认为 False) — 如果为 True，则为每个 LoRA 适配器（每一层）生成缩放值。如果为 False，则缩放值将广播到每一层，保持相同。
top_k_lora (int, *可选*, 默认为 None) — 稀疏地选择 top_k 个 LoRA 专家，而不是默认的密集方法。
xlora_depth (int, *可选*, 默认为 1) — X-LoRA 分类器的深度。
xlora_size (int, *可选*, 默认为 2048) — X-LoRA 分类器的隐藏层大小，如果 `xlora_depth=1` 则此参数无关紧要。
xlora_dropout_p (float, *可选*, 默认为 0.2) — X-LoRA 分类器的 dropout 概率，如果 `xlora_depth=1` 则此参数无关紧要。
use_trainable_adapters (bool, *可选*, 默认为 False) — 使适配器可训练。
scaling_pass_value (float, *可选*, 默认为 0) — 缩放传递值。
global_scaling_weight (float, *可选*, 默认为 1) — 用于乘以每个 LoRA 适配器输出的权重。

这是用于存储 XLoraModel 配置的配置类。当重新加载配置时，会忽略 `adapters` 字段的路径，而使用已保存的适配器。因此，在加载过程中只有键是重要的。

XLoraModel

class peft.XLoraModel

< 源码 >

( model: nn.Module config: Union[dict[str, XLoraConfig], XLoraConfig] adapter_name: str torch_device: Optional[str] = None ephemeral_gpu_offload: bool = False autocast_adapter_dtype: bool = True **kwargs ) → torch.nn.Module

参数

model (torch.nn.Module) — 需要被适配的模型。
config (XLoraConfig) — Lora 模型的配置。
adapter_name (str) — 适配器的名称，不影响 LoRA 适配器名称。

torch.nn.Module

X-LoRA 模型。

从一个预训练的 transformers 模型创建一个 X-LoRA（混合 LoRA 专家）模型。目前，这个 X-LoRA 实现只适用于具有 transformer 架构的模型。

该方法的详细描述见 https://huggingface.co/papers/2402.07148。

示例

>>> from transformers import AutoModelForCausalLM, AutoConfig, BitsAndBytesConfig
>>> from peft import LoraConfig, PeftModel, get_peft_model, prepare_model_for_kbit_training

>>> model_config = AutoConfig.from_pretrained("mistralai/Mistral-7B-Instruct-v0.1")
>>> config = XLoraConfig(
...     task_type="CAUSAL_LM",
...     hidden_size=model_config.hidden_size,
...     xlora_depth=4,
...     adapters={
...         "adapter_1": "./path/to/the/checkpoint/",
...         "adapter_2": "./path/to/the/checkpoint/",
...         "adapter_n": "./path/to/the/checkpoint/",
...     },
... )
>>> int8_config = BitsAndBytesConfig(load_in_8bit=True)
>>> model = AutoModelForCausalLM.from_pretrained(
...     "mistralai/Mistral-7B-Instruct-v0.1",
...     trust_remote_code=True,
...     attn_implementation="flash_attention_2",
...     device_map="cuda:0",
...     torch_dtype=torch.bfloat16,
...     quantization_config=int8_config,
... )
>>> model = prepare_model_for_kbit_training(4)
>>> xlora_model = get_peft_model(model, config)

< > 在 GitHub 上更新