导出您的模型

要导出托管在 Hub 上的模型，您可以使用我们的 space。转换后，仓库将被推送到您的命名空间下，此仓库可以是公开的或私有的。

使用 CLI

要使用 CLI 将您的模型导出为 OpenVINO IR 格式

optimum-cli export openvino --model meta-llama/Meta-Llama-3-8B ov_model/

要导出私有模型或需要访问权限的模型，您可以运行 huggingface-cli login 以永久登录，或将环境变量 HF_TOKEN 设置为具有模型访问权限的令牌。有关更多信息，请参阅身份验证文档。

模型参数可以是托管在 Hub 上的模型的模型 ID，也可以是本地托管模型的路径。对于本地模型，您需要从支持的任务列表中指定模型应加载以进行导出的任务。

optimum-cli export openvino --model local_llama --task text-generation-with-past ov_model/

查看帮助以获取更多选项

usage: optimum-cli export openvino [-h] -m MODEL [--task TASK] [--framework {pt,tf}] [--trust-remote-code]
[--weight-format {fp32,fp16,int8,int4,mxfp4,nf4}]
[--quant-mode {int8,f8e4m3,f8e5m2,nf4_f8e4m3,nf4_f8e5m2,int4_f8e4m3,int4_f8e5m2}]
[--library {transformers,diffusers,timm,sentence_transformers,open_clip}]
[--cache_dir CACHE_DIR] [--pad-token-id PAD_TOKEN_ID] [--ratio RATIO] [--sym]
[--group-size GROUP_SIZE] [--backup-precision {none,int8_sym,int8_asym}]
[--dataset DATASET] [--all-layers] [--awq] [--scale-estimation] [--gptq]
[--lora-correction] [--sensitivity-metric SENSITIVITY_METRIC]
[--num-samples NUM_SAMPLES] [--disable-stateful] [--disable-convert-tokenizer]
[--smooth-quant-alpha SMOOTH_QUANT_ALPHA]
output

optional arguments:
-h, --help show this help message and exit

Required arguments:
-m MODEL, --model MODEL
Model ID on huggingface.co or path on disk to load model from.
output Path indicating the directory where to store the generated OV model.

Optional arguments:
--task TASK The task to export the model for. If not specified, the task will be auto-inferred based on
the model. Available tasks depend on the model, but are among: ['image-to-image',
'image-segmentation', 'inpainting', 'sentence-similarity', 'text-to-audio', 'image-to-text',
'automatic-speech-recognition', 'token-classification', 'text-to-image', 'audio-classification',
'feature-extraction', 'semantic-segmentation', 'masked-im', 'audio-xvector',
'audio-frame-classification', 'text2text-generation', 'multiple-choice', 'depth-estimation',
'image-classification', 'fill-mask', 'zero-shot-object-detection', 'object-detection',
'question-answering', 'zero-shot-image-classification', 'mask-generation', 'text-generation',
'text-classification']. For decoder models, use 'xxx-with-past' to export the model using past
key values in the decoder.
--framework {pt,tf} The framework to use for the export. If not provided, will attempt to use the local
checkpoint's original framework or what is available in the environment.
--trust-remote-code Allows to use custom code for the modeling hosted in the model repository. This option should
only be set for repositories you trust and in which you have read the code, as it will execute
on your local machine arbitrary code present in the model repository.
--weight-format {fp32,fp16,int8,int4,mxfp4,nf4}
The weight format of the exported model.
--quant-mode {int8,f8e4m3,f8e5m2,nf4_f8e4m3,nf4_f8e5m2,int4_f8e4m3,int4_f8e5m2}
Quantization precision mode. This is used for applying full model quantization including
activations.
--library {transformers,diffusers,timm,sentence_transformers,open_clip}
The library used to load the model before export. If not provided, will attempt to infer the
local checkpoint's library
--cache_dir CACHE_DIR
The path to a directory in which the downloaded model should be cached if the standard cache
should not be used.
--pad-token-id PAD_TOKEN_ID
This is needed by some models, for some tasks. If not provided, will attempt to use the
tokenizer to guess it.
--ratio RATIO A parameter used when applying 4-bit quantization to control the ratio between 4-bit and 8-bit
quantization. If set to 0.8, 80% of the layers will be quantized to int4 while 20% will be
quantized to int8. This helps to achieve better accuracy at the sacrifice of the model size
and inference latency. Default value is 1.0. Note: If dataset is provided, and the ratio is
less than 1.0, then data-aware mixed precision assignment will be applied.
--sym Whether to apply symmetric quantization
--group-size GROUP_SIZE
The group size to use for quantization. Recommended value is 128 and -1 uses per-column
quantization.
--backup-precision {none,int8_sym,int8_asym}
Defines a backup precision for mixed-precision weight compression. Only valid for 4-bit weight
formats. If not provided, backup precision is int8_asym. 'none' stands for original floating-
point precision of the model weights, in this case weights are retained in their original
precision without any quantization. 'int8_sym' stands for 8-bit integer symmetric quantization
without zero point. 'int8_asym' stands for 8-bit integer asymmetric quantization with zero
points per each quantization group.
--dataset DATASET The dataset used for data-aware compression or quantization with NNCF. For language models you
can use the one from the list ['auto','wikitext2','c4','c4-new']. With 'auto' the dataset will
be collected from model's generations. For diffusion models it should be on of
['conceptual_captions','laion/220k-GPT4Vision-captions-from-LIVIS','laion/filtered-wit']. For
visual language models the dataset must be set to 'contextual'. Note: if none of the data-aware
compression algorithms are selected and ratio parameter is omitted or equals 1.0, the dataset
argument will not have an effect on the resulting model.
--all-layers Whether embeddings and last MatMul layers should be compressed to INT4. If not provided an
weight compression is applied, they are compressed to INT8.
--awq Whether to apply AWQ algorithm. AWQ improves generation quality of INT4-compressed LLMs, but
requires additional time for tuning weights on a calibration dataset. To run AWQ, please also
provide a dataset argument. Note: it is possible that there will be no matching patterns in
the model to apply AWQ, in such case it will be skipped.
--scale-estimation Indicates whether to apply a scale estimation algorithm that minimizes the L2 error between
the original and compressed layers. Providing a dataset is required to run scale estimation.
Please note, that applying scale estimation takes additional memory and time.
--gptq Indicates whether to apply GPTQ algorithm that optimizes compressed weights in a layer-wise
fashion to minimize the difference between activations of a compressed and original layer.
Please note, that applying GPTQ takes additional memory and time.
--lora-correction Indicates whether to apply LoRA Correction algorithm. When enabled, this algorithm introduces
low-rank adaptation layers in the model that can recover accuracy after weight compression at
some cost of inference latency. Please note, that applying LoRA Correction algorithm takes
additional memory and time.
--sensitivity-metric SENSITIVITY_METRIC
The sensitivity metric for assigning quantization precision to layers. It can be one of the
following: ['weight_quantization_error', 'hessian_input_activation',
'mean_activation_variance', 'max_activation_variance', 'mean_activation_magnitude'].
--num-samples NUM_SAMPLES
The maximum number of samples to take from the dataset for quantization.
--disable-stateful Disable stateful converted models, stateless models will be generated instead. Stateful models
are produced by default when this key is not used. In stateful models all kv-cache inputs and
outputs are hidden in the model and are not exposed as model inputs and outputs. If --disable-
stateful option is used, it may result in sub-optimal inference performance. Use it when you
intentionally want to use a stateless model, for example, to be compatible with existing
OpenVINO native inference code that expects KV-cache inputs and outputs in the model.
--disable-convert-tokenizer
Do not add converted tokenizer and detokenizer OpenVINO models.
--smooth-quant-alpha SMOOTH_QUANT_ALPHA
SmoothQuant alpha parameter that improves the distribution of activations before MatMul layers
and reduces quantization error. Valid only when activations quantization is enabled.

您还可以在导出模型时，通过将 --weight-format 分别设置为 fp16、int8 或 int4，在线性层、卷积层和嵌入层上应用 fp16、8 位或 4 位仅权重量化。

使用 INT8 权重压缩导出

optimum-cli export openvino --model meta-llama/Meta-Llama-3-8B --weight-format int8 ov_model/

使用 INT4 权重压缩导出

optimum-cli export openvino --model meta-llama/Meta-Llama-3-8B --weight-format int4 ov_model/

使用 INT4 权重压缩以及数据感知 AWQ 和 Scale Estimation 算法导出

optimum-cli export openvino --model meta-llama/Meta-Llama-3-8B \
    --weight-format int4 --awq --scale-estimation --dataset wikitext2 ov_model/

有关量化参数的更多信息，请查看文档

默认情况下，大于 10 亿参数的模型以 8 位权重导出为 OpenVINO 格式。您可以使用 --weight-format fp32 禁用它。

除了仅权重量化之外，您还可以通过将 --quant-mode 设置为首选精度来应用包括激活的全模型量化。这将把线性层、卷积层和一些其他层的权重和激活都量化到选定的模式。请参阅以下示例。

optimum-cli export openvino -m openai/whisper-large-v3-turbo --quant-mode int8 --dataset librispeech --num-samples 32 --smooth-quant-alpha 0.9 ./whisper-large-v3-turbo

解码器模型

对于带有解码器的模型，我们默认启用过去键和值的重用。这允许避免在每个生成步骤中重新计算相同的中间激活。要导出不带缓存的模型，您需要在指定任务时删除 -with-past 后缀。

使用 K-V 缓存	不使用 K-V 缓存
`text-generation-with-past`	`text-generation`
`text2text-generation-with-past`	`text2text-generation`
`automatic-speech-recognition-with-past`	`automatic-speech-recognition`

扩散模型

当 Stable Diffusion 模型导出为 OpenVINO 格式时，它们会被分解为不同的组件，这些组件稍后在推理期间组合

文本编码器
U-Net
VAE 编码器
VAE 解码器

要使用 CLI 将您的 Stable Diffusion XL 模型导出为 OpenVINO IR 格式，您可以执行以下操作

optimum-cli export openvino --model stabilityai/stable-diffusion-xl-base-1.0 ov_sdxl/

您还可以在模型导出期间应用混合量化。例如

optimum-cli export openvino --model stabilityai/stable-diffusion-xl-base-1.0 \
    --weight-format int8 --dataset conceptual_captions ov_sdxl/

有关混合量化的更多信息，请查看此 jupyter notebook。

加载模型时

您还可以通过在加载模型时设置 export=True，来动态加载您的 PyTorch 检查点并将其转换为 OpenVINO 格式。

为了轻松保存生成的模型，您可以使用 save_pretrained() 方法，该方法将保存描述图的 BIN 和 XML 文件。将 tokenizer 保存到同一目录也很有用，以便轻松加载模型的 tokenizer。

- from transformers import AutoModelForCausalLM
+ from optimum.intel import OVModelForCausalLM
  from transformers import AutoTokenizer

  model_id = "meta-llama/Meta-Llama-3-8B"
- model = AutoModelForCausalLM.from_pretrained(model_id)
+ model = OVModelForCausalLM.from_pretrained(model_id, export=True)
  tokenizer = AutoTokenizer.from_pretrained(model_id)

  save_directory = "ov_model"
  model.save_pretrained(save_directory)
  tokenizer.save_pretrained(save_directory)

加载模型后

from transformers import AutoModelForCausalLM
from optimum.exporters.openvino import export_from_model

model = AutoModelForCausalLM.from_pretrained("meta-llama/Meta-Llama-3-8B")
export_from_model(model, output="ov_model", task="text-generation-with-past")

导出模型后，您现在可以通过将 AutoModelForXxx 类替换为相应的 OVModelForXxx 类来加载您的 OpenVINO 模型。