使用扩散模型生成图像
稳定扩散
使用 OpenVINO 进行推理时也可以使用稳定扩散模型。当稳定扩散模型导出为 OpenVINO 格式时，它们会被分解成不同的组件，这些组件随后会在推理过程中合并在一起。
- 文本编码器
- U-NET
- VAE 编码器
- VAE 解码器
| 任务 | 自动分类 |
|---|---|
文本到图像 | OVStableDiffusionPipeline |
图像到图像 | OVStableDiffusionImg2ImgPipeline |
修复 | OVStableDiffusionInpaintPipeline |
文本到图像
以下是如何加载 OpenVINO 稳定扩散模型并使用 OpenVINO Runtime 运行推理的示例
from optimum.intel import OVStableDiffusionPipeline
model_id = "echarlaix/stable-diffusion-v1-5-openvino"
pipeline = OVStableDiffusionPipeline.from_pretrained(model_id)
prompt = "sailing ship in storm by Rembrandt"
images = pipeline(prompt).images要加载您的 PyTorch 模型并将其动态转换为 OpenVINO，您可以设置 export=True。
model_id = "runwayml/stable-diffusion-v1-5"
pipeline = OVStableDiffusionPipeline.from_pretrained(model_id, export=True)
# Don't forget to save the exported model
pipeline.save_pretrained("openvino-sd-v1-5")为了进一步加快推理速度，可以对模型进行静态重塑
# Define the shapes related to the inputs and desired outputs
batch_size, num_images, height, width = 1, 1, 512, 512
# Statically reshape the model
pipeline.reshape(batch_size=batch_size, height=height, width=width, num_images_per_prompt=num_images)
# Compile the model before the first inference
pipeline.compile()
# Run inference
images = pipeline(prompt, height=height, width=width, num_images_per_prompt=num_images).images如果您想更改任何参数（例如输出高度或宽度），则需要再次对模型进行静态重塑。
使用文本反转的文本到图像
以下是如何加载带有预训练文本反转嵌入的 OpenVINO 稳定扩散模型并使用 OpenVINO Runtime 运行推理的示例
首先，您可以运行没有文本反转的原始管道
from optimum.intel import OVStableDiffusionPipeline
import numpy as np
model_id = "echarlaix/stable-diffusion-v1-5-openvino"
prompt = "A <cat-toy> back-pack"
# Set a random seed for better comparison
np.random.seed(42)
pipeline = OVStableDiffusionPipeline.from_pretrained(model_id, export=False, compile=False)
pipeline.compile()
image1 = pipeline(prompt, num_inference_steps=50).images[0]
image1.save("stable_diffusion_v1_5_without_textual_inversion.png")然后，您可以加载 sd-concepts-library/cat-toy 文本反转嵌入，并使用相同的提示再次运行管道
# Reset stable diffusion pipeline
pipeline.clear_requests()
# Load textual inversion into stable diffusion pipeline
pipeline.load_textual_inversion("sd-concepts-library/cat-toy", "<cat-toy>")
# Compile the model before the first inference
pipeline.compile()
image2 = pipeline(prompt, num_inference_steps=50).images[0]
image2.save("stable_diffusion_v1_5_with_textual_inversion.png")左侧图像显示了原始稳定扩散 v1.5 的生成结果，右侧图像显示了带有文本反转的稳定扩散 v1.5 的生成结果。
 |  |
图像到图像
import requests
import torch
from PIL import Image
from io import BytesIO
from optimum.intel import OVStableDiffusionImg2ImgPipeline
model_id = "runwayml/stable-diffusion-v1-5"
pipeline = OVStableDiffusionImg2ImgPipeline.from_pretrained(model_id, export=True)
url = "https://raw.githubusercontent.com/CompVis/stable-diffusion/main/assets/stable-samples/img2img/sketch-mountains-input.jpg"
response = requests.get(url)
init_image = Image.open(BytesIO(response.content)).convert("RGB")
init_image = init_image.resize((768, 512))
prompt = "A fantasy landscape, trending on artstation"
image = pipeline(prompt=prompt, image=init_image, strength=0.75, guidance_scale=7.5).images[0]
image.save("fantasy_landscape.png")稳定扩散 XL
| 任务 | 自动分类 |
|---|---|
文本到图像 | OVStableDiffusionXLPipeline |
图像到图像 | OVStableDiffusionXLImg2ImgPipeline |
文本到图像
以下是如何从 stabilityai/stable-diffusion-xl-base-1.0 加载 SDXL OpenVINO 模型并使用 OpenVINO Runtime 运行推理的示例
from optimum.intel import OVStableDiffusionXLPipeline
model_id = "stabilityai/stable-diffusion-xl-base-1.0"
base = OVStableDiffusionXLPipeline.from_pretrained(model_id)
prompt = "train station by Caspar David Friedrich"
image = base(prompt).images[0]
image.save("train_station.png") |  |
使用文本反转的文本到图像
以下是如何从 stabilityai/stable-diffusion-xl-base-1.0 加载带有预训练文本反转嵌入的 SDXL OpenVINO 模型并使用 OpenVINO Runtime 运行推理的示例
首先，您可以运行没有文本反转的原始管道
from optimum.intel import OVStableDiffusionXLPipeline
import numpy as np
model_id = "stabilityai/stable-diffusion-xl-base-1.0"
prompt = "charturnerv2, multiple views of the same character in the same outfit, a character turnaround wearing a red jacket and black shirt, best quality, intricate details."
# Set a random seed for better comparison
np.random.seed(112)
base = OVStableDiffusionXLPipeline.from_pretrained(model_id, export=False, compile=False)
base.compile()
image1 = base(prompt, num_inference_steps=50).images[0]
image1.save("sdxl_without_textual_inversion.png")然后，您可以加载 charturnerv2 文本反转嵌入，并使用相同的提示再次运行管道
# Reset stable diffusion pipeline
base.clear_requests()
# Load textual inversion into stable diffusion pipeline
base.load_textual_inversion("./charturnerv2.pt", "charturnerv2")
# Compile the model before the first inference
base.compile()
image2 = base(prompt, num_inference_steps=50).images[0]
image2.save("sdxl_with_textual_inversion.png")图像到图像
以下是如何加载 PyTorch SDXL 模型，将其动态转换为 OpenVINO 并使用 OpenVINO Runtime 为 *图像到图像* 运行推理的示例
from optimum.intel import OVStableDiffusionXLImg2ImgPipeline
from diffusers.utils import load_image
model_id = "stabilityai/stable-diffusion-xl-refiner-1.0"
pipeline = OVStableDiffusionXLImg2ImgPipeline.from_pretrained(model_id, export=True)
url = "https://huggingface.co/datasets/optimum/documentation-images/resolve/main/intel/openvino/sd_xl/castle_friedrich.png"
image = load_image(url).convert("RGB")
prompt = "medieval castle by Caspar David Friedrich"
image = pipeline(prompt, image=image).images[0]
# Don't forget to save your OpenVINO model so that you can load it without exporting it with `export=True`
pipeline.save_pretrained("openvino-sd-xl-refiner-1.0")细化图像输出
可以使用类似 stabilityai/stable-diffusion-xl-refiner-1.0 的模型来细化图像。在这种情况下，你只需要输出基础模型的潜在表示即可。
from optimum.intel import OVStableDiffusionXLImg2ImgPipeline
model_id = "stabilityai/stable-diffusion-xl-refiner-1.0"
refiner = OVStableDiffusionXLImg2ImgPipeline.from_pretrained(model_id, export=True)
image = base(prompt=prompt, output_type="latent").images[0]
image = refiner(prompt=prompt, image=image[None, :]).images[0]潜在一致性模型
| 任务 | 自动分类 |
|---|---|
文本到图像 | OVLatentConsistencyModelPipeline |
文本到图像
以下是如何从 SimianLuo/LCM_Dreamshaper_v7 加载潜在一致性模型 (LCM) 并使用 OpenVINO 进行推理的示例。
from optimum.intel import OVLatentConsistencyModelPipeline
model_id = "SimianLuo/LCM_Dreamshaper_v7"
pipeline = OVLatentConsistencyModelPipeline.from_pretrained(model_id, export=True)
prompt = "sailing ship in storm by Leonardo da Vinci"
images = pipeline(prompt, num_inference_steps=4, guidance_scale=8.0).images