InstructPix2Pix

概述

InstructPix2Pix：学习遵循图像编辑指令由 Tim Brooks、Aleksander Holynski 和 Alexei A. Efros 撰写。

🤗 Optimum 扩展了 Diffusers 以支持在第二代 Neuron 设备（支持 Trainium 和 Inferentia 2）上进行推理。它旨在继承 Diffusers 在 Neuron 上的易用性。

导出到 Neuron

要部署模型，您需要将它们编译为针对 AWS Neuron 优化的 TorchScript。对于 Stable Diffusion，有四个组件需要导出为 .neuron 格式以提高性能

文本编码器
U-Net
VAE 编码器
VAE 解码器

您可以通过 CLI 或 NeuronStableDiffusionInstructPix2PixPipeline 类编译和导出 Stable Diffusion Checkpoint。

使用示例

使用 NeuronStableDiffusionInstructPix2PixPipeline 类，您可以利用文本和图像指导应用基于指令的图像编辑。

import requests
import PIL
from io import BytesIO
from optimum.neuron import NeuronStableDiffusionInstructPix2PixPipeline

def download_image(url):
    response = requests.get(url)
    return PIL.Image.open(BytesIO(response.content)).convert("RGB")

model_id = "timbrooks/instruct-pix2pix"
input_shapes = {"batch_size": 1, "height": 512, "width": 512}

pipe = NeuronStableDiffusionInstructPix2PixPipeline.from_pretrained(
  model_id, export=True, dynamic_batch_size=True, **input_shapes,
)
pipe.save_pretrained("sd_ip2p/")

img_url = "https://huggingface.co/datasets/diffusers/diffusers-images-docs/resolve/main/mountain.png"
init_image = download_image(img_url).resize((512, 512))

prompt = "Add a beautiful sunset"
image = pipe(prompt=prompt, image=init_image).images[0]
image.save("sunset_mountain.png")

`图像`	`提示`	输出
	*添加美丽的日落*

NeuronStableDiffusionInstructPix2PixPipeline

class optimum.neuron.NeuronStableDiffusionInstructPix2PixPipeline

< 源文件 >

( config: dict[str, typing.Any] configs: dict[str, 'PretrainedConfig'] neuron_configs: dict[str, 'NeuronDefaultConfig'] data_parallel_mode: typing.Literal['none', 'unet', 'transformer', 'all'] scheduler: diffusers.schedulers.scheduling_utils.SchedulerMixin | None vae_decoder: torch.jit._script.ScriptModule | NeuronModelVaeDecoder text_encoder: torch.jit._script.ScriptModule | NeuronModelTextEncoder | None = None text_encoder_2: torch.jit._script.ScriptModule | NeuronModelTextEncoder | None = None unet: torch.jit._script.ScriptModule | NeuronModelUnet | None = None transformer: torch.jit._script.ScriptModule | NeuronModelTransformer | None = None vae_encoder: torch.jit._script.ScriptModule | NeuronModelVaeEncoder | None = None image_encoder: torch.jit._script.ScriptModule | None = None safety_checker: torch.jit._script.ScriptModule | None = None tokenizer: transformers.models.clip.tokenization_clip.CLIPTokenizer | transformers.models.t5.tokenization_t5.T5Tokenizer | None = None tokenizer_2: transformers.models.clip.tokenization_clip.CLIPTokenizer | None = None feature_extractor: transformers.models.clip.feature_extraction_clip.CLIPFeatureExtractor | None = None controlnet: torch.jit._script.ScriptModule | list[torch.jit._script.ScriptModule]| NeuronControlNetModel | NeuronMultiControlNetModel | None = None requires_aesthetics_score: bool = False force_zeros_for_empty_prompt: bool = True add_watermarker: bool | None = None model_save_dir: str | pathlib.Path | tempfile.TemporaryDirectory | None = None model_and_config_save_paths: dict[str, tuple[str, pathlib.Path]] | None = None )

call

< 源文件 >

( *args **kwargs )

您希望我们在 🤗Optimum-neuron 中支持其他扩散功能吗？请在 Optimum-neuron Github 仓库中提交问题或在 HuggingFace 社区论坛上与我们讨论，谢谢 🤗！

AWS Trainium & Inferentia