Diffusers 文档

Flux

Diffusers

加入 Hugging Face 社区

并获得增强的文档体验

协作处理模型、数据集和 Spaces

通过加速推理获得更快的示例

切换文档主题

开始使用

Flux

Flux 是一系列基于扩散 Transformer 的文本到图像生成模型。要了解有关 Flux 的更多信息，请查看 Flux 的创建者 Black Forest Labs 发布的博客文章。

Flux 的原始模型 checkpoints 可以在这里找到。原始推理代码可以在这里找到。

Flux 在消费级硬件设备上运行的成本可能相当高昂。但是，您可以执行一系列优化来使其运行更快且内存更友好。查看此部分了解更多详情。此外，Flux 可以从量化中受益，以提高内存效率，但会牺牲推理延迟。请参阅这篇博客文章以了解更多信息。有关详尽的资源列表，请查看此 gist。

Flux 提供以下变体

模型类型	模型 ID
Timestep-distilled	`black-forest-labs/FLUX.1-schnell`
Guidance-distilled	`black-forest-labs/FLUX.1-dev`
填充/外绘（Guidance-distilled）	`black-forest-labs/FLUX.1-Fill-dev`
Canny Control（Guidance-distilled）	`black-forest-labs/FLUX.1-Canny-dev`
Depth Control（Guidance-distilled）	`black-forest-labs/FLUX.1-Depth-dev`
Canny Control (LoRA)	`black-forest-labs/FLUX.1-Canny-dev-lora`
Depth Control (LoRA)	`black-forest-labs/FLUX.1-Depth-dev-lora`
Redux (Adapter)	`black-forest-labs/FLUX.1-Redux-dev`

所有 checkpoints 都有不同的用途，我们将在下面详细介绍。

Timestep-distilled

max_sequence_length 不能超过 256。
guidance_scale 需要为 0。
由于这是一个 timestep-distilled 模型，它受益于更少的采样步骤。

import torch
from diffusers import FluxPipeline

pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-schnell", torch_dtype=torch.bfloat16)
pipe.enable_model_cpu_offload()

prompt = "A cat holding a sign that says hello world"
out = pipe(
    prompt=prompt,
    guidance_scale=0.,
    height=768,
    width=1360,
    num_inference_steps=4,
    max_sequence_length=256,
).images[0]
out.save("image.png")

Guidance-distilled

guidance-distilled 变体大约需要 50 个采样步骤才能生成高质量的图像。
它对 max_sequence_length 没有任何限制。

import torch
from diffusers import FluxPipeline

pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16)
pipe.enable_model_cpu_offload()

prompt = "a tiny astronaut hatching from an egg on the moon"
out = pipe(
    prompt=prompt,
    guidance_scale=3.5,
    height=768,
    width=1360,
    num_inference_steps=50,
).images[0]
out.save("image.png")

Fill Inpainting/Outpainting

Flux Fill 管道不需要像常规的图像修复管道那样将 strength 作为输入。
它同时支持图像修复（inpainting）和外绘（outpainting）。

import torch
from diffusers import FluxFillPipeline
from diffusers.utils import load_image

image = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/cup.png")
mask = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/cup_mask.png")

repo_id = "black-forest-labs/FLUX.1-Fill-dev"
pipe = FluxFillPipeline.from_pretrained(repo_id, torch_dtype=torch.bfloat16).to("cuda")

image = pipe(
    prompt="a white paper cup",
    image=image,
    mask_image=mask,
    height=1632,
    width=1232,
    max_sequence_length=512,
    generator=torch.Generator("cpu").manual_seed(0)
).images[0]
image.save(f"output.png")

Canny Control

注意： black-forest-labs/Flux.1-Canny-dev 不是 ControlNetModel 模型。 ControlNet 模型是与 UNet/Transformer 分离的组件，其残差被添加到实际的底层模型中。 Canny Control 是一种替代架构，它通过使用与输入控制条件的通道wise连接，并确保 Transformer 通过尽可能紧密地遵循条件来学习结构控制，从而达到与 ControlNet 模型基本相同的效果。

# !pip install -U controlnet-aux
import torch
from controlnet_aux import CannyDetector
from diffusers import FluxControlPipeline
from diffusers.utils import load_image

pipe = FluxControlPipeline.from_pretrained("black-forest-labs/FLUX.1-Canny-dev", torch_dtype=torch.bfloat16).to("cuda")

prompt = "A robot made of exotic candies and chocolates of different kinds. The background is filled with confetti and celebratory gifts."
control_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/robot.png")

processor = CannyDetector()
control_image = processor(control_image, low_threshold=50, high_threshold=200, detect_resolution=1024, image_resolution=1024)

image = pipe(
    prompt=prompt,
    control_image=control_image,
    height=1024,
    width=1024,
    num_inference_steps=50,
    guidance_scale=30.0,
).images[0]
image.save("output.png")

Canny Control 也可以通过此条件的 LoRA 变体实现。用法如下

# !pip install -U controlnet-aux
import torch
from controlnet_aux import CannyDetector
from diffusers import FluxControlPipeline
from diffusers.utils import load_image

pipe = FluxControlPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16).to("cuda")
pipe.load_lora_weights("black-forest-labs/FLUX.1-Canny-dev-lora")

prompt = "A robot made of exotic candies and chocolates of different kinds. The background is filled with confetti and celebratory gifts."
control_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/robot.png")

processor = CannyDetector()
control_image = processor(control_image, low_threshold=50, high_threshold=200, detect_resolution=1024, image_resolution=1024)

image = pipe(
    prompt=prompt,
    control_image=control_image,
    height=1024,
    width=1024,
    num_inference_steps=50,
    guidance_scale=30.0,
).images[0]
image.save("output.png")

Depth Control

注意： black-forest-labs/Flux.1-Depth-dev 不是 ControlNet 模型。 ControlNetModel 模型是与 UNet/Transformer 分离的组件，其残差被添加到实际的底层模型中。 Depth Control 是一种替代架构，它通过使用与输入控制条件的通道wise连接，并确保 Transformer 通过尽可能紧密地遵循条件来学习结构控制，从而达到与 ControlNet 模型基本相同的效果。

# !pip install git+https://github.com/huggingface/image_gen_aux
import torch
from diffusers import FluxControlPipeline, FluxTransformer2DModel
from diffusers.utils import load_image
from image_gen_aux import DepthPreprocessor

pipe = FluxControlPipeline.from_pretrained("black-forest-labs/FLUX.1-Depth-dev", torch_dtype=torch.bfloat16).to("cuda")

prompt = "A robot made of exotic candies and chocolates of different kinds. The background is filled with confetti and celebratory gifts."
control_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/robot.png")

processor = DepthPreprocessor.from_pretrained("LiheYoung/depth-anything-large-hf")
control_image = processor(control_image)[0].convert("RGB")

image = pipe(
    prompt=prompt,
    control_image=control_image,
    height=1024,
    width=1024,
    num_inference_steps=30,
    guidance_scale=10.0,
    generator=torch.Generator().manual_seed(42),
).images[0]
image.save("output.png")

Depth Control 也可以通过此条件的 LoRA 变体实现。用法如下

# !pip install git+https://github.com/huggingface/image_gen_aux
import torch
from diffusers import FluxControlPipeline, FluxTransformer2DModel
from diffusers.utils import load_image
from image_gen_aux import DepthPreprocessor

pipe = FluxControlPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16).to("cuda")
pipe.load_lora_weights("black-forest-labs/FLUX.1-Depth-dev-lora")

prompt = "A robot made of exotic candies and chocolates of different kinds. The background is filled with confetti and celebratory gifts."
control_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/robot.png")

processor = DepthPreprocessor.from_pretrained("LiheYoung/depth-anything-large-hf")
control_image = processor(control_image)[0].convert("RGB")

image = pipe(
    prompt=prompt,
    control_image=control_image,
    height=1024,
    width=1024,
    num_inference_steps=30,
    guidance_scale=10.0,
    generator=torch.Generator().manual_seed(42),
).images[0]
image.save("output.png")

Redux

Flux Redux 管道是 FLUX.1 基础模型的适配器。它可以与 flux-dev 和 flux-schnell 一起使用，用于图像到图像的生成。
您可以首先使用 FluxPriorReduxPipeline 获取 prompt_embeds 和 pooled_prompt_embeds，然后将它们馈送到 FluxPipeline 中以进行图像到图像的生成。
当将 FluxPriorReduxPipeline 与基础管道一起使用时，您可以在基础管道中设置 text_encoder=None 和 text_encoder_2=None，以节省 VRAM。

import torch
from diffusers import FluxPriorReduxPipeline, FluxPipeline
from diffusers.utils import load_image
device = "cuda"
dtype = torch.bfloat16


repo_redux = "black-forest-labs/FLUX.1-Redux-dev"
repo_base = "black-forest-labs/FLUX.1-dev" 
pipe_prior_redux = FluxPriorReduxPipeline.from_pretrained(repo_redux, torch_dtype=dtype).to(device)
pipe = FluxPipeline.from_pretrained(
    repo_base, 
    text_encoder=None,
    text_encoder_2=None,
    torch_dtype=torch.bfloat16
).to(device)

image = load_image("https://huggingface.co/datasets/YiYiXu/testing-images/resolve/main/style_ziggy/img5.png")
pipe_prior_output = pipe_prior_redux(image)
images = pipe(
    guidance_scale=2.5,
    num_inference_steps=50,
    generator=torch.Generator("cpu").manual_seed(0),
    **pipe_prior_output,
).images
images[0].save("flux-redux.png")

Combining Flux Turbo LoRAs with Flux Control, Fill, and Redux

我们可以将 Flux Turbo LoRA 与 Flux Control 和其他管道（如 Fill 和 Redux）结合使用，以实现少步推理。下面的示例展示了如何为深度 Flux Control LoRA 和来自 ByteDance/Hyper-SD 的 turbo LoRA 执行此操作。

from diffusers import FluxControlPipeline
from image_gen_aux import DepthPreprocessor
from diffusers.utils import load_image
from huggingface_hub import hf_hub_download
import torch

control_pipe = FluxControlPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16)
control_pipe.load_lora_weights("black-forest-labs/FLUX.1-Depth-dev-lora", adapter_name="depth")
control_pipe.load_lora_weights(
    hf_hub_download("ByteDance/Hyper-SD", "Hyper-FLUX.1-dev-8steps-lora.safetensors"), adapter_name="hyper-sd"
)
control_pipe.set_adapters(["depth", "hyper-sd"], adapter_weights=[0.85, 0.125])
control_pipe.enable_model_cpu_offload()

prompt = "A robot made of exotic candies and chocolates of different kinds. The background is filled with confetti and celebratory gifts."
control_image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/robot.png")

processor = DepthPreprocessor.from_pretrained("LiheYoung/depth-anything-large-hf")
control_image = processor(control_image)[0].convert("RGB")

image = control_pipe(
    prompt=prompt,
    control_image=control_image,
    height=1024,
    width=1024,
    num_inference_steps=8,
    guidance_scale=10.0,
    generator=torch.Generator().manual_seed(42),
).images[0]
image.save("output.png")

Note about unload_lora_weights() when using Flux LoRAs

当卸载 Control LoRA 权重时，调用 pipe.unload_lora_weights(reset_to_overwritten_params=True) 以将 pipe.transformer 完全重置回其原始形式。由此产生的管道可以与诸如 DiffusionPipeline.from_pipe() 之类的方法一起使用。有关此参数的更多详细信息，请参见此 PR。

Running FP16 inference

Flux 可以使用 FP16 生成高质量的图像（即，加速 Turing/Volta GPU 上的推理），但与 FP32/BF16 相比，会产生不同的输出。问题是，当在 FP16 中运行时，文本编码器中的某些激活必须被裁剪，这会影响整体图像。因此，强制文本编码器使用 FP32 推理运行可以消除这种输出差异。有关详细信息，请参阅此处。

FP16 推理代码

import torch
from diffusers import FluxPipeline

pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-schnell", torch_dtype=torch.bfloat16) # can replace schnell with dev
# to run on low vram GPUs (i.e. between 4 and 32 GB VRAM)
pipe.enable_sequential_cpu_offload()
pipe.vae.enable_slicing()
pipe.vae.enable_tiling()

pipe.to(torch.float16) # casting here instead of in the pipeline constructor because doing so in the constructor loads all models into CPU memory at once

prompt = "A cat holding a sign that says hello world"
out = pipe(
    prompt=prompt,
    guidance_scale=0.,
    height=768,
    width=1360,
    num_inference_steps=4,
    max_sequence_length=256,
).images[0]
out.save("image.png")

Single File Loading for the FluxTransformer2DModel

FluxTransformer2DModel 支持加载 Black Forest Labs 发布的原始格式的检查点。当尝试加载社区发布的模型的微调或量化版本时，这也很有用。

`FP8` 推理可能不稳定，具体取决于您使用的 GPU 类型、CUDA 版本和 `torch` 版本。建议您使用 `optimum-quanto` 库，以便在您的机器上运行 FP8 推理。

以下示例演示了如何在少于 16GB VRAM 的情况下运行 Flux。

首先安装 optimum-quanto

pip install optimum-quanto

然后运行以下示例

import torch
from diffusers import FluxTransformer2DModel, FluxPipeline
from transformers import T5EncoderModel, CLIPTextModel
from optimum.quanto import freeze, qfloat8, quantize

bfl_repo = "black-forest-labs/FLUX.1-dev"
dtype = torch.bfloat16

transformer = FluxTransformer2DModel.from_single_file("https://huggingface.co/Kijai/flux-fp8/blob/main/flux1-dev-fp8.safetensors", torch_dtype=dtype)
quantize(transformer, weights=qfloat8)
freeze(transformer)

text_encoder_2 = T5EncoderModel.from_pretrained(bfl_repo, subfolder="text_encoder_2", torch_dtype=dtype)
quantize(text_encoder_2, weights=qfloat8)
freeze(text_encoder_2)

pipe = FluxPipeline.from_pretrained(bfl_repo, transformer=None, text_encoder_2=None, torch_dtype=dtype)
pipe.transformer = transformer
pipe.text_encoder_2 = text_encoder_2

pipe.enable_model_cpu_offload()

prompt = "A cat holding a sign that says hello world"
image = pipe(
    prompt,
    guidance_scale=3.5,
    output_type="pil",
    num_inference_steps=20,
    generator=torch.Generator("cpu").manual_seed(0)
).images[0]

image.save("flux-fp8-dev.png")

Diffusers

Flux

Timestep-distilled

Guidance-distilled

Fill Inpainting/Outpainting

Canny Control

Depth Control

Redux

Combining Flux Turbo LoRAs with Flux Control, Fill, and Redux

Note about unload_lora_weights() when using Flux LoRAs

Running FP16 inference

Single File Loading for the FluxTransformer2DModel

FluxPipeline

class diffusers.FluxPipeline

__call__

disable_vae_slicing

disable_vae_tiling

enable_vae_slicing

enable_vae_tiling

encode_prompt

FluxImg2ImgPipeline

class diffusers.FluxImg2ImgPipeline

__call__

encode_prompt

FluxInpaintPipeline

class diffusers.FluxInpaintPipeline

__call__

encode_prompt

FluxControlNetInpaintPipeline

class diffusers.FluxControlNetInpaintPipeline

__call__

encode_prompt

FluxControlNetImg2ImgPipeline

class diffusers.FluxControlNetImg2ImgPipeline

__call__

encode_prompt

FluxControlPipeline

class diffusers.FluxControlPipeline

__call__

disable_vae_slicing

disable_vae_tiling

enable_vae_slicing

enable_vae_tiling

encode_prompt

FluxControlImg2ImgPipeline

class diffusers.FluxControlImg2ImgPipeline

__call__

encode_prompt

FluxPriorReduxPipeline

class diffusers.FluxPriorReduxPipeline

__call__

encode_prompt

FluxFillPipeline

class diffusers.FluxFillPipeline

__call__

disable_vae_slicing

disable_vae_tiling

enable_vae_slicing

enable_vae_tiling

encode_prompt

call

call

call

call

call

call

call

call

call