Diffusers 文档
SD3ControlNetModel
并获得增强的文档体验
开始使用
SD3ControlNetModel
SD3ControlNetModel 是 Stable Diffusion 3 的 ControlNet 实现。
ControlNet 模型由 Lvmin Zhang、Anyi Rao、Maneesh Agrawala 在向文本到图像扩散模型添加条件控制中提出。它通过对模型进行额外输入（如边缘图、深度图、分割图和姿态检测的关键点）的条件设置，从而对文本到图像生成提供更大程度的控制。
论文摘要如下：
我们提出了 ControlNet，一种神经网络架构，用于向大型预训练的文本到图像扩散模型添加空间条件控制。ControlNet 锁定生产就绪的大型扩散模型，并重用它们在数十亿图像上预训练的深度且鲁棒的编码层作为强大的骨干，以学习各种条件控制。神经网络架构通过“零卷积”（零初始化卷积层）连接，这些卷积层从零开始逐步增加参数，并确保不会有害噪声影响微调。我们使用 Stable Diffusion 测试了各种条件控制，例如边缘、深度、分割、人体姿态等，使用单一或多个条件，有或没有提示。我们表明 ControlNet 的训练对于小型（<50k）和大型（>1m）数据集都非常鲁棒。大量结果表明，ControlNet 可能有助于更广泛地应用以控制图像扩散模型。
从原始格式加载
默认情况下，SD3ControlNetModel 应使用 from_pretrained() 加载。
from diffusers import StableDiffusion3ControlNetPipeline
from diffusers.models import SD3ControlNetModel, SD3MultiControlNetModel
controlnet = SD3ControlNetModel.from_pretrained("InstantX/SD3-Controlnet-Canny")
pipe = StableDiffusion3ControlNetPipeline.from_pretrained("stabilityai/stable-diffusion-3-medium-diffusers", controlnet=controlnet)SD3ControlNetModel
class diffusers.SD3ControlNetModel
< 源 >( sample_size: int = 128 patch_size: int = 2 in_channels: int = 16 num_layers: int = 18 attention_head_dim: int = 64 num_attention_heads: int = 18 joint_attention_dim: int = 4096 caption_projection_dim: int = 1152 pooled_projection_dim: int = 2048 out_channels: int = 16 pos_embed_max_size: int = 96 extra_conditioning_channels: int = 0 dual_attention_layers: typing.Tuple[int, ...] = () qk_norm: typing.Optional[str] = None pos_embed_type: typing.Optional[str] = 'sincos' use_pos_embed: bool = True force_zeros_for_pooled_projection: bool = True )
参数
- sample_size (
int, 默认为128) — 潜在空间数据的宽度/高度。由于用于学习多个位置嵌入，因此在训练期间是固定的。 - patch_size (
int, 默认为2) — 将输入数据转换为小块的块大小。 - in_channels (
int, 默认为16) — 输入中的潜在通道数。 - num_layers (
int, 默认为18) — 要使用的 transformer 块层数。 - attention_head_dim (
int, 默认为64) — 每个注意力头的通道数。 - num_attention_heads (
int, 默认为18) — 多头注意力使用的头数。 - joint_attention_dim (
int, 默认为4096) — 用于联合文本-图像注意力的嵌入维度。 - caption_projection_dim (
int, 默认为1152) — 标题嵌入的嵌入维度。 - pooled_projection_dim (
int, 默认为2048) — 池化文本投影的嵌入维度。 - out_channels (
int, 默认为16) — 输出中的潜在通道数。 - pos_embed_max_size (
int, 默认为96) — 位置嵌入的最大潜在高度/宽度。 - extra_conditioning_channels (
int, 默认为0) — 用于补丁嵌入的额外条件通道数。 - dual_attention_layers (
Tuple[int, ...], 默认为()) — 要使用的双流 transformer 块数。 - qk_norm (
str, 可选, 默认为None) — 用于注意力层中查询和键的归一化方式。如果为None，则不使用归一化。 - pos_embed_type (
str, 默认为"sincos") — 要使用的位置嵌入类型。可在"sincos"和None之间选择。 - use_pos_embed (
bool, 默认为True) — 是否使用位置嵌入。 - force_zeros_for_pooled_projection (
bool, 默认为True) — 是否强制池化投影嵌入为零。这在管道中通过读取 ControlNet 模型的配置值进行处理。
用于 Stable Diffusion 3 的 ControlNet 模型。
前向
< 源 >( hidden_states: Tensor controlnet_cond: Tensor conditioning_scale: float = 1.0 encoder_hidden_states: Tensor = None pooled_projections: Tensor = None timestep: LongTensor = None joint_attention_kwargs: typing.Optional[typing.Dict[str, typing.Any]] = None return_dict: bool = True )
参数
- hidden_states (形状为
(batch size, channel, height, width)的torch.Tensor) — 输入的hidden_states。 - controlnet_cond (
torch.Tensor) — 形状为(batch_size, sequence_length, hidden_size)的条件输入张量。 - conditioning_scale (
float, 默认为1.0) — ControlNet 输出的比例因子。 - encoder_hidden_states (形状为
(batch size, sequence_len, embed_dims)的torch.Tensor) — 要使用的条件嵌入（从输入条件（如提示）计算出的嵌入）。 - pooled_projections (形状为
(batch_size, projection_dim)的torch.Tensor) — 从输入条件的嵌入投影而来的嵌入。 - timestep (
torch.LongTensor) — 用于指示去噪步骤。 - joint_attention_kwargs (
dict, 可选) — 一个 kwargs 字典，如果指定，将传递给 diffusers.models.attention_processor 中定义的self.processor的AttentionProcessor。 - return_dict (
bool, 可选, 默认为True) — 是否返回~models.transformer_2d.Transformer2DModelOutput而不是普通元组。
SD3Transformer2DModel 前向方法。
设置注意力处理器
< 源 >( processor: typing.Union[diffusers.models.attention_processor.AttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor2_0, diffusers.models.attention_processor.JointAttnProcessor2_0, diffusers.models.attention_processor.PAGJointAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGJointAttnProcessor2_0, diffusers.models.attention_processor.FusedJointAttnProcessor2_0, diffusers.models.attention_processor.AllegroAttnProcessor2_0, diffusers.models.attention_processor.AuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FusedAuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.CogVideoXAttnProcessor2_0, diffusers.models.attention_processor.FusedCogVideoXAttnProcessor2_0, diffusers.models.attention_processor.XFormersAttnAddedKVProcessor, diffusers.models.attention_processor.XFormersAttnProcessor, diffusers.models.attention_processor.XLAFlashAttnProcessor2_0, diffusers.models.attention_processor.AttnProcessorNPU, diffusers.models.attention_processor.AttnProcessor2_0, diffusers.models.attention_processor.MochiVaeAttnProcessor2_0, diffusers.models.attention_processor.MochiAttnProcessor2_0, diffusers.models.attention_processor.StableAudioAttnProcessor2_0, diffusers.models.attention_processor.HunyuanAttnProcessor2_0, diffusers.models.attention_processor.FusedHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.LuminaAttnProcessor2_0, diffusers.models.attention_processor.FusedAttnProcessor2_0, diffusers.models.attention_processor.CustomDiffusionXFormersAttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor2_0, diffusers.models.attention_processor.SlicedAttnProcessor, diffusers.models.attention_processor.SlicedAttnAddedKVProcessor, diffusers.models.attention_processor.SanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGSanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySanaLinearAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleLinearAttention, diffusers.models.attention_processor.SanaMultiscaleAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleAttentionProjection, diffusers.models.attention_processor.IPAdapterAttnProcessor, diffusers.models.attention_processor.IPAdapterAttnProcessor2_0, diffusers.models.attention_processor.IPAdapterXFormersAttnProcessor, diffusers.models.attention_processor.SD3IPAdapterJointAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.LoRAAttnProcessor, diffusers.models.attention_processor.LoRAAttnProcessor2_0, diffusers.models.attention_processor.LoRAXFormersAttnProcessor, diffusers.models.attention_processor.LoRAAttnAddedKVProcessor, typing.Dict[str, typing.Union[diffusers.models.attention_processor.AttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor2_0, diffusers.models.attention_processor.JointAttnProcessor2_0, diffusers.models.attention_processor.PAGJointAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGJointAttnProcessor2_0, diffusers.models.attention_processor.FusedJointAttnProcessor2_0, diffusers.models.attention_processor.AllegroAttnProcessor2_0, diffusers.models.attention_processor.AuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FusedAuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.CogVideoXAttnProcessor2_0, diffusers.models.attention_processor.FusedCogVideoXAttnProcessor2_0, diffusers.models.attention_processor.XFormersAttnAddedKVProcessor, diffusers.models.attention_processor.XFormersAttnProcessor, diffusers.models.attention_processor.XLAFlashAttnProcessor2_0, diffusers.models.attention_processor.AttnProcessorNPU, diffusers.models.attention_processor.AttnProcessor2_0, diffusers.models.attention_processor.MochiVaeAttnProcessor2_0, diffusers.models.attention_processor.MochiAttnProcessor2_0, diffusers.models.attention_processor.StableAudioAttnProcessor2_0, diffusers.models.attention_processor.HunyuanAttnProcessor2_0, diffusers.models.attention_processor.FusedHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.LuminaAttnProcessor2_0, diffusers.models.attention_processor.FusedAttnProcessor2_0, diffusers.models.attention_processor.CustomDiffusionXFormersAttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor2_0, diffusers.models.attention_processor.SlicedAttnProcessor, diffusers.models.attention_processor.SlicedAttnAddedKVProcessor, diffusers.models.attention_processor.SanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGSanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySanaLinearAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleLinearAttention, diffusers.models.attention_processor.SanaMultiscaleAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleAttentionProjection, diffusers.models.attention_processor.IPAdapterAttnProcessor, diffusers.models.attention_processor.IPAdapterAttnProcessor2_0, diffusers.models.attention_processor.IPAdapterXFormersAttnProcessor, diffusers.models.attention_processor.SD3IPAdapterJointAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.LoRAAttnProcessor, diffusers.models.attention_processor.LoRAAttnProcessor2_0, diffusers.models.attention_processor.LoRAXFormersAttnProcessor, diffusers.models.attention_processor.LoRAAttnAddedKVProcessor]]] )
设置用于计算注意力的注意力处理器。
SD3ControlNetOutput
class diffusers.models.controlnets.SD3ControlNetOutput
< 源 >( controlnet_block_samples: typing.Tuple[torch.Tensor] )