Diffusers 文档
SD3ControlNetModel
并获得增强的文档体验
开始使用
SD3ControlNetModel
SD3ControlNetModel 是 ControlNet 为 Stable Diffusion 3 的一个实现。
ControlNet 模型在 Adding Conditional Control to Text-to-Image Diffusion Models 这篇论文中被介绍，作者是 Lvmin Zhang, Anyi Rao, Maneesh Agrawala。它通过在模型上添加额外的输入（例如边缘图、深度图、分割图和姿势检测的关键点）来提供对文本到图像生成更高级别的控制。
该论文的摘要是
我们提出了 ControlNet，一种神经网络架构，用于向大型预训练的文本到图像扩散模型添加空间条件控制。ControlNet 锁定生产就绪的大型扩散模型，并重用其经过数十亿张图像预训练的深层和鲁棒的编码层，作为学习各种条件控制的强大骨干。该神经网络架构与“零卷积”（zero convolutions）（零初始化的卷积层）相连，这些卷积层逐渐从零开始增长参数，并确保没有有害噪声会影响微调。我们使用 Stable Diffusion 测试了各种条件控制，例如，边缘、深度、分割、人体姿势等，使用单个或多个条件，有或没有提示。结果表明，ControlNet 的训练对于小型（<50k）和大型（>1m）数据集都是稳健的。广泛的结果表明，ControlNet 可以促进更广泛的应用，以控制图像扩散模型。
从原始格式加载
默认情况下，SD3ControlNetModel 应该使用 from_pretrained() 加载。
from diffusers import StableDiffusion3ControlNetPipeline
from diffusers.models import SD3ControlNetModel, SD3MultiControlNetModel
controlnet = SD3ControlNetModel.from_pretrained("InstantX/SD3-Controlnet-Canny")
pipe = StableDiffusion3ControlNetPipeline.from_pretrained("stabilityai/stable-diffusion-3-medium-diffusers", controlnet=controlnet)SD3ControlNetModel
class diffusers.SD3ControlNetModel
< source >( sample_size: int = 128 patch_size: int = 2 in_channels: int = 16 num_layers: int = 18 attention_head_dim: int = 64 num_attention_heads: int = 18 joint_attention_dim: int = 4096 caption_projection_dim: int = 1152 pooled_projection_dim: int = 2048 out_channels: int = 16 pos_embed_max_size: int = 96 extra_conditioning_channels: int = 0 dual_attention_layers: typing.Tuple[int, ...] = () qk_norm: typing.Optional[str] = None pos_embed_type: typing.Optional[str] = 'sincos' use_pos_embed: bool = True force_zeros_for_pooled_projection: bool = True )
enable_forward_chunking
< source >( chunk_size: typing.Optional[int] = None dim: int = 0 )
设置 attention processor 以使用 feed forward chunking。
forward
< source >( hidden_states: FloatTensor controlnet_cond: Tensor conditioning_scale: float = 1.0 encoder_hidden_states: FloatTensor = None pooled_projections: FloatTensor = None timestep: LongTensor = None joint_attention_kwargs: typing.Optional[typing.Dict[str, typing.Any]] = None return_dict: bool = True )
参数
- hidden_states (
torch.FloatTensor，形状为(batch size, channel, height, width)) — 输入hidden_states。 - controlnet_cond (
torch.Tensor) — 形状为(batch_size, sequence_length, hidden_size)的条件输入张量。 - conditioning_scale (
float, 默认为1.0) — ControlNet 输出的缩放因子。 - encoder_hidden_states (形状为
(batch size, sequence_len, embed_dims)的torch.FloatTensor) — 要使用的条件嵌入（从输入条件（如提示）计算出的嵌入）。 - pooled_projections (形状为
(batch_size, projection_dim)的torch.FloatTensor) — 从输入条件的嵌入投影得到的嵌入。 - timestep (
torch.LongTensor) — 用于指示去噪步骤。 - joint_attention_kwargs (
dict, 可选) — 一个 kwargs 字典，如果指定，则会传递给 diffusers.models.attention_processor 中self.processor下定义的AttentionProcessor。 - return_dict (
bool, 可选, 默认为True) — 是否返回~models.transformer_2d.Transformer2DModelOutput而不是普通元组。
The SD3Transformer2DModel 的 forward 方法。
启用融合的 QKV 投影。对于自注意力模块，所有投影矩阵（即，查询、键、值）都将融合。对于交叉注意力模块，键和值投影矩阵将被融合。
此 API 为 🧪 实验性 API。
set_attn_processor
< 源码 >( processor: typing.Union[diffusers.models.attention_processor.AttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor2_0, diffusers.models.attention_processor.JointAttnProcessor2_0, diffusers.models.attention_processor.PAGJointAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGJointAttnProcessor2_0, diffusers.models.attention_processor.FusedJointAttnProcessor2_0, diffusers.models.attention_processor.AllegroAttnProcessor2_0, diffusers.models.attention_processor.AuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FusedAuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.CogVideoXAttnProcessor2_0, diffusers.models.attention_processor.FusedCogVideoXAttnProcessor2_0, diffusers.models.attention_processor.XFormersAttnAddedKVProcessor, diffusers.models.attention_processor.XFormersAttnProcessor, diffusers.models.attention_processor.XLAFlashAttnProcessor2_0, diffusers.models.attention_processor.AttnProcessorNPU, diffusers.models.attention_processor.AttnProcessor2_0, diffusers.models.attention_processor.MochiVaeAttnProcessor2_0, diffusers.models.attention_processor.MochiAttnProcessor2_0, diffusers.models.attention_processor.StableAudioAttnProcessor2_0, diffusers.models.attention_processor.HunyuanAttnProcessor2_0, diffusers.models.attention_processor.FusedHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.LuminaAttnProcessor2_0, diffusers.models.attention_processor.FusedAttnProcessor2_0, diffusers.models.attention_processor.CustomDiffusionXFormersAttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor2_0, diffusers.models.attention_processor.SlicedAttnProcessor, diffusers.models.attention_processor.SlicedAttnAddedKVProcessor, diffusers.models.attention_processor.SanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGSanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySanaLinearAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleLinearAttention, diffusers.models.attention_processor.SanaMultiscaleAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleAttentionProjection, diffusers.models.attention_processor.IPAdapterAttnProcessor, diffusers.models.attention_processor.IPAdapterAttnProcessor2_0, diffusers.models.attention_processor.IPAdapterXFormersAttnProcessor, diffusers.models.attention_processor.SD3IPAdapterJointAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.LoRAAttnProcessor, diffusers.models.attention_processor.LoRAAttnProcessor2_0, diffusers.models.attention_processor.LoRAXFormersAttnProcessor, diffusers.models.attention_processor.LoRAAttnAddedKVProcessor]]] )
设置用于计算注意力的注意力处理器。
SD3ControlNetOutput
class diffusers.models.controlnets.SD3ControlNetOutput
< 源码 >( controlnet_block_samples: typing.Tuple[torch.Tensor] )