Diffusers

加入 Hugging Face 社区

并获得增强的文档体验

在模型、数据集和 Spaces 上进行协作

通过加速推理获得更快的示例

切换文档主题

开始使用

HunyuanDiT2DControlNetModel

HunyuanDiT2DControlNetModel 是 Hunyuan-DiT 的 ControlNet 实现。

ControlNet 由 Lvmin Zhang、Anyi Rao 和 Maneesh Agrawala 在《为文本到图像扩散模型添加条件控制》中提出。

通过 ControlNet 模型，您可以提供额外的控制图像来条件化和控制 Hunyuan-DiT 的生成。例如，如果您提供深度图，ControlNet 模型将生成一张保留深度图空间信息的图像。这是一种更灵活、更准确的图像生成控制方式。

论文摘要如下：

我们提出了 ControlNet，一种用于为大型预训练文本到图像扩散模型添加空间条件控制的神经网络架构。ControlNet 锁定生产就绪的大型扩散模型，并重用它们通过数十亿张图像预训练的深度且强大的编码层作为强大的主干，以学习多样化的条件控制集。神经网络架构与“零卷积”（零初始化卷积层）连接，这些卷积层逐步从零开始增长参数，并确保没有有害噪声会影响微调。我们使用 Stable Diffusion 测试了各种条件控制，例如边缘、深度、分割、人体姿态等，可以使用单个或多个条件，带或不带提示。我们表明 ControlNet 的训练对于小型（<50k）和大型（>1m）数据集都具有鲁棒性。大量结果表明，ControlNet 可能有助于更广泛地应用以控制图像扩散模型。

此代码由腾讯混元团队实现。您可以在腾讯混元上找到 Hunyuan-DiT ControlNets 的预训练检查点。

加载 HunyuanDiT2DControlNetModel 的示例

from diffusers import HunyuanDiT2DControlNetModel
import torch
controlnet = HunyuanDiT2DControlNetModel.from_pretrained("Tencent-Hunyuan/HunyuanDiT-v1.1-ControlNet-Diffusers-Pose", torch_dtype=torch.float16)

HunyuanDiT2DControlNetModel

class diffusers.HunyuanDiT2DControlNetModel

< source >

( conditioning_channels: int = 3 num_attention_heads: int = 16 attention_head_dim: int = 88 in_channels: typing.Optional[int] = None patch_size: typing.Optional[int] = None activation_fn: str = 'gelu-approximate' sample_size = 32 hidden_size = 1152 transformer_num_layers: int = 40 mlp_ratio: float = 4.0 cross_attention_dim: int = 1024 cross_attention_dim_t5: int = 2048 pooled_projection_dim: int = 1024 text_len: int = 77 text_len_t5: int = 256 use_style_cond_and_image_meta_size: bool = True )

forward

< source >

( hidden_states timestep controlnet_cond: Tensor conditioning_scale: float = 1.0 encoder_hidden_states = None text_embedding_mask = None encoder_hidden_states_t5 = None text_embedding_mask_t5 = None image_meta_size = None style = None image_rotary_emb = None return_dict = True )

参数

hidden_states (形状为 (batch size, dim, height, width) 的 torch.Tensor) — 输入张量。
timestep (torch.LongTensor, 可选) — 用于指示去噪步骤。
controlnet_cond (torch.Tensor) — ControlNet 的条件输入。
conditioning_scale (float) — 表示条件缩放。
encoder_hidden_states (形状为 (batch size, sequence len, embed dims) 的 torch.Tensor, 可选) — 交叉注意力层的条件嵌入。这是 BertModel 的输出。
text_embedding_mask — torch.Tensor 形状为 (batch, key_tokens) 的注意力掩码应用于 encoder_hidden_states。这是 BertModel 的输出。
encoder_hidden_states_t5 (形状为 (batch size, sequence len, embed dims) 的 torch.Tensor, 可选) — 交叉注意力层的条件嵌入。这是 T5 文本编码器的输出。
text_embedding_mask_t5 — torch.Tensor 形状为 (batch, key_tokens) 的注意力掩码应用于 encoder_hidden_states。这是 T5 文本编码器的输出。
image_meta_size (torch.Tensor) — 指示图像大小的条件嵌入
style — torch.Tensor：指示风格的条件嵌入
image_rotary_emb (torch.Tensor) — 在注意力计算过程中应用于查询和键张量的图像旋转嵌入。
return_dict — bool 是否返回字典。

HunyuanDiT2DControlNetModel 的 forward 方法。

设置注意力处理器

< source >

( processor: typing.Union[diffusers.models.attention_processor.AttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor2_0, diffusers.models.attention_processor.JointAttnProcessor2_0, diffusers.models.attention_processor.PAGJointAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGJointAttnProcessor2_0, diffusers.models.attention_processor.FusedJointAttnProcessor2_0, diffusers.models.attention_processor.AllegroAttnProcessor2_0, diffusers.models.attention_processor.AuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FusedAuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.CogVideoXAttnProcessor2_0, diffusers.models.attention_processor.FusedCogVideoXAttnProcessor2_0, diffusers.models.attention_processor.XFormersAttnAddedKVProcessor, diffusers.models.attention_processor.XFormersAttnProcessor, diffusers.models.attention_processor.XLAFlashAttnProcessor2_0, diffusers.models.attention_processor.AttnProcessorNPU, diffusers.models.attention_processor.AttnProcessor2_0, diffusers.models.attention_processor.MochiVaeAttnProcessor2_0, diffusers.models.attention_processor.MochiAttnProcessor2_0, diffusers.models.attention_processor.StableAudioAttnProcessor2_0, diffusers.models.attention_processor.HunyuanAttnProcessor2_0, diffusers.models.attention_processor.FusedHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.LuminaAttnProcessor2_0, diffusers.models.attention_processor.FusedAttnProcessor2_0, diffusers.models.attention_processor.CustomDiffusionXFormersAttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor2_0, diffusers.models.attention_processor.SlicedAttnProcessor, diffusers.models.attention_processor.SlicedAttnAddedKVProcessor, diffusers.models.attention_processor.SanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGSanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySanaLinearAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleLinearAttention, diffusers.models.attention_processor.SanaMultiscaleAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleAttentionProjection, diffusers.models.attention_processor.IPAdapterAttnProcessor, diffusers.models.attention_processor.IPAdapterAttnProcessor2_0, diffusers.models.attention_processor.IPAdapterXFormersAttnProcessor, diffusers.models.attention_processor.SD3IPAdapterJointAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.LoRAAttnProcessor, diffusers.models.attention_processor.LoRAAttnProcessor2_0, diffusers.models.attention_processor.LoRAXFormersAttnProcessor, diffusers.models.attention_processor.LoRAAttnAddedKVProcessor, typing.Dict[str, typing.Union[diffusers.models.attention_processor.AttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor2_0, diffusers.models.attention_processor.JointAttnProcessor2_0, diffusers.models.attention_processor.PAGJointAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGJointAttnProcessor2_0, diffusers.models.attention_processor.FusedJointAttnProcessor2_0, diffusers.models.attention_processor.AllegroAttnProcessor2_0, diffusers.models.attention_processor.AuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FusedAuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.CogVideoXAttnProcessor2_0, diffusers.models.attention_processor.FusedCogVideoXAttnProcessor2_0, diffusers.models.attention_processor.XFormersAttnAddedKVProcessor, diffusers.models.attention_processor.XFormersAttnProcessor, diffusers.models.attention_processor.XLAFlashAttnProcessor2_0, diffusers.models.attention_processor.AttnProcessorNPU, diffusers.models.attention_processor.AttnProcessor2_0, diffusers.models.attention_processor.MochiVaeAttnProcessor2_0, diffusers.models.attention_processor.MochiAttnProcessor2_0, diffusers.models.attention_processor.StableAudioAttnProcessor2_0, diffusers.models.attention_processor.HunyuanAttnProcessor2_0, diffusers.models.attention_processor.FusedHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.LuminaAttnProcessor2_0, diffusers.models.attention_processor.FusedAttnProcessor2_0, diffusers.models.attention_processor.CustomDiffusionXFormersAttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor2_0, diffusers.models.attention_processor.SlicedAttnProcessor, diffusers.models.attention_processor.SlicedAttnAddedKVProcessor, diffusers.models.attention_processor.SanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGSanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySanaLinearAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleLinearAttention, diffusers.models.attention_processor.SanaMultiscaleAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleAttentionProjection, diffusers.models.attention_processor.IPAdapterAttnProcessor, diffusers.models.attention_processor.IPAdapterAttnProcessor2_0, diffusers.models.attention_processor.IPAdapterXFormersAttnProcessor, diffusers.models.attention_processor.SD3IPAdapterJointAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.LoRAAttnProcessor, diffusers.models.attention_processor.LoRAAttnProcessor2_0, diffusers.models.attention_processor.LoRAXFormersAttnProcessor, diffusers.models.attention_processor.LoRAAttnAddedKVProcessor]]] )

参数

processor (AttentionProcessor 字典或仅 AttentionProcessor) — 实例化的处理器类或处理器类字典，将被设置为**所有** Attention 层的处理器。如果 processor 是字典，则键需要定义到相应交叉注意力处理器的路径。强烈建议在设置可训练注意力处理器时使用此方法。

设置用于计算注意力的注意力处理器。

< > 在 GitHub 上更新

←FluxControlNetModel SanaControlNetModel→