Diffusers

加入 Hugging Face 社区

并获得增强的文档体验

在模型、数据集和 Spaces 上协作

通过加速推理获得更快的示例

切换文档主题

开始使用

混元DiT2D ControlNet 模型

HunyuanDiT2DControlNetModel 是 ControlNet 为 Hunyuan-DiT 的一个实现。

ControlNet 由 Lvmin Zhang、Anyi Rao 和 Maneesh Agrawala 在论文 Adding Conditional Control to Text-to-Image Diffusion Models 中提出。

使用 ControlNet 模型，您可以提供额外的控制图像来调节和控制 Hunyuan-DiT 的生成。例如，如果您提供深度图，ControlNet 模型将生成一个保留深度图空间信息的图像。这是一种更灵活、更准确的图像生成过程控制方法。

该论文的摘要是

我们提出了 ControlNet，这是一种神经网络架构，用于为大型预训练的文本到图像扩散模型添加空间条件控制。 ControlNet 锁定已准备好生产的大型扩散模型，并重用其经过数十亿张图像预训练的深度和鲁棒的编码层，作为学习各种条件控制的强大骨干。该神经网络架构与“零卷积”（零初始化的卷积层）相连，这些卷积层从零开始逐步增加参数，并确保没有有害噪声会影响微调。我们使用 Stable Diffusion 测试了各种条件控制，例如边缘、深度、分割、人体姿势等，使用单个或多个条件，有或没有提示。我们表明，ControlNet 的训练对于小型（<50k）和大型（>1m）数据集都是稳健的。大量结果表明，ControlNet 可以促进更广泛的应用来控制图像扩散模型。

此代码由腾讯混元团队实现。您可以在 Tencent Hunyuan 上找到 Hunyuan-DiT ControlNets 的预训练检查点。

加载 HunyuanDiT2DControlNetModel 的示例

from diffusers import HunyuanDiT2DControlNetModel
import torch
controlnet = HunyuanDiT2DControlNetModel.from_pretrained("Tencent-Hunyuan/HunyuanDiT-v1.1-ControlNet-Diffusers-Pose", torch_dtype=torch.float16)

HunyuanDiT2DControlNetModel

class diffusers.HunyuanDiT2DControlNetModel

< source >

( conditioning_channels: int = 3 num_attention_heads: int = 16 attention_head_dim: int = 88 in_channels: typing.Optional[int] = None patch_size: typing.Optional[int] = None activation_fn: str = 'gelu-approximate' sample_size = 32 hidden_size = 1152 transformer_num_layers: int = 40 mlp_ratio: float = 4.0 cross_attention_dim: int = 1024 cross_attention_dim_t5: int = 2048 pooled_projection_dim: int = 1024 text_len: int = 77 text_len_t5: int = 256 use_style_cond_and_image_meta_size: bool = True )

forward

< source >

( hidden_states timestep controlnet_cond: Tensor conditioning_scale: float = 1.0 encoder_hidden_states = None text_embedding_mask = None encoder_hidden_states_t5 = None text_embedding_mask_t5 = None image_meta_size = None style = None image_rotary_emb = None return_dict = True )

参数

hidden_states (torch.Tensor，形状为 (batch size, dim, height, width)) — 输入张量。
timestep ( torch.LongTensor, 可选) — 用于指示去噪步骤。
controlnet_cond ( torch.Tensor ) — ControlNet 的条件输入。
conditioning_scale ( float ) — 指示条件缩放比例。
encoder_hidden_states ( torch.Tensor，形状为 (batch size, sequence len, embed dims)，可选) — 交叉注意力层的条件嵌入。这是 BertModel 的输出。
text_embedding_mask — torch.Tensor。形状为 (batch, key_tokens) 的注意力掩码，应用于 encoder_hidden_states。这是 BertModel 的输出。
encoder_hidden_states_t5 ( torch.Tensor，形状为 (batch size, sequence len, embed dims)，可选) — 交叉注意力层的条件嵌入。这是 T5 文本编码器的输出。
text_embedding_mask_t5 — torch.Tensor。形状为 (batch, key_tokens) 的注意力掩码，应用于 encoder_hidden_states。这是 T5 文本编码器的输出。
image_meta_size (torch.Tensor) — 指示图像大小的条件嵌入
style — torch.Tensor: 指示风格的条件嵌入
image_rotary_emb (torch.Tensor) — 在注意力计算期间应用于查询和键张量的图像旋转嵌入。
return_dict — bool 是否返回字典。

HunyuanDiT2DControlNetModel 的 forward 方法。

set_attn_processor

< source >

( processor: typing.Union[diffusers.models.attention_processor.AttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor2_0, diffusers.models.attention_processor.JointAttnProcessor2_0, diffusers.models.attention_processor.PAGJointAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGJointAttnProcessor2_0, diffusers.models.attention_processor.FusedJointAttnProcessor2_0, diffusers.models.attention_processor.AllegroAttnProcessor2_0, diffusers.models.attention_processor.AuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FusedAuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.CogVideoXAttnProcessor2_0, diffusers.models.attention_processor.FusedCogVideoXAttnProcessor2_0, diffusers.models.attention_processor.XFormersAttnAddedKVProcessor, diffusers.models.attention_processor.XFormersAttnProcessor, diffusers.models.attention_processor.XLAFlashAttnProcessor2_0, diffusers.models.attention_processor.AttnProcessorNPU, diffusers.models.attention_processor.AttnProcessor2_0, diffusers.models.attention_processor.MochiVaeAttnProcessor2_0, diffusers.models.attention_processor.MochiAttnProcessor2_0, diffusers.models.attention_processor.StableAudioAttnProcessor2_0, diffusers.models.attention_processor.HunyuanAttnProcessor2_0, diffusers.models.attention_processor.FusedHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.LuminaAttnProcessor2_0, diffusers.models.attention_processor.FusedAttnProcessor2_0, diffusers.models.attention_processor.CustomDiffusionXFormersAttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor2_0, diffusers.models.attention_processor.SlicedAttnProcessor, diffusers.models.attention_processor.SlicedAttnAddedKVProcessor, diffusers.models.attention_processor.SanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGSanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySanaLinearAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleLinearAttention, diffusers.models.attention_processor.SanaMultiscaleAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleAttentionProjection, diffusers.models.attention_processor.IPAdapterAttnProcessor, diffusers.models.attention_processor.IPAdapterAttnProcessor2_0, diffusers.models.attention_processor.IPAdapterXFormersAttnProcessor, diffusers.models.attention_processor.SD3IPAdapterJointAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.LoRAAttnProcessor, diffusers.models.attention_processor.LoRAAttnProcessor2_0, diffusers.models.attention_processor.LoRAXFormersAttnProcessor, diffusers.models.attention_processor.LoRAAttnAddedKVProcessor, typing.Dict[str, typing.Union[diffusers.models.attention_processor.AttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor2_0, diffusers.models.attention_processor.JointAttnProcessor2_0, diffusers.models.attention_processor.PAGJointAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGJointAttnProcessor2_0, diffusers.models.attention_processor.FusedJointAttnProcessor2_0, diffusers.models.attention_processor.AllegroAttnProcessor2_0, diffusers.models.attention_processor.AuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FusedAuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.CogVideoXAttnProcessor2_0, diffusers.models.attention_processor.FusedCogVideoXAttnProcessor2_0, diffusers.models.attention_processor.XFormersAttnAddedKVProcessor, diffusers.models.attention_processor.XFormersAttnProcessor, diffusers.models.attention_processor.XLAFlashAttnProcessor2_0, diffusers.models.attention_processor.AttnProcessorNPU, diffusers.models.attention_processor.AttnProcessor2_0, diffusers.models.attention_processor.MochiVaeAttnProcessor2_0, diffusers.models.attention_processor.MochiAttnProcessor2_0, diffusers.models.attention_processor.StableAudioAttnProcessor2_0, diffusers.models.attention_processor.HunyuanAttnProcessor2_0, diffusers.models.attention_processor.FusedHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.LuminaAttnProcessor2_0, diffusers.models.attention_processor.FusedAttnProcessor2_0, diffusers.models.attention_processor.CustomDiffusionXFormersAttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor2_0, diffusers.models.attention_processor.SlicedAttnProcessor, diffusers.models.attention_processor.SlicedAttnAddedKVProcessor, diffusers.models.attention_processor.SanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGSanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySanaLinearAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleLinearAttention, diffusers.models.attention_processor.SanaMultiscaleAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleAttentionProjection, diffusers.models.attention_processor.IPAdapterAttnProcessor, diffusers.models.attention_processor.IPAdapterAttnProcessor2_0, diffusers.models.attention_processor.IPAdapterXFormersAttnProcessor, diffusers.models.attention_processor.SD3IPAdapterJointAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.LoRAAttnProcessor, diffusers.models.attention_processor.LoRAAttnProcessor2_0, diffusers.models.attention_processor.LoRAXFormersAttnProcessor, diffusers.models.attention_processor.LoRAAttnAddedKVProcessor]]] )

参数

processor ( AttentionProcessor 的 dict 或仅 AttentionProcessor) — 实例化的处理器类或处理器类字典，它将被设置为所有 Attention 层的处理器。如果 processor 是一个字典，则键需要定义到相应交叉注意力处理器的路径。当设置可训练的注意力处理器时，强烈建议这样做。

设置用于计算注意力的注意力处理器。

< > Update on GitHub

←FluxControlNetModel SD3ControlNetModel→