Diffusers 文档
SD3 Transformer 模型
并获取增强的文档体验
开始使用
SD3 Transformer 模型
在 Stable Diffusion 3 中引入的 Transformer 模型。其新颖之处在于 MMDiT transformer 模块。
SD3Transformer2DModel
class diffusers.SD3Transformer2DModel
< source >( sample_size: int = 128 patch_size: int = 2 in_channels: int = 16 num_layers: int = 18 attention_head_dim: int = 64 num_attention_heads: int = 18 joint_attention_dim: int = 4096 caption_projection_dim: int = 1152 pooled_projection_dim: int = 2048 out_channels: int = 16 pos_embed_max_size: int = 96 dual_attention_layers: typing.Tuple[int, ...] = () qk_norm: typing.Optional[str] = None )
参数
- sample_size (
int) — 潜在图像的宽度。这在训练期间是固定的，因为它用于学习位置嵌入的数量。 - patch_size (
int) — 用于将输入数据转换为小块的块大小。 - in_channels (
int, 可选，默认为 16) — 输入中的通道数。 - num_layers (
int, 可选，默认为 18) — 要使用的 Transformer 块的层数。 - attention_head_dim (
int, 可选，默认为 64) — 每个头部的通道数。 - num_attention_heads (
int, 可选，默认为 18) — 用于多头注意力的头数。 - cross_attention_dim (
int, 可选) — 要使用的encoder_hidden_states维度数。 - caption_projection_dim (
int) — 投影encoder_hidden_states时使用的维度数。 - pooled_projection_dim (
int) — 投影pooled_projections时使用的维度数。 - out_channels (
int, defaults to 16) — 输出通道的数量，默认为 16。
Stable Diffusion 3 中引入的 Transformer 模型。
参考链接: https://arxiv.org/abs/2403.03206
enable_forward_chunking
< source >( chunk_size: typing.Optional[int] = None dim: int = 0 )
设置注意力处理器以使用 前馈分块。
forward
< source >( hidden_states: FloatTensor encoder_hidden_states: FloatTensor = None pooled_projections: FloatTensor = None timestep: LongTensor = None block_controlnet_hidden_states: typing.List = None joint_attention_kwargs: typing.Optional[typing.Dict[str, typing.Any]] = None return_dict: bool = True skip_layers: typing.Optional[typing.List[int]] = None )
参数
- hidden_states (
torch.FloatTensor，形状为(batch size, channel, height, width)) — 输入的hidden_states。 - encoder_hidden_states (
torch.FloatTensor，形状为(batch size, sequence_len, embed_dims)) — 要使用的条件嵌入（从输入条件（如提示）计算出的嵌入）。 - pooled_projections (
torch.FloatTensor，形状为(batch_size, projection_dim)) — 从输入条件的嵌入投影的嵌入。 - timestep (
torch.LongTensor) — 用于指示去噪步骤。 - block_controlnet_hidden_states (
listoftorch.Tensor) — 张量列表，如果指定，则添加到 Transformer 块的残差中。 - joint_attention_kwargs (
dict, optional) — 一个 kwargs 字典，如果指定，则传递给 diffusers.models.attention_processor 中self.processor下定义的AttentionProcessor。 - return_dict (
bool, optional, defaults toTrue) — 是否返回~models.transformer_2d.Transformer2DModelOutput而不是普通元组。 - skip_layers (
listofint, optional) — 在前向传播期间要跳过的层索引列表。
SD3Transformer2DModel 的前向方法。
启用融合 QKV 投影。对于自注意力模块，所有投影矩阵（即，查询、键、值）都融合在一起。对于交叉注意力模块，键和值投影矩阵融合在一起。
此 API 是 🧪 实验性的。
set_attn_processor
< source >( processor: typing.Union[diffusers.models.attention_processor.AttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor, diffusers.models.attention_processor.AttnAddedKVProcessor2_0, diffusers.models.attention_processor.JointAttnProcessor2_0, diffusers.models.attention_processor.PAGJointAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGJointAttnProcessor2_0, diffusers.models.attention_processor.FusedJointAttnProcessor2_0, diffusers.models.attention_processor.AllegroAttnProcessor2_0, diffusers.models.attention_processor.AuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FusedAuraFlowAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0, diffusers.models.attention_processor.FluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0, diffusers.models.attention_processor.FusedFluxAttnProcessor2_0_NPU, diffusers.models.attention_processor.CogVideoXAttnProcessor2_0, diffusers.models.attention_processor.FusedCogVideoXAttnProcessor2_0, diffusers.models.attention_processor.XFormersAttnAddedKVProcessor, diffusers.models.attention_processor.XFormersAttnProcessor, diffusers.models.attention_processor.XLAFlashAttnProcessor2_0, diffusers.models.attention_processor.AttnProcessorNPU, diffusers.models.attention_processor.AttnProcessor2_0, diffusers.models.attention_processor.MochiVaeAttnProcessor2_0, diffusers.models.attention_processor.MochiAttnProcessor2_0, diffusers.models.attention_processor.StableAudioAttnProcessor2_0, diffusers.models.attention_processor.HunyuanAttnProcessor2_0, diffusers.models.attention_processor.FusedHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGHunyuanAttnProcessor2_0, diffusers.models.attention_processor.LuminaAttnProcessor2_0, diffusers.models.attention_processor.FusedAttnProcessor2_0, diffusers.models.attention_processor.CustomDiffusionXFormersAttnProcessor, diffusers.models.attention_processor.CustomDiffusionAttnProcessor2_0, diffusers.models.attention_processor.SlicedAttnProcessor, diffusers.models.attention_processor.SlicedAttnAddedKVProcessor, diffusers.models.attention_processor.SanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGSanaLinearAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySanaLinearAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleLinearAttention, diffusers.models.attention_processor.SanaMultiscaleAttnProcessor2_0, diffusers.models.attention_processor.SanaMultiscaleAttentionProjection, diffusers.models.attention_processor.IPAdapterAttnProcessor, diffusers.models.attention_processor.IPAdapterAttnProcessor2_0, diffusers.models.attention_processor.IPAdapterXFormersAttnProcessor, diffusers.models.attention_processor.SD3IPAdapterJointAttnProcessor2_0, diffusers.models.attention_processor.PAGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.PAGCFGIdentitySelfAttnProcessor2_0, diffusers.models.attention_processor.LoRAAttnProcessor, diffusers.models.attention_processor.LoRAAttnProcessor2_0, diffusers.models.attention_processor.LoRAXFormersAttnProcessor, diffusers.models.attention_processor.LoRAAttnAddedKVProcessor]]] )
设置要用于计算注意力的注意力处理器。