Hugging Face 的 LOGO

Transformers 文档

Longformer

Hugging Face's logo
加入 Hugging Face 社区

并获得增强的文档体验

开始使用

该模型于 2020-04-10 发布，并于 2020-11-16 添加到 Hugging Face Transformers。

PyTorch

Longformer

Longformer 是一个为处理长文档而设计的 transformer 模型。自注意力操作的计算复杂度通常与序列长度呈平方关系，这阻碍了 transformer 处理更长序列的能力。Longformer 的注意力机制通过与序列长度呈线性关系来克服这个问题。它将局部窗口注意力与特定于任务的全局注意力相结合，从而能够高效地处理包含数千个 token 的文档。

你可以在 Ai2 组织下找到所有原始的 Longformer 检查点。

点击右侧边栏中的 Longformer 模型，了解如何将 Longformer 应用于不同语言任务的更多示例。

下面的示例演示了如何使用 Pipeline、AutoModel 以及从命令行来填充 <mask> token。

流水线
自动模型
Transformers CLI
import torch
from transformers import pipeline

pipeline = pipeline(
    task="fill-mask",
    model="allenai/longformer-base-4096",
    dtype=torch.float16,
    device=0
)
pipeline("""San Francisco 49ers cornerback Shawntae Spencer will miss the rest of the <mask> with a torn ligament in his left knee.
Spencer, a fifth-year pro, will be placed on injured reserve soon after undergoing surgery Wednesday to repair the ligament. He injured his knee late in the 49ers’ road victory at Seattle on Sept. 14, and missed last week’s victory over Detroit.
Tarell Brown and Donald Strickland will compete to replace Spencer with the 49ers, who kept 12 defensive backs on their 53-man roster to start the season. Brown, a second-year pro, got his first career interception last weekend while filling in for Strickland, who also sat out with a knee injury.""")

注意事项

  • Longformer 基于 RoBERTa，并且不包含 token_type_ids。你无需指明哪个 token 属于哪个片段。你只需要用分隔符 token </s> 或 tokenizer.sep_token 来分隔这些片段。

  • 在推理时，你可以通过 global_attention_mask 设置哪些 token 可以进行局部注意力，哪些 token 可以进行全局注意力（有关更多详细信息，请参阅此 示例）。值为 0 表示 token 进行局部注意力，值为 1 表示 token 进行全局注意力。

  • LongformerForMaskedLM 的训练方式与 RobertaForMaskedLM 类似，应如下所示使用。

      input_ids = tokenizer.encode("This is a sentence from [MASK] training data", return_tensors="pt")
  mlm_labels = tokenizer.encode("This is a sentence from the training data", return_tensors="pt")
  loss = model(input_ids, labels=input_ids, masked_lm_labels=mlm_labels)[0]

LongformerConfig

class transformers.LongformerConfig

< >

( attention_window: list[int] | int = 512 sep_token_id: int = 2 pad_token_id: int = 1 bos_token_id: int = 0 eos_token_id: int = 2 vocab_size: int = 30522 hidden_size: int = 768 num_hidden_layers: int = 12 num_attention_heads: int = 12 intermediate_size: int = 3072 hidden_act: str = 'gelu' hidden_dropout_prob: float = 0.1 attention_probs_dropout_prob: float = 0.1 max_position_embeddings: int = 512 type_vocab_size: int = 2 initializer_range: float = 0.02 layer_norm_eps: float = 1e-12 onnx_export: bool = False tie_word_embeddings = True **kwargs )

参数

  • vocab_size (int, optional, 默认为 30522) — Longformer 模型词汇表大小。定义调用 LongformerModel 时，inputs_ids 可以表示的不同 token 的数量。
  • hidden_size (int, optional, 默认为 768) — 编码器层和池化层的维度。
  • num_hidden_layers (int, optional, 默认为 12) — Transformer 编码器中的隐藏层数。
  • num_attention_heads (int, optional, 默认为 12) — Transformer 编码器中每个注意力层的注意力头数量。
  • intermediate_size (int, optional, 默认为 3072) — Transformer 编码器中“中间”（通常称为前馈）层的大小。
  • hidden_act (str 或 Callable, optional, 默认为 "gelu") — 编码器和池化层中的非线性激活函数（函数或字符串）。如果是字符串，支持 "gelu", "relu", "silu" 和 "gelu_new"。
  • hidden_dropout_prob (float, optional, 默认为 0.1) — 所有全连接层（嵌入层、编码器和池化层）的丢弃概率。
  • attention_probs_dropout_prob (float, optional, 默认为 0.1) — 注意力概率的丢弃率。
  • max_position_embeddings (int, optional, 默认为 512) — 此模型可能使用的最大序列长度。通常将其设置得足够大以防万一（例如 512、1024 或 2048）。
  • type_vocab_size (int, optional, 默认为 2) — 调用 LongformerModel 时传入的 token_type_ids 的词汇表大小。
  • initializer_range (float, optional, 默认为 0.02) — 用于初始化所有权重矩阵的 truncated_normal_initializer 的标准差。
  • layer_norm_eps (float, optional, 默认为 1e-12) — 层归一化层使用的 epsilon 值。
  • attention_window (int 或 list[int], optional, 默认为 512) — 每个 token 周围的注意力窗口大小。如果是一个 int，则所有层使用相同的尺寸。要为每层指定不同的窗口大小，请使用 list[int]，其中 len(attention_window) == num_hidden_layers。

这是用于存储 LongformerModel 配置的配置类。它用于根据指定的参数实例化 Longformer 模型，定义模型的架构。

这是用于存储 LongformerModel 配置的配置类。它用于根据指定的参数实例化 Longformer 模型，定义模型的架构。使用默认值实例化配置将得到与具有序列长度 4,096 的 LongFormer allenai/longformer-base-4096 架构相似的配置。

配置对象继承自 PreTrainedConfig，可用于控制模型输出。有关更多信息，请阅读 PreTrainedConfig 的文档。

示例

>>> from transformers import LongformerConfig, LongformerModel

>>> # Initializing a Longformer configuration
>>> configuration = LongformerConfig()

>>> # Initializing a model from the configuration
>>> model = LongformerModel(configuration)

>>> # Accessing the model configuration
>>> configuration = model.config

LongformerTokenizer

class transformers.RobertaTokenizer

< >

( vocab: str | dict[str, int] | None = None merges: str | list[str] | None = None errors: str = 'replace' bos_token: str = '<s>' eos_token: str = '</s>' sep_token: str = '</s>' cls_token: str = '<s>' unk_token: str = '<unk>' pad_token: str = '<pad>' mask_token: str = '<mask>' add_prefix_space: bool = False trim_offsets: bool = True **kwargs )

参数

  • vocab (str, dict 或 list, optional) — 自定义词汇表字典。如果未提供，则从 vocab_file 加载词汇表。
  • merges (str 或 list, optional) — 自定义合并列表。如果未提供，则从 merges_file 加载合并。
  • errors (str, optional, 默认为 "replace") — 解码字节为 UTF-8 时遵循的范例。有关更多信息，请参阅 bytes.decode。
  • bos_token (str, optional, 默认为 "<s>") — 在预训练期间使用的起始序列 token。可用于序列分类器 token。

    在构建包含特殊 token 的序列时，这不是用于序列开头的 token。使用的 token 是 cls_token。
  • eos_token (str, optional, 默认为 "</s>") — 序列结束 token。

    在构建包含特殊 token 的序列时，这不是用于序列结束的 token。使用的 token 是 sep_token。
  • sep_token (str, optional, 默认为 "</s>") — 分隔符 token，用于在构建多序列（例如用于序列分类的两个序列或用于问答的文本和问题）时使用。它也用作使用特殊 token 构建的序列的最后一个 token。
  • cls_token (str, optional, 默认为 "<s>") — 分类器 token，在进行序列分类（对整个序列而不是每个 token 进行分类）时使用。它是使用特殊 token 构建的序列的第一个 token。
  • unk_token (str, 可选, 默认为 "<unk>") — 未知标记。词汇表中不存在的标记无法转换为ID，而是被设置为此标记。
  • pad_token (str, 可选, 默认为 "<pad>") — 用于填充的标记，例如在批处理不同长度的序列时使用。
  • mask_token (str, 可选, 默认为 "<mask>") — 用于掩码值（masking values）的标记。这是在使用掩码语言建模训练此模型时使用的标记。这是模型将尝试预测的标记。
  • add_prefix_space (bool, 可选, 默认为 False) — 是否在输入前添加一个空格。这使得将第一个词视为任何其他词一样处理。（RoBERTa tokenizer 通过前导空格检测词语的开始）。
  • trim_offsets (bool, 可选, 默认为 True) — 后处理步骤是否应修剪偏移量以避免包含空格。

Construct a RoBERTa tokenizer (backed by HuggingFace’s tokenizers library). Based on Byte-Pair-Encoding.

这个分词器经过训练，将空格视为词元的一部分（有点像 sentencepiece），所以一个词会

无论是否在句子开头（无空格），编码方式都会不同

>>> from transformers import RobertaTokenizer

>>> tokenizer = RobertaTokenizer.from_pretrained("FacebookAI/roberta-base")
>>> tokenizer("Hello world")["input_ids"]
[0, 31414, 232, 2]

>>> tokenizer(" Hello world")["input_ids"]
[0, 20920, 232, 2]

您可以通过在实例化此分词器时或在对某些文本调用它时传递 add_prefix_space=True 来绕过此行为，但由于模型并非以这种方式进行预训练，这可能会导致性能下降。

当与 is_split_into_words=True 一起使用时，此分词器需要以 add_prefix_space=True 进行实例化。

此分词器继承自 TokenizersBackend，其中包含大部分主要方法。用户应参考此父类了解有关这些方法的更多信息。

LongformerTokenizerFast

class transformers.RobertaTokenizer

< >

( vocab: str | dict[str, int] | None = None merges: str | list[str] | None = None errors: str = 'replace' bos_token: str = '<s>' eos_token: str = '</s>' sep_token: str = '</s>' cls_token: str = '<s>' unk_token: str = '<unk>' pad_token: str = '<pad>' mask_token: str = '<mask>' add_prefix_space: bool = False trim_offsets: bool = True **kwargs )

参数

  • vocab (str, dict 或 list, optional) — 自定义词汇表字典。如果未提供，则从 vocab_file 加载词汇表。
  • merges (str 或 list, optional) — 自定义合并列表。如果未提供，则从 merges_file 加载合并。
  • errors (str, optional, 默认为 "replace") — 解码字节为 UTF-8 时遵循的范例。有关更多信息，请参阅 bytes.decode。
  • bos_token (str, optional, 默认为 "<s>") — 在预训练期间使用的起始序列 token。可用于序列分类器 token。

    在构建包含特殊 token 的序列时，这不是用于序列开头的 token。使用的 token 是 cls_token。
  • eos_token (str, optional, 默认为 "</s>") — 序列结束 token。

    在构建包含特殊 token 的序列时，这不是用于序列结束的 token。使用的 token 是 sep_token。
  • sep_token (str, optional, 默认为 "</s>") — 分隔符 token，用于在构建多序列（例如用于序列分类的两个序列或用于问答的文本和问题）时使用。它也用作使用特殊 token 构建的序列的最后一个 token。
  • cls_token (str, optional, 默认为 "<s>") — 分类器 token，在进行序列分类（对整个序列而不是每个 token 进行分类）时使用。它是使用特殊 token 构建的序列的第一个 token。
  • unk_token (str, 可选, 默认为 "<unk>") — 未知标记。词汇表中不存在的标记无法转换为ID，而是被设置为此标记。
  • pad_token (str, 可选, 默认为 "<pad>") — 用于填充的标记，例如在批处理不同长度的序列时使用。
  • mask_token (str, 可选, 默认为 "<mask>") — 用于掩码值（masking values）的标记。这是在使用掩码语言建模训练此模型时使用的标记。这是模型将尝试预测的标记。
  • add_prefix_space (bool, 可选, 默认为 False) — 是否在输入前添加一个空格。这使得将第一个词视为任何其他词一样处理。（RoBERTa tokenizer 通过前导空格检测词语的开始）。
  • trim_offsets (bool, 可选, 默认为 True) — 后处理步骤是否应修剪偏移量以避免包含空格。

Construct a RoBERTa tokenizer (backed by HuggingFace’s tokenizers library). Based on Byte-Pair-Encoding.

这个分词器经过训练，将空格视为词元的一部分（有点像 sentencepiece），所以一个词会

无论是否在句子开头（无空格），编码方式都会不同

>>> from transformers import RobertaTokenizer

>>> tokenizer = RobertaTokenizer.from_pretrained("FacebookAI/roberta-base")
>>> tokenizer("Hello world")["input_ids"]
[0, 31414, 232, 2]

>>> tokenizer(" Hello world")["input_ids"]
[0, 20920, 232, 2]

您可以通过在实例化此分词器时或在对某些文本调用它时传递 add_prefix_space=True 来绕过此行为，但由于模型并非以这种方式进行预训练，这可能会导致性能下降。

当与 is_split_into_words=True 一起使用时，此分词器需要以 add_prefix_space=True 进行实例化。

此分词器继承自 TokenizersBackend，其中包含大部分主要方法。用户应参考此父类了解有关这些方法的更多信息。

Longformer 特有输出

class transformers.models.longformer.modeling_longformer.LongformerBaseModelOutput

< >

( last_hidden_state: FloatTensor hidden_states: tuple[torch.FloatTensor, ...] | None = None attentions: tuple[torch.FloatTensor, ...] | None = None global_attentions: tuple[torch.FloatTensor, ...] | None = None )

参数

  • last_hidden_state (<class 'torch.FloatTensor'>.last_hidden_state 的形状为 (batch_size, sequence_length, hidden_size)) — 模型最后一层的隐藏状态序列。
  • hidden_states (tuple[torch.FloatTensor, ...] | None.hidden_states, 当传入 output_hidden_states=True 或 config.output_hidden_states=True 时返回) — torch.FloatTensor 的元组（一个用于嵌入输出（如果有嵌入层），加上每个层的输出），形状为 (batch_size, sequence_length, hidden_size)。

    模型每层输出的隐藏状态，以及可选的初始嵌入输出。

  • attentions (tuple(torch.FloatTensor), 可选, 当传入 output_attentions=True 或 config.output_attentions=True 时返回) — torch.FloatTensor 的元组（每个层一个）形状为 (batch_size, num_heads, sequence_length, x + attention_window + 1)，其中 x 是具有全局注意力掩码的 token 数量。

    局部注意力权重在 attention softmax 之后，用于在自注意力头中计算加权平均值。这些是序列中每个 token 对具有全局注意力的每个 token（前 x 个值）以及注意力窗口中每个 token（剩余的 attention_window + 1 个值）的注意力权重。请注意，前 x 个值指的是文本中具有固定位置的 token，但剩余的 attention_window + 1 个值指的是具有相对位置的 token：一个 token 对自身的注意力权重位于索引 x + attention_window / 2 处，前一个（后一个）attention_window / 2 个值是到前一个（后一个）attention_window / 2 个 token 的注意力权重。如果注意力窗口包含具有全局注意力的 token，则相应索引处的注意力权重设置为 0；该值应从前 x 个注意力权重中访问。如果一个 token 具有全局注意力，则 attentions 中所有其他 token 的注意力权重设置为 0，应从 global_attentions 中访问这些值。
  • global_attentions (tuple(torch.FloatTensor), 可选, 当传入 output_attentions=True 或 config.output_attentions=True 时返回) — torch.FloatTensor 的元组（每个层一个）形状为 (batch_size, num_heads, sequence_length, x)，其中 x 是具有全局注意力掩码的 token 数量。

    全局注意力权重在 attention softmax 之后，用于在自注意力头中计算加权平均值。这些是具有全局注意力的每个 token 对序列中每个 token 的注意力权重。

Base class for Longformer’s outputs, with potential hidden states, local and global attentions.

class transformers.models.longformer.modeling_longformer.LongformerBaseModelOutputWithPooling

< >

( last_hidden_state: FloatTensor pooler_output: torch.FloatTensor | None = None hidden_states: tuple[torch.FloatTensor, ...] | None = None attentions: tuple[torch.FloatTensor, ...] | None = None global_attentions: tuple[torch.FloatTensor, ...] | None = None )

参数

  • last_hidden_state (<class 'torch.FloatTensor'>.last_hidden_state 的形状为 (batch_size, sequence_length, hidden_size)) — 模型最后一层的隐藏状态序列。
  • pooler_output (torch.FloatTensor 的形状为 (batch_size, hidden_size)) — 序列中第一个 token（分类 token）的最后一层隐藏状态，经过一个线性层和 Tanh 激活函数处理。线性层权重在预训练期间从下一个句子预测（分类）目标中学习。
  • hidden_states (tuple[torch.FloatTensor, ...] | None.hidden_states, 当传入 output_hidden_states=True 或 config.output_hidden_states=True 时返回) — torch.FloatTensor 的元组（一个用于嵌入输出（如果有嵌入层），加上每个层的输出），形状为 (batch_size, sequence_length, hidden_size)。

    模型每层输出的隐藏状态，以及可选的初始嵌入输出。

  • attentions (tuple(torch.FloatTensor), 可选, 当传入 output_attentions=True 或 config.output_attentions=True 时返回) — torch.FloatTensor 的元组（每个层一个）形状为 (batch_size, num_heads, sequence_length, x + attention_window + 1)，其中 x 是具有全局注意力掩码的 token 数量。

    局部注意力权重在 attention softmax 之后，用于在自注意力头中计算加权平均值。这些是序列中每个 token 对具有全局注意力的每个 token（前 x 个值）以及注意力窗口中每个 token（剩余的 attention_window + 1 个值）的注意力权重。请注意，前 x 个值指的是文本中具有固定位置的 token，但剩余的 attention_window + 1 个值指的是具有相对位置的 token：一个 token 对自身的注意力权重位于索引 x + attention_window / 2 处，前一个（后一个）attention_window / 2 个值是到前一个（后一个）attention_window / 2 个 token 的注意力权重。如果注意力窗口包含具有全局注意力的 token，则相应索引处的注意力权重设置为 0；该值应从前 x 个注意力权重中访问。如果一个 token 具有全局注意力，则 attentions 中所有其他 token 的注意力权重设置为 0，应从 global_attentions 中访问这些值。
  • global_attentions (tuple(torch.FloatTensor), 可选, 当传入 output_attentions=True 或 config.output_attentions=True 时返回) — torch.FloatTensor 的元组（每个层一个）形状为 (batch_size, num_heads, sequence_length, x)，其中 x 是具有全局注意力掩码的 token 数量。

    全局注意力权重在 attention softmax 之后，用于在自注意力头中计算加权平均值。这些是具有全局注意力的每个 token 对序列中每个 token 的注意力权重。

Base class for Longformer’s outputs that also contains a pooling of the last hidden states.

class transformers.models.longformer.modeling_longformer.LongformerMaskedLMOutput

< >

( loss: torch.FloatTensor | None = None logits: torch.FloatTensor | None = None hidden_states: tuple[torch.FloatTensor, ...] | None = None attentions: tuple[torch.FloatTensor, ...] | None = None global_attentions: tuple[torch.FloatTensor, ...] | None = None )

参数

  • loss (torch.FloatTensor of shape (1,), optional, 当提供 labels 时返回) — 掩码语言模型 (MLM) 损失。
  • logits (torch.FloatTensor of shape (batch_size, sequence_length, config.vocab_size)) — 语言模型头的预测分数（SoftMax 前的每个词汇 token 分数）。
  • hidden_states (tuple[torch.FloatTensor, ...] | None.hidden_states, 当传递 output_hidden_states=True 或 config.output_hidden_states=True 时返回) — torch.FloatTensor 的元组（如果模型有嵌入层，则为嵌入输出，加上每个层的输出），形状为 (batch_size, sequence_length, hidden_size)。

    模型在每个层输出处的隐藏状态，以及可选的初始嵌入输出。

  • attentions (tuple(torch.FloatTensor), optional, 当传递 output_attentions=True 或 config.output_attentions=True 时返回) — torch.FloatTensor 的元组（每个层一个），形状为 (batch_size, num_heads, sequence_length, x + attention_window + 1)，其中 x 是具有全局注意力掩码的 token 数量。

    注意力 softmax 之后的局部注意力权重，用于计算自注意力头中的加权平均。这些是序列中每个 token 到具有全局注意力的每个 token（前 x 个值）以及注意力窗口中的每个 token（剩余 attention_window + 1 个值）的注意力权重。请注意，前 x 个值是指文本中具有固定位置的 token，而剩余的 attention_window + 1 个值是指具有相对位置的 token：一个 token 到自身的注意力权重位于索引 x + attention_window / 2，而前面（后面）attention_window / 2 个值是到前面（后面）attention_window / 2 个 token 的注意力权重。如果注意力窗口包含具有全局注意力的 token，则相应索引处的注意力权重设置为 0；应从前 x 个注意力权重中访问该值。如果一个 token 具有全局注意力，那么到 attentions 中所有其他 token 的注意力权重将设置为 0，应从 global_attentions 中访问这些值。

  • global_attentions (tuple(torch.FloatTensor), optional, 当传递 output_attentions=True 或 config.output_attentions=True 时返回) — torch.FloatTensor 的元组（每个层一个），形状为 (batch_size, num_heads, sequence_length, x)，其中 x 是具有全局注意力掩码的 token 数量。

    注意力 softmax 之后的全局注意力权重，用于计算自注意力头中的加权平均。这些是具有全局注意力的每个 token 到序列中每个 token 的注意力权重。

    • 掩码语言模型输出的基类。

    class transformers.models.longformer.modeling_longformer.LongformerQuestionAnsweringModelOutput

    < >

    ( loss: torch.FloatTensor | None = None start_logits: torch.FloatTensor | None = None end_logits: torch.FloatTensor | None = None hidden_states: tuple[torch.FloatTensor, ...] | None = None attentions: tuple[torch.FloatTensor, ...] | None = None global_attentions: tuple[torch.FloatTensor, ...] | None = None )

    参数

    • loss (torch.FloatTensor of shape (1,), optional, 当提供 labels 时返回) — 总跨度提取损失是开始和结束位置的交叉熵之和。
    • start_logits (torch.FloatTensor | None.start_logits of shape (batch_size, sequence_length), defaults to None) — 跨度开始分数（SoftMax 前）。
    • end_logits (torch.FloatTensor | None.end_logits of shape (batch_size, sequence_length), defaults to None) — 跨度结束分数（SoftMax 前）。
    • hidden_states (tuple[torch.FloatTensor, ...] | None.hidden_states, 当传递 output_hidden_states=True 或 config.output_hidden_states=True 时返回) — torch.FloatTensor 的元组（如果模型有嵌入层，则为嵌入输出，加上每个层的输出），形状为 (batch_size, sequence_length, hidden_size)。

      模型在每个层输出处的隐藏状态，以及可选的初始嵌入输出。

    • attentions (tuple(torch.FloatTensor), optional, 当传递 output_attentions=True 或 config.output_attentions=True 时返回) — torch.FloatTensor 的元组（每个层一个），形状为 (batch_size, num_heads, sequence_length, x + attention_window + 1)，其中 x 是具有全局注意力掩码的 token 数量。

      注意力 softmax 之后的局部注意力权重，用于计算自注意力头中的加权平均。这些是序列中每个 token 到具有全局注意力的每个 token（前 x 个值）以及注意力窗口中的每个 token（剩余 attention_window + 1 个值）的注意力权重。请注意，前 x 个值是指文本中具有固定位置的 token，而剩余的 attention_window + 1 个值是指具有相对位置的 token：一个 token 到自身的注意力权重位于索引 x + attention_window / 2，而前面（后面）attention_window / 2 个值是到前面（后面）attention_window / 2 个 token 的注意力权重。如果注意力窗口包含具有全局注意力的 token，则相应索引处的注意力权重设置为 0；应从前 x 个注意力权重中访问该值。如果一个 token 具有全局注意力，那么到 attentions 中所有其他 token 的注意力权重将设置为 0，应从 global_attentions 中访问这些值。

  • global_attentions (tuple(torch.FloatTensor), optional, 当传递 output_attentions=True 或 config.output_attentions=True 时返回) — torch.FloatTensor 的元组（每个层一个），形状为 (batch_size, num_heads, sequence_length, x)，其中 x 是具有全局注意力掩码的 token 数量。

    注意力 softmax 之后的全局注意力权重，用于计算自注意力头中的加权平均。这些是具有全局注意力的每个 token 到序列中每个 token 的注意力权重。

    • Longformer 模型输出的基类。

    class transformers.models.longformer.modeling_longformer.LongformerSequenceClassifierOutput

    < >

    ( loss: torch.FloatTensor | None = None logits: torch.FloatTensor | None = None hidden_states: tuple[torch.FloatTensor, ...] | None = None attentions: tuple[torch.FloatTensor, ...] | None = None global_attentions: tuple[torch.FloatTensor, ...] | None = None )

    参数

    • loss (torch.FloatTensor of shape (1,), optional, 当提供 labels 时返回) — 分类（或回归，如果 config.num_labels==1）损失。
    • logits (torch.FloatTensor of shape (batch_size, config.num_labels)) — 分类（或回归，如果 config.num_labels==1）分数（SoftMax 前）。
    • hidden_states (tuple[torch.FloatTensor, ...] | None.hidden_states, 当传递 output_hidden_states=True 或 config.output_hidden_states=True 时返回) — torch.FloatTensor 的元组（如果模型有嵌入层，则为嵌入输出，加上每个层的输出），形状为 (batch_size, sequence_length, hidden_size)。

      模型在每个层输出处的隐藏状态，以及可选的初始嵌入输出。

    • attentions (tuple(torch.FloatTensor), optional, 当传递 output_attentions=True 或 config.output_attentions=True 时返回) — torch.FloatTensor 的元组（每个层一个），形状为 (batch_size, num_heads, sequence_length, x + attention_window + 1)，其中 x 是具有全局注意力掩码的 token 数量。

      注意力 softmax 之后的局部注意力权重，用于计算自注意力头中的加权平均。这些是序列中每个 token 到具有全局注意力的每个 token（前 x 个值）以及注意力窗口中的每个 token（剩余 attention_window + 1 个值）的注意力权重。请注意，前 x 个值是指文本中具有固定位置的 token，而剩余的 attention_window + 1 个值是指具有相对位置的 token：一个 token 到自身的注意力权重位于索引 x + attention_window / 2，而前面（后面）attention_window / 2 个值是到前面（后面）attention_window / 2 个 token 的注意力权重。如果注意力窗口包含具有全局注意力的 token，则相应索引处的注意力权重设置为 0；应从前 x 个注意力权重中访问该值。如果一个 token 具有全局注意力，那么到 attentions 中所有其他 token 的注意力权重将设置为 0，应从 global_attentions 中访问这些值。

  • global_attentions (tuple(torch.FloatTensor), optional, 当传递 output_attentions=True 或 config.output_attentions=True 时返回) — torch.FloatTensor 的元组（每个层一个），形状为 (batch_size, num_heads, sequence_length, x)，其中 x 是具有全局注意力掩码的 token 数量。

    注意力 softmax 之后的全局注意力权重，用于计算自注意力头中的加权平均。这些是具有全局注意力的每个 token 到序列中每个 token 的注意力权重。

    • 句子分类模型输出的基类。

    class transformers.models.longformer.modeling_longformer.LongformerMultipleChoiceModelOutput

    < >

    ( loss: torch.FloatTensor | None = None logits: torch.FloatTensor | None = None hidden_states: tuple[torch.FloatTensor, ...] | None = None attentions: tuple[torch.FloatTensor, ...] | None = None global_attentions: tuple[torch.FloatTensor, ...] | None = None )

    参数

    • loss (torch.FloatTensor of shape (1,), optional, 当提供 labels 时返回) — 分类损失。
    • logits (torch.FloatTensor of shape (batch_size, num_choices)) — num_choices 是输入张量的第二个维度。（见上面的 input_ids）。

      分类分数（SoftMax 前）。

    • hidden_states (tuple[torch.FloatTensor, ...] | None.hidden_states, 当传递 output_hidden_states=True 或 config.output_hidden_states=True 时返回) — torch.FloatTensor 的元组（如果模型有嵌入层，则为嵌入输出，加上每个层的输出），形状为 (batch_size, sequence_length, hidden_size)。

      模型在每个层输出处的隐藏状态，以及可选的初始嵌入输出。

    • attentions (tuple(torch.FloatTensor), optional, 当传递 output_attentions=True 或 config.output_attentions=True 时返回) — torch.FloatTensor 的元组（每个层一个），形状为 (batch_size, num_heads, sequence_length, x + attention_window + 1)，其中 x 是具有全局注意力掩码的 token 数量。

      注意力 softmax 之后的局部注意力权重，用于计算自注意力头中的加权平均。这些是序列中每个 token 到具有全局注意力的每个 token（前 x 个值）以及注意力窗口中的每个 token（剩余 attention_window + 1 个值）的注意力权重。请注意，前 x 个值是指文本中具有固定位置的 token，而剩余的 attention_window + 1 个值是指具有相对位置的 token：一个 token 到自身的注意力权重位于索引 x + attention_window / 2，而前面（后面）attention_window / 2 个值是到前面（后面）attention_window / 2 个 token 的注意力权重。如果注意力窗口包含具有全局注意力的 token，则相应索引处的注意力权重设置为 0；应从前 x 个注意力权重中访问该值。如果一个 token 具有全局注意力，那么到 attentions 中所有其他 token 的注意力权重将设置为 0，应从 global_attentions 中访问这些值。

  • global_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x), where x is the number of tokens with global attention mask.

    全局注意力权重，在注意力 Softmax 之后，用于计算自注意力头中的加权平均。这些是每个具有全局注意力的 token 对序列中所有 token 的注意力权重。

    • 多选 Longformer 模型的基类。

    class transformers.models.longformer.modeling_longformer.LongformerTokenClassifierOutput

    < >

    ( loss: torch.FloatTensor | None = None logits: torch.FloatTensor | None = None hidden_states: tuple[torch.FloatTensor, ...] | None = None attentions: tuple[torch.FloatTensor, ...] | None = None global_attentions: tuple[torch.FloatTensor, ...] | None = None )

    参数

    • loss (torch.FloatTensor of shape (1,), optional, returned when labels is provided) — 分类损失。
    • logits (torch.FloatTensor of shape (batch_size, sequence_length, config.num_labels)) — 分类得分（SoftMax 之前）。
    • hidden_states (tuple[torch.FloatTensor, ...] | None.hidden_states, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) — Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + one for the output of each layer) of shape (batch_size, sequence_length, hidden_size).

      模型在每一层输出以及可选的初始嵌入输出后的隐藏状态。

    • attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x + attention_window + 1), where x is the number of tokens with global attention mask.

      局部注意力权重，在注意力 Softmax 之后，用于计算自注意力头中的加权平均。这些是序列中每个 token 对每个具有全局注意力的 token（前 x 个值）以及注意力窗口中的每个 token（剩余 `attention_window

      • 1个值）的注意力权重。注意，前x个值指的是文本中固定位置的 token，但剩余的attention_window + 1个值指的是相对位置的 token：一个 token 对自身的注意力权重位于索引x + attention_window / 2处，并且前面的（后面的）attention_window / 2个值是到前面（后面）attention_window / 2个 token 的注意力权重。如果注意力窗口包含一个具有全局注意力的 token，则相应索引处的注意力权重设置为 0；应从前x个注意力权重中访问该值。如果一个 token 具有全局注意力，则其对attentions中所有其他 token 的注意力权重设置为 0，应从global_attentions中访问该值。
    • global_attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x), where x is the number of tokens with global attention mask.

      全局注意力权重，在注意力 Softmax 之后，用于计算自注意力头中的加权平均。这些是每个具有全局注意力的 token 对序列中所有 token 的注意力权重。

    Token分类模型输出的基类。

    LongformerModel

    class transformers.LongformerModel

    < >

    ( config add_pooling_layer = True )

    参数

    • config (LongformerModel) — 模型配置类，包含模型的所有参数。仅加载配置，不加载与模型相关的权重。请查看 from_pretrained() 方法加载模型权重。
    • add_pooling_layer (bool, optional, defaults to True) — 是否添加池化层

    Bare Longformer 模型，输出原始隐藏状态，不带任何特定头。

    此模型继承自 PreTrainedModel。查看其父类文档，了解库为所有模型实现的通用方法（例如下载或保存、调整输入嵌入大小、修剪头等）。

    此模型也是一个 PyTorch torch.nn.Module 子类。像普通的 PyTorch Module 一样使用它，并参考 PyTorch 文档了解一般用法和行为的所有相关信息。

    forward

    < >

    ( input_ids: torch.Tensor | None = None attention_mask: torch.Tensor | None = None global_attention_mask: torch.Tensor | None = None token_type_ids: torch.Tensor | None = None position_ids: torch.Tensor | None = None inputs_embeds: torch.Tensor | None = None output_attentions: bool | None = None output_hidden_states: bool | None = None return_dict: bool | None = None **kwargs ) → transformers.models.longformer.modeling_longformer.LongformerBaseModelOutputWithPooling or tuple(torch.FloatTensor)

    参数

    • input_ids (torch.Tensor of shape (batch_size, sequence_length), optional) — Token 序列在词汇表中的索引。默认情况下会忽略填充。

      可以使用 AutoTokenizer 获取索引。有关详细信息，请参阅 PreTrainedTokenizer.encode() 和 PreTrainedTokenizer.call()。

      输入 ID 是什么？

    • attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:

      • 1 for tokens that are not masked,
      • 0 for tokens that are masked.

      注意掩码是什么？

    • global_attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) — Mask to decide the attention given on each token, local attention or global attention. Tokens with global attention attends to all other tokens, and all other tokens attend to them. This is important for task-specific finetuning because it makes the model more flexible at representing the task. For example, for classification, the token should be given global attention. For QA, all question tokens should also have global attention. Please refer to the Longformer paper for more details. Mask values selected in [0, 1]:

      • 0 for local attention (a sliding window attention),
      • 1 for global attention (tokens that attend to all other tokens, and all other tokens attend to them).
    • token_type_ids (torch.Tensor of shape (batch_size, sequence_length), optional) — 用于指示输入的第一和第二部分的分段 token 索引。索引选择在 [0, 1] 之间：

      • 0 对应于句子 A token，
      • 1 对应于句子 B token。

      Token 类型 ID 是什么？

    • position_ids (torch.Tensor of shape (batch_size, sequence_length), optional) — 序列中每个 token 在位置嵌入中的位置索引。选择范围为 [0, config.n_positions - 1]。

      位置 ID 是什么？

    • inputs_embeds (torch.Tensor of shape (batch_size, sequence_length, hidden_size), optional) — 可选地，您可以直接传入嵌入表示，而不是 input_ids。如果您想比模型内部的嵌入查找矩阵获得对 input_ids 索引到关联向量的转换方式更多的控制，这将很有用。
    • output_attentions (bool, optional) — 是否返回所有注意力层的注意力张量。有关更多详细信息，请参见返回张量下的 attentions。
    • output_hidden_states (bool, optional) — 是否返回所有层的隐藏状态。有关更多详细信息，请参见返回张量下的 hidden_states。
    • return_dict (bool, optional) — 是否返回一个 ModelOutput 而不是一个普通的元组。

    返回

    transformers.models.longformer.modeling_longformer.LongformerBaseModelOutputWithPooling or tuple(torch.FloatTensor)

    一个 transformers.models.longformer.modeling_longformer.LongformerBaseModelOutputWithPooling 或一个 tuple(torch.FloatTensor)（如果传递了 return_dict=False 或当 config.return_dict=False 时），包含根据配置（LongformerConfig）和输入而变化的各种元素。

    • last_hidden_state (<class 'torch.FloatTensor'>.last_hidden_state of shape (batch_size, sequence_length, hidden_size)) — 模型最后一层输出的隐藏状态序列。

    • pooler_output (torch.FloatTensor of shape (batch_size, hidden_size)) — 序列的第一个 token（分类 token）的最后一层隐藏状态，经过线性层和 Tanh 激活函数进一步处理。线性层权重在预训练期间通过下一句预测（分类）目标进行训练。

    • hidden_states (tuple[torch.FloatTensor, ...] | None.hidden_states, 当传入 output_hidden_states=True 或当 config.output_hidden_states=True 时返回) — 形状为 (batch_size, sequence_length, hidden_size) 的 torch.FloatTensor 元组（一个用于嵌入层的输出，如果模型有嵌入层，+ 每个层的输出）。

      模型在每个层输出的隐藏状态以及可选的初始嵌入输出。

    • attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x + attention_window + 1), where x is the number of tokens with global attention mask.

      局部注意力权重，在注意力 Softmax 之后，用于计算自注意力头中的加权平均。这些是序列中每个 token 对每个具有全局注意力的 token（前 x 个值）以及注意力窗口中的每个 token（剩余 `attention_window

      • 1值）。请注意，前x个值指的是具有固定文本位置的 token，而其余attention_window + 1个值指的是具有相对位置的 token：一个 token 对自身的注意力权重位于索引x + attention_window / 2处，而前面（后面）的attention_window / 2个值是注意力权重对前面（后面）attention_window / 2个 token 的注意力权重。如果注意力窗口包含一个具有全局注意力的 token，则相应索引处的注意力权重设置为 0；该值应从前x个注意力权重中访问。如果一个 token 具有全局注意力，则其在attentions中对所有其他 token 的注意力权重均设置为 0，该值应从global_attentions` 中访问。

    • global_attentions (tuple(torch.FloatTensor), 可选, 当传递 output_attentions=True 或 config.output_attentions=True 时返回) — 形状为 (batch_size, num_heads, sequence_length, x) 的 torch.FloatTensor 元组（每个层一个），其中 x 是具有全局注意力掩码的 token 数量。

      注意力 softmax 之后的全局注意力权重，用于计算自注意力头中的加权平均值。这些是每个具有全局注意力的 token 对序列中每个 token 的注意力权重。

    LongformerModel 的 forward 方法，覆盖了 __call__ 特殊方法。

    虽然 forward pass 的实现需要在此函数中定义，但你应该在之后调用 Module 实例而不是这个，因为前者负责运行预处理和后处理步骤，而后者会静默地忽略它们。

    示例

    >>> import torch
>>> from transformers import LongformerModel, AutoTokenizer

>>> model = LongformerModel.from_pretrained("allenai/longformer-base-4096")
>>> tokenizer = AutoTokenizer.from_pretrained("allenai/longformer-base-4096")

>>> SAMPLE_TEXT = " ".join(["Hello world! "] * 1000)  # long input document
>>> input_ids = torch.tensor(tokenizer.encode(SAMPLE_TEXT)).unsqueeze(0)  # batch of size 1

>>> attention_mask = torch.ones(
...     input_ids.shape, dtype=torch.long, device=input_ids.device
... )  # initialize to local attention
>>> global_attention_mask = torch.zeros(
...     input_ids.shape, dtype=torch.long, device=input_ids.device
... )  # initialize to global attention to be deactivated for all tokens
>>> global_attention_mask[
...     :,
...     [
...         1,
...         4,
...         21,
...     ],
... ] = 1  # Set global attention to random tokens for the sake of this example
>>> # Usually, set global attention based on the task. For example,
>>> # classification: the <s> token
>>> # QA: question tokens
>>> # LM: potentially on the beginning of sentences and paragraphs
>>> outputs = model(input_ids, attention_mask=attention_mask, global_attention_mask=global_attention_mask)
>>> sequence_output = outputs.last_hidden_state
>>> pooled_output = outputs.pooler_output

    LongformerForMaskedLM

    class transformers.LongformerForMaskedLM

    < >

    ( config model_args: ~utils.generic.ModelArgs | None = None adapter_args: ~utils.generic.AdapterArgs | None = None lora_args: ~utils.generic.LoRAArgs | None = None tokenizer_args: ~utils.generic.TokenizerArgs | None = None dataset_args: ~utils.generic.DatasetArgs | None = None data_args: ~utils.generic.DataArgs | None = None training_args: ~utils.generic.TrainingArgs | None = None generation_args: ~utils.generic.GenerationArgs | None = None vision_tower_args: ~utils.generic.VisionTowerArgs | None = None qlora_args: ~utils.generic.QLoRAArgs | None = None vision_tower_template_args: ~utils.generic.VisionTowerTemplateArgs | None = None video_tower_args: ~utils.generic.VideoTowerArgs | None = None vision_config: ~utils.generic.VisionConfig | None = None video_config: ~utils.generic.VideoConfig | None = None load_dataset: bool | None = None load_data_collator: bool | None = None load_processor: bool | None = None load_lora_adapter: bool | None = None load_adapter: bool | None = None load_qlora_adapter: bool | None = None **kwargs: typing_extensions.Unpack[transformers.modeling_utils.PreTrainedModelKwargs] )

    参数

    • config (LongformerForMaskedLM) — 模型配置类，包含模型的所有参数。使用配置文件初始化不会加载与模型相关的权重，只加载配置。请查看 from_pretrained() 方法以加载模型权重。

    顶部带有 语言建模 头的 Longformer 模型。

    此模型继承自 PreTrainedModel。查看其父类文档，了解库为所有模型实现的通用方法（例如下载或保存、调整输入嵌入大小、修剪头等）。

    此模型也是一个 PyTorch torch.nn.Module 子类。像普通的 PyTorch Module 一样使用它，并参考 PyTorch 文档了解一般用法和行为的所有相关信息。

    forward

    < >

    ( input_ids: torch.Tensor | None = None attention_mask: torch.Tensor | None = None global_attention_mask: torch.Tensor | None = None token_type_ids: torch.Tensor | None = None position_ids: torch.Tensor | None = None inputs_embeds: torch.Tensor | None = None labels: torch.Tensor | None = None output_attentions: bool | None = None output_hidden_states: bool | None = None return_dict: bool | None = None **kwargs ) → transformers.models.longformer.modeling_longformer.LongformerMaskedLMOutput 或 tuple(torch.FloatTensor)

    参数

    • input_ids (形状为 (batch_size, sequence_length) 的 torch.Tensor, 可选) — 词汇表中输入序列 token 的索引。默认会忽略填充。

      可以使用 AutoTokenizer 获取索引。有关详细信息，请参阅 PreTrainedTokenizer.encode() 和 PreTrainedTokenizer.call()。

      什么是输入 ID？

    • attention_mask (形状为 (batch_size, sequence_length) 的 torch.Tensor, 可选) — 用于避免对填充 token 索引执行注意力操作的掩码。掩码值选择在 [0, 1] 中：

      • 1 表示未被掩码的 token，
      • 0 表示被掩码的 token。

      什么是注意力掩码？

    • global_attention_mask (形状为 (batch_size, sequence_length) 的 torch.FloatTensor, 可选) — 决定每个 token 的注意力（局部注意力或全局注意力）的掩码。具有全局注意力的 token 会关注所有其他 token，而所有其他 token 也会关注它们。这对于特定任务的微调很重要，因为它使模型在表示任务方面更具灵活性。例如，对于分类任务， token 应具有全局注意力。对于 QA 任务，所有问题 token 也应具有全局注意力。有关更多详细信息，请参阅 Longformer 论文。掩码值选择在 [0, 1] 中：

      • 0 表示局部注意力（滑动窗口注意力），
      • 1 表示全局注意力（关注所有其他 token 的 token，以及所有其他 token 关注它们的 token）。
    • token_type_ids (形状为 (batch_size, sequence_length) 的 torch.Tensor, 可选) — 用于指示输入第一部分和第二部分的 segment token 索引。索引选择在 [0, 1] 中：

      • 0 对应于句子 A 的 token，
      • 1 对应于句子 B 的 token。

      什么是 token 类型 ID？

    • position_ids (形状为 (batch_size, sequence_length) 的 torch.Tensor, 可选) — 输入序列 token 在位置嵌入中的索引。选择范围为 [0, config.n_positions - 1]。

      什么是位置 ID？

    • inputs_embeds (形状为 (batch_size, sequence_length, hidden_size) 的 torch.Tensor, 可选) — 可选地，您可以直接传入嵌入表示，而不是传入 input_ids。如果您想更精细地控制如何将 input_ids 索引转换为关联向量，而不是使用模型内部的嵌入查找矩阵，这将非常有用。
    • labels (形状为 (batch_size, sequence_length) 的 torch.LongTensor, 可选) — 用于计算掩码语言模型损失的标签。索引应在 [-100, 0, ..., config.vocab_size] 范围内（参见 input_ids 文档字符串）。索引设置为 -100 的 token 将被忽略（掩码），损失仅为具有 [0, ..., config.vocab_size] 范围内标签的 token 计算。
    • output_attentions (bool, 可选) — 是否返回所有注意力层的注意力张量。有关详细信息，请参阅返回张量下的 attentions。
    • output_hidden_states (bool, 可选) — 是否返回所有层的隐藏状态。有关详细信息，请参阅返回张量下的 hidden_states。
    • return_dict (bool, 可选) — 是否返回一个 ModelOutput 而不是一个普通元组。

    返回

    transformers.models.longformer.modeling_longformer.LongformerMaskedLMOutput 或 tuple(torch.FloatTensor)

    根据配置（LongformerConfig）和输入，返回一个 transformers.models.longformer.modeling_longformer.LongformerMaskedLMOutput 或一个 torch.FloatTensor 元组（如果传递了 return_dict=False 或 config.return_dict=False），其中包含各种元素。

    • loss (形状为 (1,) 的 torch.FloatTensor，可选，当提供 labels 时返回) — 掩码语言建模 (MLM) 损失。

    • logits (形状为 (batch_size, sequence_length, config.vocab_size) 的 torch.FloatTensor) — 语言建模头部的预测分数（SoftMax 之前的每个词汇标记的分数）。

    • hidden_states (tuple[torch.FloatTensor, ...] | None.hidden_states, 当传入 output_hidden_states=True 或当 config.output_hidden_states=True 时返回) — 形状为 (batch_size, sequence_length, hidden_size) 的 torch.FloatTensor 元组（一个用于嵌入层的输出，如果模型有嵌入层，+ 每个层的输出）。

      模型在每个层输出的隐藏状态以及可选的初始嵌入输出。

    • attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x + attention_window + 1), where x is the number of tokens with global attention mask.

      局部注意力权重，在注意力 Softmax 之后，用于计算自注意力头中的加权平均。这些是序列中每个 token 对每个具有全局注意力的 token（前 x 个值）以及注意力窗口中的每个 token（剩余 `attention_window

      • 1值）。请注意，前x个值指的是具有固定文本位置的 token，而其余attention_window + 1个值指的是具有相对位置的 token：一个 token 对自身的注意力权重位于索引x + attention_window / 2处，而前面（后面）的attention_window / 2个值是注意力权重对前面（后面）attention_window / 2个 token 的注意力权重。如果注意力窗口包含一个具有全局注意力的 token，则相应索引处的注意力权重设置为 0；该值应从前x个注意力权重中访问。如果一个 token 具有全局注意力，则其在attentions中对所有其他 token 的注意力权重均设置为 0，该值应从global_attentions` 中访问。

    • global_attentions (tuple(torch.FloatTensor), 可选, 当传递 output_attentions=True 或 config.output_attentions=True 时返回) — 形状为 (batch_size, num_heads, sequence_length, x) 的 torch.FloatTensor 元组（每个层一个），其中 x 是具有全局注意力掩码的 token 数量。

      注意力 softmax 之后的全局注意力权重，用于计算自注意力头中的加权平均值。这些是每个具有全局注意力的 token 对序列中每个 token 的注意力权重。

    LongformerForMaskedLM 的 forward 方法，覆盖了 __call__ 特殊方法。

    虽然 forward pass 的实现需要在此函数中定义，但你应该在之后调用 Module 实例而不是这个，因为前者负责运行预处理和后处理步骤，而后者会静默地忽略它们。

    示例：掩码填充

    >>> from transformers import AutoTokenizer, LongformerForMaskedLM

>>> tokenizer = AutoTokenizer.from_pretrained("allenai/longformer-base-4096")
>>> model = LongformerForMaskedLM.from_pretrained("allenai/longformer-base-4096")

    让我们尝试一个非常长的输入。

    >>> TXT = (
...     "My friends are <mask> but they eat too many carbs."
...     + " That's why I decide not to eat with them." * 300
... )
>>> input_ids = tokenizer([TXT], return_tensors="pt")["input_ids"]
>>> logits = model(input_ids).logits

>>> masked_index = (input_ids[0] == tokenizer.mask_token_id).nonzero().item()
>>> probs = logits[0, masked_index].softmax(dim=0)
>>> values, predictions = probs.topk(5)

>>> tokenizer.decode(predictions).split()
['healthy', 'skinny', 'thin', 'good', 'vegetarian']

    LongformerForSequenceClassification

    class transformers.LongformerForSequenceClassification

    < >

    ( config model_args: ~utils.generic.ModelArgs | None = None adapter_args: ~utils.generic.AdapterArgs | None = None lora_args: ~utils.generic.LoRAArgs | None = None tokenizer_args: ~utils.generic.TokenizerArgs | None = None dataset_args: ~utils.generic.DatasetArgs | None = None data_args: ~utils.generic.DataArgs | None = None training_args: ~utils.generic.TrainingArgs | None = None generation_args: ~utils.generic.GenerationArgs | None = None vision_tower_args: ~utils.generic.VisionTowerArgs | None = None qlora_args: ~utils.generic.QLoRAArgs | None = None vision_tower_template_args: ~utils.generic.VisionTowerTemplateArgs | None = None video_tower_args: ~utils.generic.VideoTowerArgs | None = None vision_config: ~utils.generic.VisionConfig | None = None video_config: ~utils.generic.VideoConfig | None = None load_dataset: bool | None = None load_data_collator: bool | None = None load_processor: bool | None = None load_lora_adapter: bool | None = None load_adapter: bool | None = None load_qlora_adapter: bool | None = None **kwargs: typing_extensions.Unpack[transformers.modeling_utils.PreTrainedModelKwargs] )

    参数

    • config (LongformerForSequenceClassification) — 模型配置类，包含模型的所有参数。使用配置文件初始化不会加载与模型相关的权重，只加载配置。请查看 from_pretrained() 方法以加载模型权重。

    顶部带有序列分类/回归头的 Longformer Model transformer（在池化输出之上是一个线性层），例如用于 GLUE 任务。

    此模型继承自 PreTrainedModel。查看其父类文档，了解库为所有模型实现的通用方法（例如下载或保存、调整输入嵌入大小、修剪头等）。

    此模型也是一个 PyTorch torch.nn.Module 子类。像普通的 PyTorch Module 一样使用它，并参考 PyTorch 文档了解一般用法和行为的所有相关信息。

    forward

    < >

    ( input_ids: torch.Tensor | None = None attention_mask: torch.Tensor | None = None global_attention_mask: torch.Tensor | None = None token_type_ids: torch.Tensor | None = None position_ids: torch.Tensor | None = None inputs_embeds: torch.Tensor | None = None labels: torch.Tensor | None = None output_attentions: bool | None = None output_hidden_states: bool | None = None return_dict: bool | None = None **kwargs ) → transformers.models.longformer.modeling_longformer.LongformerSequenceClassifierOutput 或 tuple(torch.FloatTensor)

    参数

    • input_ids (形状为 (batch_size, sequence_length) 的 torch.Tensor, 可选) — 词汇表中输入序列 token 的索引。默认会忽略填充。

      可以使用 AutoTokenizer 获取索引。有关详细信息，请参阅 PreTrainedTokenizer.encode() 和 PreTrainedTokenizer.call()。

      什么是输入 ID？

    • attention_mask (形状为 (batch_size, sequence_length) 的 torch.Tensor, 可选) — 用于避免对填充 token 索引执行注意力操作的掩码。掩码值选择在 [0, 1] 中：

      • 1 表示未被掩码的 token，
      • 0 表示被掩码的 token。

      什么是注意力掩码？

    • global_attention_mask (形状为 (batch_size, sequence_length) 的 torch.FloatTensor, 可选) — 决定每个 token 的注意力（局部注意力或全局注意力）的掩码。具有全局注意力的 token 会关注所有其他 token，而所有其他 token 也会关注它们。这对于特定任务的微调很重要，因为它使模型在表示任务方面更具灵活性。例如，对于分类任务， token 应具有全局注意力。对于 QA 任务，所有问题 token 也应具有全局注意力。有关更多详细信息，请参阅 Longformer 论文。掩码值选择在 [0, 1] 中：

      • 0 表示局部注意力（滑动窗口注意力），
      • 1 表示全局注意力（关注所有其他 token 的 token，以及所有其他 token 关注它们的 token）。
    • token_type_ids (形状为 (batch_size, sequence_length) 的 torch.Tensor, 可选) — 用于指示输入第一部分和第二部分的 segment token 索引。索引选择在 [0, 1] 中：

      • 0 对应于句子 A 的 token，
      • 1 对应于句子 B 的 token。

      什么是 token 类型 ID？

    • position_ids (形状为 (batch_size, sequence_length) 的 torch.Tensor, 可选) — 输入序列 token 在位置嵌入中的索引。选择范围为 [0, config.n_positions - 1]。

      什么是位置 ID？

    • inputs_embeds (torch.Tensor of shape (batch_size, sequence_length, hidden_size), optional) — Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert input_ids indices into associated vectors than the model’s internal embedding lookup matrix.
    • labels (torch.LongTensor of shape (batch_size,), optional) — Labels for computing the sequence classification/regression loss. Indices should be in [0, ..., config.num_labels - 1]. If config.num_labels == 1 a regression loss is computed (Mean-Square loss), If config.num_labels > 1 a classification loss is computed (Cross-Entropy).
    • output_attentions (bool, optional) — Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail.
    • output_hidden_states (bool, optional) — Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail.
    • return_dict (bool, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

    返回

    transformers.models.longformer.modeling_longformer.LongformerSequenceClassifierOutput or tuple(torch.FloatTensor)

    transformers.models.longformer.modeling_longformer.LongformerSequenceClassifierOutput or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (LongformerConfig) and inputs.

    • loss (形状为 (1,) 的 torch.FloatTensor，可选，当提供 labels 时返回) — 分类损失（如果 config.num_labels==1，则为回归损失）。

    • logits (形状为 (batch_size, config.num_labels) 的 torch.FloatTensor) — 分类（如果 config.num_labels==1，则为回归）分数（SoftMax 之前）。

    • hidden_states (tuple[torch.FloatTensor, ...] | None.hidden_states, 当传入 output_hidden_states=True 或当 config.output_hidden_states=True 时返回) — 形状为 (batch_size, sequence_length, hidden_size) 的 torch.FloatTensor 元组（一个用于嵌入层的输出，如果模型有嵌入层，+ 每个层的输出）。

      模型在每个层输出的隐藏状态以及可选的初始嵌入输出。

    • attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x + attention_window + 1), where x is the number of tokens with global attention mask.

      局部注意力权重，在注意力 Softmax 之后，用于计算自注意力头中的加权平均。这些是序列中每个 token 对每个具有全局注意力的 token（前 x 个值）以及注意力窗口中的每个 token（剩余 `attention_window

      • 1值）。请注意，前x个值指的是具有固定文本位置的 token，而其余attention_window + 1个值指的是具有相对位置的 token：一个 token 对自身的注意力权重位于索引x + attention_window / 2处，而前面（后面）的attention_window / 2个值是注意力权重对前面（后面）attention_window / 2个 token 的注意力权重。如果注意力窗口包含一个具有全局注意力的 token，则相应索引处的注意力权重设置为 0；该值应从前x个注意力权重中访问。如果一个 token 具有全局注意力，则其在attentions中对所有其他 token 的注意力权重均设置为 0，该值应从global_attentions` 中访问。

    • global_attentions (tuple(torch.FloatTensor), 可选, 当传递 output_attentions=True 或 config.output_attentions=True 时返回) — 形状为 (batch_size, num_heads, sequence_length, x) 的 torch.FloatTensor 元组（每个层一个），其中 x 是具有全局注意力掩码的 token 数量。

      注意力 softmax 之后的全局注意力权重，用于计算自注意力头中的加权平均值。这些是每个具有全局注意力的 token 对序列中每个 token 的注意力权重。

    The LongformerForSequenceClassification forward method, overrides the __call__ special method.

    虽然 forward pass 的实现需要在此函数中定义，但你应该在之后调用 Module 实例而不是这个，因为前者负责运行预处理和后处理步骤，而后者会静默地忽略它们。

    单标签分类示例

    >>> import torch
>>> from transformers import AutoTokenizer, LongformerForSequenceClassification

>>> tokenizer = AutoTokenizer.from_pretrained("allenai/longformer-base-4096")
>>> model = LongformerForSequenceClassification.from_pretrained("allenai/longformer-base-4096")

>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")

>>> with torch.no_grad():
...     logits = model(**inputs).logits

>>> predicted_class_id = logits.argmax().item()
>>> model.config.id2label[predicted_class_id]
...

>>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)`
>>> num_labels = len(model.config.id2label)
>>> model = LongformerForSequenceClassification.from_pretrained("allenai/longformer-base-4096", num_labels=num_labels)

>>> labels = torch.tensor([1])
>>> loss = model(**inputs, labels=labels).loss
>>> round(loss.item(), 2)
...

    多标签分类示例

    >>> import torch
>>> from transformers import AutoTokenizer, LongformerForSequenceClassification

>>> tokenizer = AutoTokenizer.from_pretrained("allenai/longformer-base-4096")
>>> model = LongformerForSequenceClassification.from_pretrained("allenai/longformer-base-4096", problem_type="multi_label_classification")

>>> inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")

>>> with torch.no_grad():
...     logits = model(**inputs).logits

>>> predicted_class_ids = torch.arange(0, logits.shape[-1])[torch.sigmoid(logits).squeeze(dim=0) > 0.5]

>>> # To train a model on `num_labels` classes, you can pass `num_labels=num_labels` to `.from_pretrained(...)`
>>> num_labels = len(model.config.id2label)
>>> model = LongformerForSequenceClassification.from_pretrained(
...     "allenai/longformer-base-4096", num_labels=num_labels, problem_type="multi_label_classification"
... )

>>> labels = torch.sum(
...     torch.nn.functional.one_hot(predicted_class_ids[None, :].clone(), num_classes=num_labels), dim=1
... ).to(torch.float)
>>> loss = model(**inputs, labels=labels).loss

    LongformerForMultipleChoice

    class transformers.LongformerForMultipleChoice

    < >

    ( config model_args: ~utils.generic.ModelArgs | None = None adapter_args: ~utils.generic.AdapterArgs | None = None lora_args: ~utils.generic.LoRAArgs | None = None tokenizer_args: ~utils.generic.TokenizerArgs | None = None dataset_args: ~utils.generic.DatasetArgs | None = None data_args: ~utils.generic.DataArgs | None = None training_args: ~utils.generic.TrainingArgs | None = None generation_args: ~utils.generic.GenerationArgs | None = None vision_tower_args: ~utils.generic.VisionTowerArgs | None = None qlora_args: ~utils.generic.QLoRAArgs | None = None vision_tower_template_args: ~utils.generic.VisionTowerTemplateArgs | None = None video_tower_args: ~utils.generic.VideoTowerArgs | None = None vision_config: ~utils.generic.VisionConfig | None = None video_config: ~utils.generic.VideoConfig | None = None load_dataset: bool | None = None load_data_collator: bool | None = None load_processor: bool | None = None load_lora_adapter: bool | None = None load_adapter: bool | None = None load_qlora_adapter: bool | None = None **kwargs: typing_extensions.Unpack[transformers.modeling_utils.PreTrainedModelKwargs] )

    参数

    • config (LongformerForMultipleChoice) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

    Longformer 模型，顶部带有一个多选题分类头（在池化输出之上有一个线性层和一个 softmax），例如用于 RocStories/SWAG 任务。

    此模型继承自 PreTrainedModel。查看其父类文档，了解库为所有模型实现的通用方法（例如下载或保存、调整输入嵌入大小、修剪头等）。

    此模型也是一个 PyTorch torch.nn.Module 子类。像普通的 PyTorch Module 一样使用它，并参考 PyTorch 文档了解一般用法和行为的所有相关信息。

    forward

    < >

    ( input_ids: torch.Tensor | None = None token_type_ids: torch.Tensor | None = None attention_mask: torch.Tensor | None = None global_attention_mask: torch.Tensor | None = None labels: torch.Tensor | None = None position_ids: torch.Tensor | None = None inputs_embeds: torch.Tensor | None = None output_attentions: bool | None = None output_hidden_states: bool | None = None return_dict: bool | None = None **kwargs ) → transformers.models.longformer.modeling_longformer.LongformerMultipleChoiceModelOutput or tuple(torch.FloatTensor)

    参数

    • input_ids (torch.LongTensor of shape (batch_size, num_choices, sequence_length)) — Indices of input sequence tokens in the vocabulary.

      Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

      What are input IDs?

    • token_type_ids (torch.LongTensor of shape (batch_size, num_choices, sequence_length), optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:

      • 0 corresponds to a sentence A token,
      • 1 corresponds to a sentence B token.

      What are token type IDs?

    • attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:

      • 1 for tokens that are not masked,
      • 0 for tokens that are masked.

      What are attention masks?

    • global_attention_mask (torch.FloatTensor of shape (batch_size, num_choices, sequence_length), optional) — Mask to decide the attention given on each token, local attention or global attention. Tokens with global attention attends to all other tokens, and all other tokens attend to them. This is important for task-specific finetuning because it makes the model more flexible at representing the task. For example, for classification, the token should be given global attention. For QA, all question tokens should also have global attention. Please refer to the Longformer paper for more details. Mask values selected in [0, 1]:

      • 0 for local attention (a sliding window attention),
      • 1 for global attention (tokens that attend to all other tokens, and all other tokens attend to them).
    • labels (torch.LongTensor of shape (batch_size,), optional) — Labels for computing the multiple choice classification loss. Indices should be in [0, ..., num_choices-1] where num_choices is the size of the second dimension of the input tensors. (See input_ids above)
    • position_ids (torch.LongTensor of shape (batch_size, num_choices, sequence_length), optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.max_position_embeddings - 1].

      What are position IDs?

    • inputs_embeds (torch.FloatTensor of shape (batch_size, num_choices, sequence_length, hidden_size), optional) — Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert input_ids indices into associated vectors than the model’s internal embedding lookup matrix.
    • output_attentions (bool, optional) — Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail.
    • output_hidden_states (bool, optional) — Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail.
    • return_dict (bool, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

    返回

    transformers.models.longformer.modeling_longformer.LongformerMultipleChoiceModelOutput or tuple(torch.FloatTensor)

    A transformers.models.longformer.modeling_longformer.LongformerMultipleChoiceModelOutput or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (LongformerConfig) and inputs.

    • loss (形状为 (1,) 的 torch.FloatTensor，可选，当提供 labels 时返回) — 分类损失。

    • logits (形状为 (batch_size, num_choices) 的 torch.FloatTensor) — num_choices 是输入张量的第二维大小。（请参阅上面的 input_ids）。

      分类分数（SoftMax 之前）。

    • hidden_states (tuple[torch.FloatTensor, ...] | None.hidden_states, 当传入 output_hidden_states=True 或当 config.output_hidden_states=True 时返回) — 形状为 (batch_size, sequence_length, hidden_size) 的 torch.FloatTensor 元组（一个用于嵌入层的输出，如果模型有嵌入层，+ 每个层的输出）。

      模型在每个层输出的隐藏状态以及可选的初始嵌入输出。

    • attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x + attention_window + 1), where x is the number of tokens with global attention mask.

      局部注意力权重，在注意力 Softmax 之后，用于计算自注意力头中的加权平均。这些是序列中每个 token 对每个具有全局注意力的 token（前 x 个值）以及注意力窗口中的每个 token（剩余 `attention_window

      • 1值）。请注意，前x个值指的是具有固定文本位置的 token，而其余attention_window + 1个值指的是具有相对位置的 token：一个 token 对自身的注意力权重位于索引x + attention_window / 2处，而前面（后面）的attention_window / 2个值是注意力权重对前面（后面）attention_window / 2个 token 的注意力权重。如果注意力窗口包含一个具有全局注意力的 token，则相应索引处的注意力权重设置为 0；该值应从前x个注意力权重中访问。如果一个 token 具有全局注意力，则其在attentions中对所有其他 token 的注意力权重均设置为 0，该值应从global_attentions` 中访问。

    • global_attentions (tuple(torch.FloatTensor), 可选, 当传递 output_attentions=True 或 config.output_attentions=True 时返回) — 形状为 (batch_size, num_heads, sequence_length, x) 的 torch.FloatTensor 元组（每个层一个），其中 x 是具有全局注意力掩码的 token 数量。

      注意力 softmax 之后的全局注意力权重，用于计算自注意力头中的加权平均值。这些是每个具有全局注意力的 token 对序列中每个 token 的注意力权重。

    The LongformerForMultipleChoice forward method, overrides the __call__ special method.

    虽然 forward pass 的实现需要在此函数中定义，但你应该在之后调用 Module 实例而不是这个，因为前者负责运行预处理和后处理步骤，而后者会静默地忽略它们。

    示例

    >>> from transformers import AutoTokenizer, LongformerForMultipleChoice
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("allenai/longformer-base-4096")
>>> model = LongformerForMultipleChoice.from_pretrained("allenai/longformer-base-4096")

>>> prompt = "In Italy, pizza served in formal settings, such as at a restaurant, is presented unsliced."
>>> choice0 = "It is eaten with a fork and a knife."
>>> choice1 = "It is eaten while held in the hand."
>>> labels = torch.tensor(0).unsqueeze(0)  # choice0 is correct (according to Wikipedia ;)), batch size 1

>>> encoding = tokenizer([prompt, prompt], [choice0, choice1], return_tensors="pt", padding=True)
>>> outputs = model(**{k: v.unsqueeze(0) for k, v in encoding.items()}, labels=labels)  # batch size is 1

>>> # the linear classifier still needs to be trained
>>> loss = outputs.loss
>>> logits = outputs.logits

    LongformerForTokenClassification

    class transformers.LongformerForTokenClassification

    < >

    ( config model_args: ~utils.generic.ModelArgs | None = None adapter_args: ~utils.generic.AdapterArgs | None = None lora_args: ~utils.generic.LoRAArgs | None = None tokenizer_args: ~utils.generic.TokenizerArgs | None = None dataset_args: ~utils.generic.DatasetArgs | None = None data_args: ~utils.generic.DataArgs | None = None training_args: ~utils.generic.TrainingArgs | None = None generation_args: ~utils.generic.GenerationArgs | None = None vision_tower_args: ~utils.generic.VisionTowerArgs | None = None qlora_args: ~utils.generic.QLoRAArgs | None = None vision_tower_template_args: ~utils.generic.VisionTowerTemplateArgs | None = None video_tower_args: ~utils.generic.VideoTowerArgs | None = None vision_config: ~utils.generic.VisionConfig | None = None video_config: ~utils.generic.VideoConfig | None = None load_dataset: bool | None = None load_data_collator: bool | None = None load_processor: bool | None = None load_lora_adapter: bool | None = None load_adapter: bool | None = None load_qlora_adapter: bool | None = None **kwargs: typing_extensions.Unpack[transformers.modeling_utils.PreTrainedModelKwargs] )

    参数

    • config (LongformerForTokenClassification) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

    Longformer transformer，顶部带有一个 token 分类头（在 hidden-states 输出之上有一个线性层），例如用于命名实体识别 (NER) 任务。

    此模型继承自 PreTrainedModel。查看其父类文档，了解库为所有模型实现的通用方法（例如下载或保存、调整输入嵌入大小、修剪头等）。

    此模型也是一个 PyTorch torch.nn.Module 子类。像普通的 PyTorch Module 一样使用它，并参考 PyTorch 文档了解一般用法和行为的所有相关信息。

    forward

    < >

    ( input_ids: torch.Tensor | None = None attention_mask: torch.Tensor | None = None global_attention_mask: torch.Tensor | None = None token_type_ids: torch.Tensor | None = None position_ids: torch.Tensor | None = None inputs_embeds: torch.Tensor | None = None labels: torch.Tensor | None = None output_attentions: bool | None = None output_hidden_states: bool | None = None return_dict: bool | None = None **kwargs ) → transformers.models.longformer.modeling_longformer.LongformerTokenClassifierOutput or tuple(torch.FloatTensor)

    参数

    • input_ids (torch.Tensor of shape (batch_size, sequence_length), optional) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.

      Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

      What are input IDs?

    • attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:

      • 1 for tokens that are not masked,
      • 0 for tokens that are masked.

      What are attention masks?

    • global_attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) — Mask to decide the attention given on each token, local attention or global attention. Tokens with global attention attends to all other tokens, and all other tokens attend to them. This is important for task-specific finetuning because it makes the model more flexible at representing the task. For example, for classification, the token should be given global attention. For QA, all question tokens should also have global attention. Please refer to the Longformer paper for more details. Mask values selected in [0, 1]:

      • 0 for local attention (a sliding window attention),
      • 1 for global attention (tokens that attend to all other tokens, and all other tokens attend to them).
    • token_type_ids (torch.Tensor of shape (batch_size, sequence_length), optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:

      • 0 corresponds to a sentence A token,
      • 1 corresponds to a sentence B token.

      What are token type IDs?

    • position_ids (torch.Tensor of shape (batch_size, sequence_length), optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.n_positions - 1].

      What are position IDs?

    • inputs_embeds (torch.Tensor of shape (batch_size, sequence_length, hidden_size), optional) — Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert input_ids indices into associated vectors than the model’s internal embedding lookup matrix.
    • labels (torch.LongTensor of shape (batch_size, sequence_length), optional) — Labels for computing the token classification loss. Indices should be in [0, ..., config.num_labels - 1].
    • output_attentions (bool, optional) — Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail.
    • output_hidden_states (bool, optional) — Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail.
    • return_dict (bool, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

    返回

    transformers.models.longformer.modeling_longformer.LongformerTokenClassifierOutput or tuple(torch.FloatTensor)

    A transformers.models.longformer.modeling_longformer.LongformerTokenClassifierOutput or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (LongformerConfig) and inputs.

    • loss (形状为 (1,) 的 torch.FloatTensor，可选，当提供 labels 时返回) — 分类损失。

    • logits (形状为 (batch_size, sequence_length, config.num_labels) 的 torch.FloatTensor) — 分类分数（SoftMax 之前）。

    • hidden_states (tuple[torch.FloatTensor, ...] | None.hidden_states, 当传入 output_hidden_states=True 或当 config.output_hidden_states=True 时返回) — 形状为 (batch_size, sequence_length, hidden_size) 的 torch.FloatTensor 元组（一个用于嵌入层的输出，如果模型有嵌入层，+ 每个层的输出）。

      模型在每个层输出的隐藏状态以及可选的初始嵌入输出。

    • attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x + attention_window + 1), where x is the number of tokens with global attention mask.

      局部注意力权重，在注意力 Softmax 之后，用于计算自注意力头中的加权平均。这些是序列中每个 token 对每个具有全局注意力的 token（前 x 个值）以及注意力窗口中的每个 token（剩余 `attention_window

      • 1值）。请注意，前x个值指的是具有固定文本位置的 token，而其余attention_window + 1个值指的是具有相对位置的 token：一个 token 对自身的注意力权重位于索引x + attention_window / 2处，而前面（后面）的attention_window / 2个值是注意力权重对前面（后面）attention_window / 2个 token 的注意力权重。如果注意力窗口包含一个具有全局注意力的 token，则相应索引处的注意力权重设置为 0；该值应从前x个注意力权重中访问。如果一个 token 具有全局注意力，则其在attentions中对所有其他 token 的注意力权重均设置为 0，该值应从global_attentions` 中访问。

    • global_attentions (tuple(torch.FloatTensor), 可选, 当传递 output_attentions=True 或 config.output_attentions=True 时返回) — 形状为 (batch_size, num_heads, sequence_length, x) 的 torch.FloatTensor 元组（每个层一个），其中 x 是具有全局注意力掩码的 token 数量。

      注意力 softmax 之后的全局注意力权重，用于计算自注意力头中的加权平均值。这些是每个具有全局注意力的 token 对序列中每个 token 的注意力权重。

    The LongformerForTokenClassification forward method, overrides the __call__ special method.

    虽然 forward pass 的实现需要在此函数中定义，但你应该在之后调用 Module 实例而不是这个，因为前者负责运行预处理和后处理步骤，而后者会静默地忽略它们。

    示例

    >>> from transformers import AutoTokenizer, LongformerForTokenClassification
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("allenai/longformer-base-4096")
>>> model = LongformerForTokenClassification.from_pretrained("allenai/longformer-base-4096")

>>> inputs = tokenizer(
...     "HuggingFace is a company based in Paris and New York", add_special_tokens=False, return_tensors="pt"
... )

>>> with torch.no_grad():
...     logits = model(**inputs).logits

>>> predicted_token_class_ids = logits.argmax(-1)

>>> # Note that tokens are classified rather then input words which means that
>>> # there might be more predicted token classes than words.
>>> # Multiple token classes might account for the same word
>>> predicted_tokens_classes = [model.config.id2label[t.item()] for t in predicted_token_class_ids[0]]
>>> predicted_tokens_classes
...

>>> labels = predicted_token_class_ids
>>> loss = model(**inputs, labels=labels).loss
>>> round(loss.item(), 2)
...

    LongformerForQuestionAnswering

    class transformers.LongformerForQuestionAnswering

    < >

    ( config model_args: ~utils.generic.ModelArgs | None = None adapter_args: ~utils.generic.AdapterArgs | None = None lora_args: ~utils.generic.LoRAArgs | None = None tokenizer_args: ~utils.generic.TokenizerArgs | None = None dataset_args: ~utils.generic.DatasetArgs | None = None data_args: ~utils.generic.DataArgs | None = None training_args: ~utils.generic.TrainingArgs | None = None generation_args: ~utils.generic.GenerationArgs | None = None vision_tower_args: ~utils.generic.VisionTowerArgs | None = None qlora_args: ~utils.generic.QLoRAArgs | None = None vision_tower_template_args: ~utils.generic.VisionTowerTemplateArgs | None = None video_tower_args: ~utils.generic.VideoTowerArgs | None = None vision_config: ~utils.generic.VisionConfig | None = None video_config: ~utils.generic.VideoConfig | None = None load_dataset: bool | None = None load_data_collator: bool | None = None load_processor: bool | None = None load_lora_adapter: bool | None = None load_adapter: bool | None = None load_qlora_adapter: bool | None = None **kwargs: typing_extensions.Unpack[transformers.modeling_utils.PreTrainedModelKwargs] )

    参数

    • config (LongformerForQuestionAnswering) — Model configuration class with all the parameters of the model. Initializing with a config file does not load the weights associated with the model, only the configuration. Check out the from_pretrained() method to load the model weights.

    Longformer transformer with a span classification head on top for extractive question-answering tasks like SQuAD (a linear layer on top of the hidden-states output to compute span start logits and span end logits).

    此模型继承自 PreTrainedModel。查看其父类文档，了解库为所有模型实现的通用方法（例如下载或保存、调整输入嵌入大小、修剪头等）。

    此模型也是一个 PyTorch torch.nn.Module 子类。像普通的 PyTorch Module 一样使用它，并参考 PyTorch 文档了解一般用法和行为的所有相关信息。

    forward

    < >

    ( input_ids: torch.Tensor | None = None attention_mask: torch.Tensor | None = None global_attention_mask: torch.Tensor | None = None token_type_ids: torch.Tensor | None = None position_ids: torch.Tensor | None = None inputs_embeds: torch.Tensor | None = None start_positions: torch.Tensor | None = None end_positions: torch.Tensor | None = None output_attentions: bool | None = None output_hidden_states: bool | None = None return_dict: bool | None = None **kwargs ) → transformers.models.longformer.modeling_longformer.LongformerQuestionAnsweringModelOutput or tuple(torch.FloatTensor)

    参数

    • input_ids (torch.Tensor of shape (batch_size, sequence_length), optional) — Indices of input sequence tokens in the vocabulary. Padding will be ignored by default.

      Indices can be obtained using AutoTokenizer. See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for details.

      What are input IDs?

    • attention_mask (torch.Tensor of shape (batch_size, sequence_length), optional) — Mask to avoid performing attention on padding token indices. Mask values selected in [0, 1]:

      • 1 for tokens that are not masked,
      • 0 for tokens that are masked.

      What are attention masks?

    • global_attention_mask (torch.FloatTensor of shape (batch_size, sequence_length), optional) — Mask to decide the attention given on each token, local attention or global attention. Tokens with global attention attends to all other tokens, and all other tokens attend to them. This is important for task-specific finetuning because it makes the model more flexible at representing the task. For example, for classification, the token should be given global attention. For QA, all question tokens should also have global attention. Please refer to the Longformer paper for more details. Mask values selected in [0, 1]:

      • 0 for local attention (a sliding window attention),
      • 1 for global attention (tokens that attend to all other tokens, and all other tokens attend to them).
    • token_type_ids (torch.Tensor of shape (batch_size, sequence_length), optional) — Segment token indices to indicate first and second portions of the inputs. Indices are selected in [0, 1]:

      • 0 corresponds to a sentence A token,
      • 1 corresponds to a sentence B token.

      What are token type IDs?

    • position_ids (torch.Tensor of shape (batch_size, sequence_length), optional) — Indices of positions of each input sequence tokens in the position embeddings. Selected in the range [0, config.n_positions - 1].

      What are position IDs?

    • inputs_embeds (torch.Tensor of shape (batch_size, sequence_length, hidden_size), optional) — Optionally, instead of passing input_ids you can choose to directly pass an embedded representation. This is useful if you want more control over how to convert input_ids indices into associated vectors than the model’s internal embedding lookup matrix.
    • start_positions (torch.Tensor of shape (batch_size,), optional) — Labels for position (index) of the start of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence are not taken into account for computing the loss.
    • end_positions (torch.Tensor of shape (batch_size,), optional) — Labels for position (index) of the end of the labelled span for computing the token classification loss. Positions are clamped to the length of the sequence (sequence_length). Position outside of the sequence are not taken into account for computing the loss.
    • output_attentions (bool, optional) — Whether or not to return the attentions tensors of all attention layers. See attentions under returned tensors for more detail.
    • output_hidden_states (bool, optional) — Whether or not to return the hidden states of all layers. See hidden_states under returned tensors for more detail.
    • return_dict (bool, optional) — Whether or not to return a ModelOutput instead of a plain tuple.

    返回

    transformers.models.longformer.modeling_longformer.LongformerQuestionAnsweringModelOutput or tuple(torch.FloatTensor)

    A transformers.models.longformer.modeling_longformer.LongformerQuestionAnsweringModelOutput or a tuple of torch.FloatTensor (if return_dict=False is passed or when config.return_dict=False) comprising various elements depending on the configuration (LongformerConfig) and inputs.

    • loss (torch.FloatTensor of shape (1,), 可选, 当提供 labels 时返回) — 总范围提取损失是起始位置和结束位置的交叉熵之和。

    • start_logits (torch.FloatTensor | None.start_logits of shape (batch_size, sequence_length), defaults to None) — Span 起始得分（SoftMax 之前）。

    • end_logits (torch.FloatTensor | None.end_logits of shape (batch_size, sequence_length), defaults to None) — Span 结束得分（SoftMax 之前）。

    • hidden_states (tuple[torch.FloatTensor, ...] | None.hidden_states, 当传入 output_hidden_states=True 或当 config.output_hidden_states=True 时返回) — 形状为 (batch_size, sequence_length, hidden_size) 的 torch.FloatTensor 元组（一个用于嵌入层的输出，如果模型有嵌入层，+ 每个层的输出）。

      模型在每个层输出的隐藏状态以及可选的初始嵌入输出。

    • attentions (tuple(torch.FloatTensor), optional, returned when output_attentions=True is passed or when config.output_attentions=True) — Tuple of torch.FloatTensor (one for each layer) of shape (batch_size, num_heads, sequence_length, x + attention_window + 1), where x is the number of tokens with global attention mask.

      局部注意力权重，在注意力 Softmax 之后，用于计算自注意力头中的加权平均。这些是序列中每个 token 对每个具有全局注意力的 token（前 x 个值）以及注意力窗口中的每个 token（剩余 `attention_window

      • 1值）。请注意，前x个值指的是具有固定文本位置的 token，而其余attention_window + 1个值指的是具有相对位置的 token：一个 token 对自身的注意力权重位于索引x + attention_window / 2处，而前面（后面）的attention_window / 2个值是注意力权重对前面（后面）attention_window / 2个 token 的注意力权重。如果注意力窗口包含一个具有全局注意力的 token，则相应索引处的注意力权重设置为 0；该值应从前x个注意力权重中访问。如果一个 token 具有全局注意力，则其在attentions中对所有其他 token 的注意力权重均设置为 0，该值应从global_attentions` 中访问。

    • global_attentions (tuple(torch.FloatTensor), 可选, 当传递 output_attentions=True 或 config.output_attentions=True 时返回) — 形状为 (batch_size, num_heads, sequence_length, x) 的 torch.FloatTensor 元组（每个层一个），其中 x 是具有全局注意力掩码的 token 数量。

      注意力 softmax 之后的全局注意力权重，用于计算自注意力头中的加权平均值。这些是每个具有全局注意力的 token 对序列中每个 token 的注意力权重。

    The LongformerForQuestionAnswering forward method, overrides the __call__ special method.

    虽然 forward pass 的实现需要在此函数中定义，但你应该在之后调用 Module 实例而不是这个，因为前者负责运行预处理和后处理步骤，而后者会静默地忽略它们。

    示例

    >>> from transformers import AutoTokenizer, LongformerForQuestionAnswering
>>> import torch

>>> tokenizer = AutoTokenizer.from_pretrained("allenai/longformer-large-4096-finetuned-triviaqa")
>>> model = LongformerForQuestionAnswering.from_pretrained("allenai/longformer-large-4096-finetuned-triviaqa")

>>> question, text = "Who was Jim Henson?", "Jim Henson was a nice puppet"
>>> encoding = tokenizer(question, text, return_tensors="pt")
>>> input_ids = encoding["input_ids"]

>>> # default is local attention everywhere
>>> # the forward method will automatically set global attention on question tokens
>>> attention_mask = encoding["attention_mask"]

>>> outputs = model(input_ids, attention_mask=attention_mask)
>>> start_logits = outputs.start_logits
>>> end_logits = outputs.end_logits
>>> all_tokens = tokenizer.convert_ids_to_tokens(input_ids[0].tolist())

>>> answer_tokens = all_tokens[torch.argmax(start_logits) : torch.argmax(end_logits) + 1]
>>> answer = tokenizer.decode(
...     tokenizer.convert_tokens_to_ids(answer_tokens)
... )  # remove space prepending space token
    在 GitHub 上更新

    ←LongCatFlash LongT5→
    © . This site is unofficial and not affiliated with Hugging Face, Inc.