因果语言建模
语言建模有两种类型：因果语言建模和掩码语言建模。本指南介绍因果语言建模。因果语言模型常用于文本生成。您可以将这些模型用于创意应用，例如选择你自己的文本冒险游戏或智能编码助手，例如Copilot或CodeParrot。
因果语言建模预测序列中下一个标记，并且模型只能关注左侧的标记。这意味着模型无法看到未来的标记。GPT-2就是一个因果语言模型的例子。
本指南将向您展示如何
- 在DistilGPT2上微调r/askscience子集的ELI5数据集。
- 将您微调后的模型用于推理。
要查看与此任务兼容的所有架构和检查点，我们建议您查看任务页面
在开始之前，请确保您已安装所有必要的库
pip install transformers datasets evaluate
我们鼓励您登录您的Hugging Face帐户，以便您可以上传模型并与社区共享。当系统提示时，输入您的令牌以登录
>>> from huggingface_hub import notebook_login
>>> notebook_login()加载ELI5数据集
首先使用 🤗 Datasets 库加载ELI5-Category数据集的前 5000 个示例。这将使您有机会进行实验并确保一切正常，然后再花费更多时间在完整数据集上进行训练。
>>> from datasets import load_dataset
>>> eli5 = load_dataset("eli5_category", split="train[:5000]")使用train_test_split方法将数据集的train拆分拆分为训练集和测试集
>>> eli5 = eli5.train_test_split(test_size=0.2)然后查看一个示例
>>> eli5["train"][0]
{'q_id': '7h191n',
'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',
'selftext': '',
'category': 'Economics',
'subreddit': 'explainlikeimfive',
'answers': {'a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],
'text': ["The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.",
'None yet. It has to be reconciled with a vastly different house bill and then passed again.',
'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',
'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level. URL_0'],
'score': [21, 19, 5, 3],
'text_urls': [[],
[],
[],
['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']]},
'title_urls': ['url'],
'selftext_urls': ['url']}虽然这看起来可能很多，但您实际上只对text字段感兴趣。语言建模任务的妙处在于您不需要标签（也称为无监督任务），因为下一个词就是标签。
预处理
下一步是加载一个DistilGPT2分词器来处理text子字段
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("distilbert/distilgpt2")您会从上面的示例中注意到，text字段实际上嵌套在answers中。这意味着您需要使用flatten方法从其嵌套结构中提取text子字段
>>> eli5 = eli5.flatten()
>>> eli5["train"][0]
{'q_id': '7h191n',
'title': 'What does the tax bill that was passed today mean? How will it affect Americans in each tax bracket?',
'selftext': '',
'category': 'Economics',
'subreddit': 'explainlikeimfive',
'answers.a_id': ['dqnds8l', 'dqnd1jl', 'dqng3i1', 'dqnku5x'],
'answers.text': ["The tax bill is 500 pages long and there were a lot of changes still going on right to the end. It's not just an adjustment to the income tax brackets, it's a whole bunch of changes. As such there is no good answer to your question. The big take aways are: - Big reduction in corporate income tax rate will make large companies very happy. - Pass through rate change will make certain styles of business (law firms, hedge funds) extremely happy - Income tax changes are moderate, and are set to expire (though it's the kind of thing that might just always get re-applied without being made permanent) - People in high tax states (California, New York) lose out, and many of them will end up with their taxes raised.",
'None yet. It has to be reconciled with a vastly different house bill and then passed again.',
'Also: does this apply to 2017 taxes? Or does it start with 2018 taxes?',
'This article explains both the House and senate bills, including the proposed changes to your income taxes based on your income level. URL_0'],
'answers.score': [21, 19, 5, 3],
'answers.text_urls': [[],
[],
[],
['https://www.investopedia.com/news/trumps-tax-reform-what-can-be-done/']],
'title_urls': ['url'],
'selftext_urls': ['url']}现在每个子字段都是一个单独的列，如answers前缀所示，并且text字段现在是一个列表。与其分别对每个句子进行分词，不如将列表转换为字符串，以便您可以联合对其进行分词。
这是一个用于连接每个示例的字符串列表并对结果进行分词的第一个预处理函数
>>> def preprocess_function(examples):
... return tokenizer([" ".join(x) for x in examples["answers.text"]])要将此预处理函数应用于整个数据集，请使用 🤗 Datasets map方法。您可以通过设置batched=True来加速map函数，以一次处理数据集的多个元素，并使用num_proc增加进程数。删除任何不需要的列
>>> tokenized_eli5 = eli5.map(
... preprocess_function,
... batched=True,
... num_proc=4,
... remove_columns=eli5["train"].column_names,
... )此数据集包含标记序列，但其中一些序列比模型的最大输入长度更长。
您现在可以使用第二个预处理函数来
- 连接所有序列
- 将连接的序列拆分为由
block_size定义的较短的块，该块应该短于最大输入长度，并且足够短以适合您的GPU RAM。
>>> block_size = 128
>>> def group_texts(examples):
... # Concatenate all texts.
... concatenated_examples = {k: sum(examples[k], []) for k in examples.keys()}
... total_length = len(concatenated_examples[list(examples.keys())[0]])
... # We drop the small remainder, we could add padding if the model supported it instead of this drop, you can
... # customize this part to your needs.
... if total_length >= block_size:
... total_length = (total_length // block_size) * block_size
... # Split by chunks of block_size.
... result = {
... k: [t[i : i + block_size] for i in range(0, total_length, block_size)]
... for k, t in concatenated_examples.items()
... }
... result["labels"] = result["input_ids"].copy()
... return result将group_texts函数应用于整个数据集
>>> lm_dataset = tokenized_eli5.map(group_texts, batched=True, num_proc=4)现在使用DataCollatorForLanguageModeling创建一批示例。在整理过程中，动态填充句子到批次中最长的长度比将整个数据集填充到最大长度更有效率。
Pytorch
隐藏Pytorch内容
使用序列结束标记作为填充标记并设置mlm=False。这将使用向右移动一个元素的输入作为标签
>>> from transformers import DataCollatorForLanguageModeling
>>> tokenizer.pad_token = tokenizer.eos_token
>>> data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False) TensorFlow
隐藏TensorFlow内容
使用序列结束标记作为填充标记并设置mlm=False。这将使用向右移动一个元素的输入作为标签
>>> from transformers import DataCollatorForLanguageModeling
>>> data_collator = DataCollatorForLanguageModeling(tokenizer=tokenizer, mlm=False, return_tensors="tf")训练
Pytorch
隐藏Pytorch内容
您现在可以开始训练您的模型了！使用AutoModelForCausalLM加载DistilGPT2
>>> from transformers import AutoModelForCausalLM, TrainingArguments, Trainer
>>> model = AutoModelForCausalLM.from_pretrained("distilbert/distilgpt2")此时，只剩下三个步骤
- 在TrainingArguments中定义您的训练超参数。唯一必需的参数是
output_dir，它指定保存模型的位置。您将通过设置push_to_hub=True将此模型推送到Hub（您需要登录Hugging Face才能上传模型）。 - 将训练参数传递给Trainer，以及模型、数据集和数据整理器。
- 调用train()以微调您的模型。
>>> training_args = TrainingArguments(
... output_dir="my_awesome_eli5_clm-model",
... eval_strategy="epoch",
... learning_rate=2e-5,
... weight_decay=0.01,
... push_to_hub=True,
... )
>>> trainer = Trainer(
... model=model,
... args=training_args,
... train_dataset=lm_dataset["train"],
... eval_dataset=lm_dataset["test"],
... data_collator=data_collator,
... tokenizer=tokenizer,
... )
>>> trainer.train()训练完成后，使用evaluate()方法评估您的模型并获取其困惑度
>>> import math
>>> eval_results = trainer.evaluate()
>>> print(f"Perplexity: {math.exp(eval_results['eval_loss']):.2f}")
Perplexity: 49.61然后使用push_to_hub()方法将您的模型共享到Hub，以便每个人都可以使用您的模型
>>> trainer.push_to_hub() TensorFlow
隐藏TensorFlow内容
如果您不熟悉使用Keras微调模型，请查看基本教程！
>>> from transformers import create_optimizer, AdamWeightDecay
>>> optimizer = AdamWeightDecay(learning_rate=2e-5, weight_decay_rate=0.01)然后，您可以使用TFAutoModelForCausalLM加载DistilGPT2
>>> from transformers import TFAutoModelForCausalLM
>>> model = TFAutoModelForCausalLM.from_pretrained("distilbert/distilgpt2")使用prepare_tf_dataset()将您的数据集转换为tf.data.Dataset格式
>>> tf_train_set = model.prepare_tf_dataset(
... lm_dataset["train"],
... shuffle=True,
... batch_size=16,
... collate_fn=data_collator,
... )
>>> tf_test_set = model.prepare_tf_dataset(
... lm_dataset["test"],
... shuffle=False,
... batch_size=16,
... collate_fn=data_collator,
... )使用compile配置模型以进行训练。请注意，Transformers模型都具有默认的任务相关损失函数，因此您无需指定一个，除非您想指定。
>>> import tensorflow as tf
>>> model.compile(optimizer=optimizer) # No loss argument!这可以通过在PushToHubCallback中指定要将模型和分词器推送到哪里来完成
>>> from transformers.keras_callbacks import PushToHubCallback
>>> callback = PushToHubCallback(
... output_dir="my_awesome_eli5_clm-model",
... tokenizer=tokenizer,
... )最后，您就可以开始训练您的模型了！使用您的训练集和验证集、时期数和回调函数调用fit来微调模型
>>> model.fit(x=tf_train_set, validation_data=tf_test_set, epochs=3, callbacks=[callback])训练完成后，您的模型会自动上传到Hub，以便每个人都可以使用它！
有关如何微调因果语言建模模型的更深入示例，请查看相应的PyTorch笔记本或TensorFlow笔记本。
推理
太好了，既然您已经微调了一个模型，那么您就可以将其用于推理了！
想出一个您想从中生成文本的提示
>>> prompt = "Somatic hypermutation allows the immune system to"尝试使用您微调的模型进行推理的最简单方法是在pipeline()中使用它。使用您的模型实例化一个用于文本生成的pipeline，并将您的文本传递给它
>>> from transformers import pipeline
>>> generator = pipeline("text-generation", model="username/my_awesome_eli5_clm-model")
>>> generator(prompt)
[{'generated_text': "Somatic hypermutation allows the immune system to be able to effectively reverse the damage caused by an infection.\n\n\nThe damage caused by an infection is caused by the immune system's ability to perform its own self-correcting tasks."}] Pytorch
隐藏Pytorch内容
对文本进行分词并将input_ids作为PyTorch张量返回
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("username/my_awesome_eli5_clm-model")
>>> inputs = tokenizer(prompt, return_tensors="pt").input_ids使用generate()方法生成文本。有关控制生成的各种文本生成策略和参数的更多详细信息，请查看文本生成策略页面。
>>> from transformers import AutoModelForCausalLM
>>> model = AutoModelForCausalLM.from_pretrained("username/my_awesome_eli5_clm-model")
>>> outputs = model.generate(inputs, max_new_tokens=100, do_sample=True, top_k=50, top_p=0.95)将生成的标记ID解码回文本
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
["Somatic hypermutation allows the immune system to react to drugs with the ability to adapt to a different environmental situation. In other words, a system of 'hypermutation' can help the immune system to adapt to a different environmental situation or in some cases even a single life. In contrast, researchers at the University of Massachusetts-Boston have found that 'hypermutation' is much stronger in mice than in humans but can be found in humans, and that it's not completely unknown to the immune system. A study on how the immune system"] TensorFlow
隐藏TensorFlow内容
对文本进行分词并将input_ids作为TensorFlow张量返回
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("username/my_awesome_eli5_clm-model")
>>> inputs = tokenizer(prompt, return_tensors="tf").input_ids使用~transformers.generation_tf_utils.TFGenerationMixin.generate方法创建摘要。有关控制生成的各种文本生成策略和参数的更多详细信息，请查看文本生成策略页面。
>>> from transformers import TFAutoModelForCausalLM
>>> model = TFAutoModelForCausalLM.from_pretrained("username/my_awesome_eli5_clm-model")
>>> outputs = model.generate(input_ids=inputs, max_new_tokens=100, do_sample=True, top_k=50, top_p=0.95)将生成的标记ID解码回文本
>>> tokenizer.batch_decode(outputs, skip_special_tokens=True)
['Somatic hypermutation allows the immune system to detect the presence of other viruses as they become more prevalent. Therefore, researchers have identified a high proportion of human viruses. The proportion of virus-associated viruses in our study increases with age. Therefore, we propose a simple algorithm to detect the presence of these new viruses in our samples as a sign of improved immunity. A first study based on this algorithm, which will be published in Science on Friday, aims to show that this finding could translate into the development of a better vaccine that is more effective for']