开源 AI 食谱文档
使用 LLM-as-a-judge 🧑⚖️ 进行自动化和多功能评估
并获得增强的文档体验
开始使用
使用 LLM-as-a-judge 🧑⚖️ 进行自动化和多功能评估
评估大型语言模型 (LLM) 通常是一项艰巨的任务:鉴于其广泛的功能,分配给它们的任务通常应根据非常广泛且定义松散的要求进行判断。例如,助手对问题的回答可能是
- 没有根据上下文
- 重复,重复,重复
- 语法不正确
- 过度冗长,并以过多的词语为特征,导致话语或书面内容变得过于详细和冗长
- 语无伦次
- ……
标准列表还在不断增加。即使我们有一个有限的列表,这些标准中的每一个都很难衡量:“设计一个基于规则的程序来评估输出是极其具有挑战性的。基于输出和参考答案之间相似性的传统评估指标(例如,ROUGE、BLEU)对于这些问题也无效。”
✅ 一种以人为方式评估输出的强大解决方案,无需花费昂贵的人工时间,就是 LLM-as-a-judge。这种方法在 Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena 中介绍 - 我鼓励您阅读。
💡 这个想法很简单:让 LLM 为你评分。🤖✓
但我们将看到,它不会开箱即用:你需要仔细设置它才能获得良好的结果。
!pip install huggingface_hub datasets pandas tqdm -q
import re
import pandas as pd
from tqdm.auto import tqdm
from datasets import load_dataset
from huggingface_hub import InferenceClient, notebook_login
tqdm.pandas() # load tqdm's pandas support
pd.set_option("display.max_colwidth", None)
notebook_login()
repo_id = "mistralai/Mixtral-8x7B-Instruct-v0.1"
llm_client = InferenceClient(
model=repo_id,
timeout=120,
)
# Test your LLM client
llm_client.text_generation(prompt="How are you today?", max_new_tokens=20)
1. 准备创建和评估我们的 LLM judge
假设你想给 LLM 一个特定的任务,比如回答开放式问题。
困难在于,正如我们上面讨论的那样,衡量答案的质量很困难,例如,精确的字符串匹配会将太多正确但措辞不同的答案标记为错误。
你可以让人工标注员来判断输出,但这对于他们来说非常耗时,而且如果你想更新模型或问题,你必须重新做一遍。
✅ 在这种情况下,你可以设置一个 LLM-as-a-judge。
但是要使用 LLM-as-a-judge,你首先需要评估它对你的模型输出评分的可靠性。
➡️ 所以第一步将是... 创建一个人为评估数据集。但是你可以只对少数几个示例进行人工标注 - 大约 30 个就足以很好地了解性能。你将能够在每次想要测试你的 LLM-as-a-judge 时重复使用此数据集。
在我们的例子中,我们将使用 feedbackQA
,它包含 2 个人为评估和每个问题/答案对的分数:使用 30 个示例的样本将代表你的小型评估数据集可能是什么样子。
ratings = load_dataset("McGill-NLP/feedbackQA")["train"]
ratings = pd.DataFrame(ratings)
ratings["review_1"] = ratings["feedback"].apply(lambda x: x["rating"][0])
ratings["explanation_1"] = ratings["feedback"].apply(lambda x: x["explanation"][0])
ratings["review_2"] = ratings["feedback"].apply(lambda x: x["rating"][1])
ratings["explanation_2"] = ratings["feedback"].apply(lambda x: x["explanation"][1])
ratings = ratings.drop(columns=["feedback"])
# Map scores to numeric values
conversion_dict = {"Excellent": 4, "Acceptable": 3, "Could be Improved": 2, "Bad": 1}
ratings["score_1"] = ratings["review_1"].map(conversion_dict)
ratings["score_2"] = ratings["review_2"].map(conversion_dict)
计算性能基线总是一个好主意:在这里,它可以是例如两个人工评分者之间的一致性,通过他们给出的分数的 皮尔逊相关性 来衡量。
>>> print("Correlation between 2 human raters:")
>>> print(f"{ratings['score_1'].corr(ratings['score_2'], method='pearson'):.3f}")
Correlation between 2 human raters: 0.563
这两个人工评分者之间的相关性不是很好。如果你的人工评分真的很差,则可能意味着评分标准不够清晰。
这意味着我们的“真实情况”包含噪声:因此我们不能期望任何算法评估都能非常接近它。
但是,我们可以减少这种噪声
- 通过取平均分作为我们的真实情况,而不是任何单一分数,我们应该消除一些不规则性。
- 仅选择人工审核员意见一致的样本。
在这里,我们将选择最后一个选项,只保留 2 个人工审核员意见一致的示例。
# Sample examples
ratings_where_raters_agree = ratings.loc[ratings["score_1"] == ratings["score_2"]]
examples = ratings_where_raters_agree.groupby("score_1").sample(7, random_state=1214)
examples["human_score"] = examples["score_1"]
# Visualize 1 sample for each score
display(examples.groupby("human_score").first())
2. 创建我们的 LLM judge
我们使用基本提示构建我们的 LLM judge,其中包含以下元素
- 任务描述
- 量表描述:
minimum
、maximum
、值类型(此处为float
) - 输出格式的说明
- 答案的开头,尽可能引导 LLM
JUDGE_PROMPT = """
You will be given a user_question and system_answer couple.
Your task is to provide a 'total rating' scoring how well the system_answer answers the user concerns expressed in the user_question.
Give your answer as a float on a scale of 0 to 10, where 0 means that the system_answer is not helpful at all, and 10 means that the answer completely and helpfully addresses the question.
Provide your feedback as follows:
Feedback:::
Total rating: (your rating, as a float between 0 and 10)
Now here are the question and answer.
Question: {question}
Answer: {answer}
Feedback:::
Total rating: """
examples["llm_judge"] = examples.progress_apply(
lambda x: llm_client.text_generation(
prompt=JUDGE_PROMPT.format(question=x["question"], answer=x["answer"]),
max_new_tokens=1000,
),
axis=1,
)
def extract_judge_score(answer: str, split_str: str = "Total rating:") -> int:
try:
if split_str in answer:
rating = answer.split(split_str)[1]
else:
rating = answer
digit_groups = [el.strip() for el in re.findall(r"\d+(?:\.\d+)?", rating)]
return float(digit_groups[0])
except Exception as e:
print(e)
return None
examples["llm_judge_score"] = examples["llm_judge"].apply(extract_judge_score)
# Rescale the score given by the LLM on the same scale as the human score
examples["llm_judge_score"] = (examples["llm_judge_score"] / 10) + 1
>>> print("Correlation between LLM-as-a-judge and the human raters:")
>>> print(f"{examples['llm_judge_score'].corr(examples['human_score'], method='pearson'):.3f}")
Correlation between LLM-as-a-judge and the human raters: 0.567
考虑到 2 个随机独立变量之间的皮尔逊相关性为 0!这还不错!
但我们可以轻松做得更好。🔝
3. 改进 LLM judge
正如 Aparna Dhinakaran 所展示的那样,LLM 不擅长在连续范围内评估输出。本文 为我们提供了构建更好提示的一些最佳实践
- ⏳ 留出更多思考时间,在最终答案之前添加一个
Evaluation
字段。 - 🔢 使用小的整数量表,如 1-4 或 1-5,而不是我们之前使用的大浮点数量表。
- 👩🏫 提供指示性量表以进行指导。
- 我们甚至添加了一个胡萝卜来激励 LLM!
IMPROVED_JUDGE_PROMPT = """
You will be given a user_question and system_answer couple.
Your task is to provide a 'total rating' scoring how well the system_answer answers the user concerns expressed in the user_question.
Give your answer on a scale of 1 to 4, where 1 means that the system_answer is not helpful at all, and 4 means that the system_answer completely and helpfully addresses the user_question.
Here is the scale you should use to build your answer:
1: The system_answer is terrible: completely irrelevant to the question asked, or very partial
2: The system_answer is mostly not helpful: misses some key aspects of the question
3: The system_answer is mostly helpful: provides support, but still could be improved
4: The system_answer is excellent: relevant, direct, detailed, and addresses all the concerns raised in the question
Provide your feedback as follows:
Feedback:::
Evaluation: (your rationale for the rating, as a text)
Total rating: (your rating, as a number between 1 and 4)
You MUST provide values for 'Evaluation:' and 'Total rating:' in your answer.
Now here are the question and answer.
Question: {question}
Answer: {answer}
Provide your feedback. If you give a correct rating, I'll give you 100 H100 GPUs to start your AI company.
Feedback:::
Evaluation: """
examples["llm_judge_improved"] = examples.progress_apply(
lambda x: llm_client.text_generation(
prompt=IMPROVED_JUDGE_PROMPT.format(question=x["question"], answer=x["answer"]),
max_new_tokens=500,
),
axis=1,
)
examples["llm_judge_improved_score"] = examples["llm_judge_improved"].apply(extract_judge_score)
>>> print("Correlation between LLM-as-a-judge and the human raters:")
>>> print(f"{examples['llm_judge_improved_score'].corr(examples['human_score'], method='pearson'):.3f}")
Correlation between LLM-as-a-judge and the human raters: 0.843
仅通过对提示进行一些调整(其中几个百分点是由于我对 LLM 的无耻提示,我在此声明该提示不具有法律约束力),相关性提高了近 30%。
非常令人印象深刻!👏
让我们显示 LLM judge 的一些错误来分析它们
errors = pd.concat(
[
examples.loc[examples["llm_judge_improved_score"] > examples["human_score"]].head(1),
examples.loc[examples["llm_judge_improved_score"] < examples["human_score"]].head(2),
]
)
display(
errors[
[
"question",
"answer",
"human_score",
"explanation_1",
"llm_judge_improved_score",
"llm_judge_improved",
]
]
)
分歧很小:总的来说,我们的系统似乎已经达到了良好的性能水平!
4. 我们如何进一步改进 LLM judge?
🎯 你永远不会达到 100%: 首先请注意,我们的人工真实情况肯定有一些噪音,因此即使使用完美的 LLM judge,一致性/相关性也永远不会达到 100%。
🧭 提供参考答案: 如果你可以访问每个问题的参考答案,你绝对应该在提示中将其提供给 Judge LLM,以获得更好的结果!
▶️ 提供少样本示例: 在提示中添加一些问题和真实情况评估的少样本示例可以提高结果。(我在这里尝试过,在这种情况下并没有提高结果,所以我跳过了它,但它可能适用于你的数据集!)
➕ 累加量表: 当判断可以分为原子标准时,使用累加量表可以进一步提高结果:请参阅下文 👇
ADDITIVE_PROMPT = """
(...)
- Award 1 point if the answer is related to the question.
- Give 1 additional point if the answer is clear and precise.
- Provide 1 further point if the answer is true.
- One final point should be awarded if the answer provides additional resources to support the user.
...
"""
使用结构化生成实现
使用结构化生成,你可以配置 LLM judge 以直接将其输出作为带有字段 Evaluation
和 Total rating
的 JSON 提供,这使得解析更容易:请参阅我们的 结构化生成 食谱以了解更多信息!
结论
今天就到这里,恭喜你坚持到了最后!🥳
我得先走了,一些怪人在敲我的门,声称他们代表 Mixtral 来收取 H100。🤔
< > 在 GitHub 上更新