开源 AI 食谱文档

使用 LLM-as-a-judge 🧑‍⚖️ 进行自动化和多功能评估

Hugging Face's logo
加入 Hugging Face 社区

并获得增强的文档体验

开始使用

Open In Colab

使用 LLM-as-a-judge 🧑‍⚖️ 进行自动化和多功能评估

作者:Aymeric Roucher

评估大型语言模型 (LLM) 通常是一项艰巨的任务:鉴于其广泛的功能,分配给它们的任务通常应根据非常广泛且定义松散的要求进行判断。例如,助手对问题的回答可能是

  • 没有根据上下文
  • 重复,重复,重复
  • 语法不正确
  • 过度冗长,并以过多的词语为特征,导致话语或书面内容变得过于详细和冗长
  • 语无伦次
  • ……

标准列表还在不断增加。即使我们有一个有限的列表,这些标准中的每一个都很难衡量:“设计一个基于规则的程序来评估输出是极其具有挑战性的。基于输出和参考答案之间相似性的传统评估指标(例如,ROUGE、BLEU)对于这些问题也无效。”

✅ 一种以人为方式评估输出的强大解决方案,无需花费昂贵的人工时间,就是 LLM-as-a-judge。这种方法在 Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena 中介绍 - 我鼓励您阅读。

💡 这个想法很简单:让 LLM 为你评分。🤖✓

但我们将看到,它不会开箱即用:你需要仔细设置它才能获得良好的结果。

!pip install huggingface_hub datasets pandas tqdm -q
import re
import pandas as pd
from tqdm.auto import tqdm
from datasets import load_dataset
from huggingface_hub import InferenceClient, notebook_login

tqdm.pandas()  # load tqdm's pandas support
pd.set_option("display.max_colwidth", None)

notebook_login()
repo_id = "mistralai/Mixtral-8x7B-Instruct-v0.1"

llm_client = InferenceClient(
    model=repo_id,
    timeout=120,
)

# Test your LLM client
llm_client.text_generation(prompt="How are you today?", max_new_tokens=20)

1. 准备创建和评估我们的 LLM judge

假设你想给 LLM 一个特定的任务,比如回答开放式问题。

困难在于,正如我们上面讨论的那样,衡量答案的质量很困难,例如,精确的字符串匹配会将太多正确但措辞不同的答案标记为错误。

你可以让人工标注员来判断输出,但这对于他们来说非常耗时,而且如果你想更新模型或问题,你必须重新做一遍。

✅ 在这种情况下,你可以设置一个 LLM-as-a-judge。

但是要使用 LLM-as-a-judge,你首先需要评估它对你的模型输出评分的可靠性。

➡️ 所以第一步将是... 创建一个人为评估数据集。但是你可以只对少数几个示例进行人工标注 - 大约 30 个就足以很好地了解性能。你将能够在每次想要测试你的 LLM-as-a-judge 时重复使用此数据集。

在我们的例子中,我们将使用 feedbackQA,它包含 2 个人为评估和每个问题/答案对的分数:使用 30 个示例的样本将代表你的小型评估数据集可能是什么样子。

ratings = load_dataset("McGill-NLP/feedbackQA")["train"]
ratings = pd.DataFrame(ratings)

ratings["review_1"] = ratings["feedback"].apply(lambda x: x["rating"][0])
ratings["explanation_1"] = ratings["feedback"].apply(lambda x: x["explanation"][0])
ratings["review_2"] = ratings["feedback"].apply(lambda x: x["rating"][1])
ratings["explanation_2"] = ratings["feedback"].apply(lambda x: x["explanation"][1])
ratings = ratings.drop(columns=["feedback"])

# Map scores to numeric values
conversion_dict = {"Excellent": 4, "Acceptable": 3, "Could be Improved": 2, "Bad": 1}
ratings["score_1"] = ratings["review_1"].map(conversion_dict)
ratings["score_2"] = ratings["review_2"].map(conversion_dict)

计算性能基线总是一个好主意:在这里,它可以是例如两个人工评分者之间的一致性,通过他们给出的分数的 皮尔逊相关性 来衡量。

>>> print("Correlation between 2 human raters:")
>>> print(f"{ratings['score_1'].corr(ratings['score_2'], method='pearson'):.3f}")
Correlation between 2 human raters:
0.563

这两个人工评分者之间的相关性不是很好。如果你的人工评分真的很差,则可能意味着评分标准不够清晰。

这意味着我们的“真实情况”包含噪声:因此我们不能期望任何算法评估都能非常接近它。

但是,我们可以减少这种噪声

  • 通过取平均分作为我们的真实情况,而不是任何单一分数,我们应该消除一些不规则性。
  • 仅选择人工审核员意见一致的样本。

在这里,我们将选择最后一个选项,只保留 2 个人工审核员意见一致的示例

# Sample examples
ratings_where_raters_agree = ratings.loc[ratings["score_1"] == ratings["score_2"]]
examples = ratings_where_raters_agree.groupby("score_1").sample(7, random_state=1214)
examples["human_score"] = examples["score_1"]

# Visualize 1 sample for each score
display(examples.groupby("human_score").first())

2. 创建我们的 LLM judge

我们使用基本提示构建我们的 LLM judge,其中包含以下元素

  • 任务描述
  • 量表描述:minimummaximum、值类型(此处为 float
  • 输出格式的说明
  • 答案的开头,尽可能引导 LLM
JUDGE_PROMPT = """
You will be given a user_question and system_answer couple.
Your task is to provide a 'total rating' scoring how well the system_answer answers the user concerns expressed in the user_question.
Give your answer as a float on a scale of 0 to 10, where 0 means that the system_answer is not helpful at all, and 10 means that the answer completely and helpfully addresses the question.

Provide your feedback as follows:

Feedback:::
Total rating: (your rating, as a float between 0 and 10)

Now here are the question and answer.

Question: {question}
Answer: {answer}

Feedback:::
Total rating: """
examples["llm_judge"] = examples.progress_apply(
    lambda x: llm_client.text_generation(
        prompt=JUDGE_PROMPT.format(question=x["question"], answer=x["answer"]),
        max_new_tokens=1000,
    ),
    axis=1,
)
def extract_judge_score(answer: str, split_str: str = "Total rating:") -> int:
    try:
        if split_str in answer:
            rating = answer.split(split_str)[1]
        else:
            rating = answer
        digit_groups = [el.strip() for el in re.findall(r"\d+(?:\.\d+)?", rating)]
        return float(digit_groups[0])
    except Exception as e:
        print(e)
        return None


examples["llm_judge_score"] = examples["llm_judge"].apply(extract_judge_score)
# Rescale the score given by the LLM on the same scale as the human score
examples["llm_judge_score"] = (examples["llm_judge_score"] / 10) + 1
>>> print("Correlation between LLM-as-a-judge and the human raters:")
>>> print(f"{examples['llm_judge_score'].corr(examples['human_score'], method='pearson'):.3f}")
Correlation between LLM-as-a-judge and the human raters:
0.567

考虑到 2 个随机独立变量之间的皮尔逊相关性为 0!这还不错!

但我们可以轻松做得更好。🔝

3. 改进 LLM judge

正如 Aparna Dhinakaran 所展示的那样,LLM 不擅长在连续范围内评估输出。本文 为我们提供了构建更好提示的一些最佳实践

  • 留出更多思考时间,在最终答案之前添加一个 Evaluation 字段。
  • 🔢 使用小的整数量表,如 1-4 或 1-5,而不是我们之前使用的大浮点数量表。
  • 👩‍🏫 提供指示性量表以进行指导
  • 我们甚至添加了一个胡萝卜来激励 LLM!
IMPROVED_JUDGE_PROMPT = """
You will be given a user_question and system_answer couple.
Your task is to provide a 'total rating' scoring how well the system_answer answers the user concerns expressed in the user_question.
Give your answer on a scale of 1 to 4, where 1 means that the system_answer is not helpful at all, and 4 means that the system_answer completely and helpfully addresses the user_question.

Here is the scale you should use to build your answer:
1: The system_answer is terrible: completely irrelevant to the question asked, or very partial
2: The system_answer is mostly not helpful: misses some key aspects of the question
3: The system_answer is mostly helpful: provides support, but still could be improved
4: The system_answer is excellent: relevant, direct, detailed, and addresses all the concerns raised in the question

Provide your feedback as follows:

Feedback:::
Evaluation: (your rationale for the rating, as a text)
Total rating: (your rating, as a number between 1 and 4)

You MUST provide values for 'Evaluation:' and 'Total rating:' in your answer.

Now here are the question and answer.

Question: {question}
Answer: {answer}

Provide your feedback. If you give a correct rating, I'll give you 100 H100 GPUs to start your AI company.
Feedback:::
Evaluation: """
examples["llm_judge_improved"] = examples.progress_apply(
    lambda x: llm_client.text_generation(
        prompt=IMPROVED_JUDGE_PROMPT.format(question=x["question"], answer=x["answer"]),
        max_new_tokens=500,
    ),
    axis=1,
)
examples["llm_judge_improved_score"] = examples["llm_judge_improved"].apply(extract_judge_score)
>>> print("Correlation between LLM-as-a-judge and the human raters:")
>>> print(f"{examples['llm_judge_improved_score'].corr(examples['human_score'], method='pearson'):.3f}")
Correlation between LLM-as-a-judge and the human raters:
0.843

仅通过对提示进行一些调整(其中几个百分点是由于我对 LLM 的无耻提示,我在此声明该提示不具有法律约束力),相关性提高了近 30%

非常令人印象深刻!👏

让我们显示 LLM judge 的一些错误来分析它们

errors = pd.concat(
    [
        examples.loc[examples["llm_judge_improved_score"] > examples["human_score"]].head(1),
        examples.loc[examples["llm_judge_improved_score"] < examples["human_score"]].head(2),
    ]
)

display(
    errors[
        [
            "question",
            "answer",
            "human_score",
            "explanation_1",
            "llm_judge_improved_score",
            "llm_judge_improved",
        ]
    ]
)

分歧很小:总的来说,我们的系统似乎已经达到了良好的性能水平!

4. 我们如何进一步改进 LLM judge?

🎯 你永远不会达到 100%: 首先请注意,我们的人工真实情况肯定有一些噪音,因此即使使用完美的 LLM judge,一致性/相关性也永远不会达到 100%。

🧭 提供参考答案: 如果你可以访问每个问题的参考答案,你绝对应该在提示中将其提供给 Judge LLM,以获得更好的结果!

▶️ 提供少样本示例: 在提示中添加一些问题和真实情况评估的少样本示例可以提高结果。(我在这里尝试过,在这种情况下并没有提高结果,所以我跳过了它,但它可能适用于你的数据集!)

累加量表: 当判断可以分为原子标准时,使用累加量表可以进一步提高结果:请参阅下文 👇

ADDITIVE_PROMPT = """
(...)
- Award 1 point if the answer is related to the question.
- Give 1 additional point if the answer is clear and precise.
- Provide 1 further point if the answer is true.
- One final point should be awarded if the answer provides additional resources to support the user.
...
"""

使用结构化生成实现

使用结构化生成,你可以配置 LLM judge 以直接将其输出作为带有字段 EvaluationTotal rating 的 JSON 提供,这使得解析更容易:请参阅我们的 结构化生成 食谱以了解更多信息!

结论

今天就到这里,恭喜你坚持到了最后!🥳

我得先走了,一些怪人在敲我的门,声称他们代表 Mixtral 来收取 H100。🤔

< > 在 GitHub 上更新