🚀 重现 Search-R1 之旅

申承允 · 2025

我刚刚完成了一个深入的项目，旨在重现并略微超越原始的 Search-R1 检索增强方法。在此过程中，我发现了一个关键见解：**使用反思性推理短语（例如“再想想”、“重新验证”）与基准性能的提高呈强比例关系**。以下是我的旅程 👇

---

🛠 我做了什么

• 在 VERL 内部，使用 **GRPO** 对 **Qwen 2.5-3B-Instruct** 在 **维基百科语料库** 上进行了 RL 调优。
• 实施并跟踪了反思性推理短语（例如“再想想”、“重新验证”）的使用情况。
• **在 Weights & Biases 中记录了每一次运行，以实现完全透明。** wandb 报告

---

📊 结果 (Pass@1)

检索到的段落**从损失中被屏蔽**，因此模型学习何时搜索而不是记忆答案。

**这项工作**的平均得分达到 0.349，比 RAG 基线（0.270）提高了约 **1.29 倍**。值得注意的是，它也略优于原始的 search-r1 模型。

---

🧠 推理风格

<think> → <tool_call> → <tool_response> → <answer>

---

✅ 反思行为

**反思短语**（例如“再想想”、“重新验证”、“让我们重新考虑”）在训练过程中逐渐增加使用频率。此图比较了 7 个基准（**NQ、TriviaQA、PopQA、Musique、HotpotQA、Bamboogle 和 2wikimultihopqa**）的平均性能得分与反思短语比率。值得注意的是，反思短语比率与平均得分显示出**强烈的正向关系，表明更多的反思性语言对应着更好的基准性能**。

你可以观察到诸如**“经仔细检查”**和**“再想想”**等自我验证语句，显示了模型在回答前重新评估自身主张的内在倾向。

---

💻 完整推理脚本（放在权重文件旁并运行）

#!/usr/bin/env python3
"""
Minimal multi-turn tool-calling demo for the Qwen2.5-3B Search-R1 model
"""
from __future__ import annotations
import json, re, sys
from typing import List
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
from duckduckgo_search import DDGS

SYS = "You are a helpful and harmless assistant."
USR_PRE = ("Answer the given question. You must conduct reasoning inside <think> and "
           "</think> first every time you get new information. After reasoning, if you "
           "lack knowledge, call a search engine by <tool_call> query </tool_call>. "
           "It returns results between <tool_response> and </tool_response>. "
           "Provide the final answer inside <answer>. For example, <answer> Beijing </answer>. Question: ")

MODEL = "Seungyoun/qwen2.5-3b-it_searchR1-like-multiturn"
MAX_TURNS, MAX_NEW = 4, 512
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

SEARCH_SCHEMA = {
    "type": "function",
    "function": {
        "name": "search",
        "description": "DuckDuckGo web search",
        "parameters": {
            "type": "object",
            "properties": {
                "query_list": {
                    "type": "array",
                    "items": {"type": "string"},
                    "description": "Fully-formed semantic queries."
                }
            },
            "required": ["query_list"],
        },
    },
}

def prompt(q: str) -> List[dict]:
    return [{"role": "system", "content": SYS},
            {"role": "user", "content": USR_PRE + q}]

def ddg(q: str, k: int = 5) -> str:
    with DDGS() as ddgs:
        hits = list(ddgs.text(q, safesearch="moderate", max_results=k))
    return "\n".join(f"{i+1}. {h['title']} – {h['body']} ({h['href']})" for i, h in enumerate(hits))

def parse_queries(raw: str) -> List[str]:
    try:
        p = json.loads(raw)
        if p.get("name") == "search":
            return p.get("arguments", {}).get("query_list", [])
    except json.JSONDecodeError:
        pass
    return [raw]

def main() -> None:
    q = sys.argv[1] if len(sys.argv) > 1 else "How is the weather in Seoul?"
    tok = AutoTokenizer.from_pretrained(MODEL, padding_side="left")
    model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16, device_map="auto")
    hist = tok.apply_chat_template(prompt(q), tools=[SEARCH_SCHEMA], add_generation_prompt=True, tokenize=False)
    pat = re.compile(r"<tool_call>\s*(.*?)\s*</tool_call>", re.S)

    for turn in range(MAX_TURNS):
        enc = tok(hist, return_tensors="pt").to(DEVICE)
        out = model.generate(**enc, max_new_tokens=MAX_NEW, temperature=0.7, do_sample=True)
        delta = tok.decode(out[0][enc.input_ids.shape[1]:], skip_special_tokens=True)
        print(f"\n===== Assistant (turn {turn+1}) =====\n{delta}\n")
        hist += delta
        m = pat.search(delta)
        if not m: break
        results = "\n---\n".join(ddg(s, 5) for s in parse_queries(m.group(1)))
        hist += f"<tool_response>\n{results}\n</tool_response>"

if __name__ == "__main__":
    main()

这将得到

---

🔗 亲自尝试

权重**+**脚本：https://huggingface.co/Seungyoun/qwen2.5-3b-it_searchR1-like-multiturn

---

🙏 致谢

Qwen 团队、Search-R1 作者、VERL 维护者以及使重现成为可能的所有开源社区。