TRL

论文索引

此部分正在建设中。欢迎贡献！

组序列策略优化 (Group Sequence Policy Optimization)

📜 论文: https://huggingface.co/papers/2507.18071

GSPO 是 GRPO 的一种变体，它在序列级别而不是每个标记级别计算重要性采样权重。要复现论文中的设置，请使用此配置

from trl import GRPOConfig

training_args = GRPOConfig(
    importance_sampling_level="sequence",
    loss_type="grpo",
    beta=0.0,  # GSPO set kl regularization to zero: https://github.com/volcengine/verl/pull/2775#issuecomment-3131807306 
    epsilon=3e-4,  # GSPO paper (v2), section 5.1
    epsilon_high=4e-4,  # GSPO paper (v2), section 5.1
    gradient_accumulation_steps=1,
    steps_per_generation=4,  # partition rollout batch into 4 mini-batches. GSPO paper (v2), section 5.1. Must be 4 times gradient_accumulation_steps
)

< > 在 GitHub 上更新