GRPO Objective Explorer

Per-sample objective min(ρA, clip(ρ, 1−ε, 1+ε) A) − β·KL — drag the parameters and watch where the gradient lives, where clipping kills it, and how group statistics set the advantage scale.

Objective parameters

clip ε

Width of the trust region around ρ = 1

KL weight β

k₃ estimator: KL ≈ r − log r − 1, r = π_ref(o | q)π_θ(o | q) = cρ

ref offset c

c = π_ref(o | q)π_old(o | q) — drift already accumulated between reference and rollout policy; c = 1 ⇒ π_ref = π_old

Group reward statistics

group size G

# correct k

Binary GSM8K-style reward: k of G completions correct

Total objective vs. importance ratio ρ

min(ρA, clip(ρ, 1±ε) A) − β(r − log r − 1), r = c / ρ. Shaded band = clip interval; flat ⇒ zero gradient.