TIL: GRPO will silently collapse if your SFT policy is too confident
Training the SlotKeeper manager, GRPO from my SFT checkpoint flatlined to 0.0167.
The cause was upstream: the SFT eval-loss had dropped to 0.0525, leaving the
policy near-deterministic. GRPO's advantage estimate needs variance across
sampled completions — if every sample is basically identical, the gradient is
~0 and nothing learns.
Low loss is not the goal. A policy you intend to RL on needs entropy left in it.
What recovered the score: a DPO + Best-of-N path on preference pairs mined from the teacher. Next time I'd rejection-sample against teacher trajectories first, then SFT on that higher-entropy distribution before touching RL.