让 LLM 更简单：从 SFT 到 GRPO

LLM NLP Deep Learning Reinforcement Learning AI Making LLMS Simpler: From SFT to GRPO Andre Tan 7 min read · Just now -- Listen Share In my previous article “Get LLMs To Say More With Less”, we established the following goal: To train a small language model to write simpler English without sacrificing correctness. Supervised fine-tuning (SFT) worked surprisingly well. With a relatively small dataset, the model learned to produce responses that were shorter, clearer, and closer to ASD-STE100-style technical English. But SFT also exposed a harder problem. Improving one behaviour can distort others. In our case, it sometimes made the model’s output distribution too sharp, leading to repetitive loops especially under greedy decoding strategies. This article is a more technical follow-up to the original post. We will look at the failure modes encountered after SFT, why common inference-time fixes only treat the symptoms, and how Group Relative Policy Optimisation (GRPO) was used to optimise the model against explicit rewards for correctness, simplicity, clarity, and ASD-STE100 compliance. SFT worked… Until it didn’t Supervised fine-tuning produced a meaningful shift in behaviour. Semantic simplicity improved from 65.82% to 85.46%, while technical adequacy slightly improved from 68.30% to 70.15%. The model also preserved most of its task-fulfilment and clarity scores. In other words, SFT largely achieved what we wanted: simpler responses without an obvious collapse in correctness. The problem appeared during generation. Under some prompts, especially with greedy decoding, the SFT model could fall into repetitive loops. For example: …from a single exit point from a single next hop from a single next hop from a single next hop… Once the model entered one of these loops, it often had difficulty escaping. This was not enough to invalidate the SFT result. Most generations were fine, and the aggregate evaluation metrics were strong. But it suggested that the behavioural shift we had learned came with an undesirable side effect. What went wrong? SFT trains the model to increase the probability of the target tokens in the training examples. At a high level, it minimises the negative log-likelihood: If the training data consistently contains shorter, simpler, more direct answers, the model learns to assign more probability to those kinds of continuations. But the objective does not directly care about the shape of the full output distribution. In our case, the looping behaviour was consistent with the fine-tuned model becoming too confident in a small set of continuations under certain contexts. This becomes especially problematic with greedy decoding: Greedy decoding always picks the single highest-probability next token. That works well when the distribution is well behaved, but it can amplify local pathologies. If one continuation produces a context that again strongly favours a similar continuation, the process can become self-reinforcing: context → high-probability continuation → similar context → same continuation → ⋯ The model had learned the right general behaviour, but in some local regions of the generation space, the learned update appeared to be too aggressive. A practical workaround Rather than retrain immediately, I first tried to make generation more robust at inference time. First, I moved away from purely greedy decoding and introduced stochastic decoding. Instead of always choosing the highest-probability token, sampling allows lower-ranked but still plausible tokens to be selected. This gives the model a way to escape deterministic cycles rather than repeatedly following the same local maximum. Second, I added repetition and presence penalties. These reduce the attractiveness of tokens or phrases that have already appeared in the generated text. They directly discourage the model from repeatedly returning to the same continuation. Third, I reduced the LoRA adapter scale. A LoRA-adapted layer can be written approximately as: Reducing s weakens the contribution of the adapter while keeping the underlying base model unchanged. This was particularly interesting because it suggested something about what the SFT adapter had learned. The direction of the update appeared useful: the adapter clearly pushed the model toward simpler language. But the full-strength update seemed too strong in some contexts. Get Andre Tan’s stories in your inbox Join Medium for free to get updates from this writer. Subscribe Subscribe Remember me for faster sign in Scaling the adapter down preserved much of the desired behaviour while reducing the tendency to fall into repetitive loops. But this was still unsatisfying These inference time fixes worked, but it felt like an operational hack. Stochastic decoding, repetition penalties, and adapter scaling were effectively acting as guardrails around the fine-tuned model. Ideally, the model should learn the desired behavioural shift without needing those guardrails. We could probably have squeezed out more performance from SFT by experimenting with a lower learning rate or earlier stopping. More diverse training data could also reduce the chances of over-specialising on a narrow set of response patterns. But I decided to try a slightly different approach. SFT only tells the model to imitate the examples it is given. What if we could instead tell the model directly which outputs were better? Enter, GRPO Group Relative Policy Optimisation (GRPO) is a reinforcement-learning method for post-training language models. The key difference from SFT is the learning signal. SFT learns by imitation: given a prompt and a target response, it increases the probability of the target tokens. GRPO does not require a single “correct” target response. Instead, it generates several candidate responses for the same prompt, scores them with a reward function, and updates the model to make higher-reward responses more likely relative to the rest of the group. This makes GRPO useful when the desired behaviour is easier to evaluate than to demonstrate perfectly in a dataset. A reward function with the wrong incentives The first reward function that we tried treated correctness as a multiplier over the style objectives: where C is correctness, S is simplicity, A is ASD-STE100 compliance, and the weights control the relative importance of the style objectives. The intuition was that incorrect answers should receive little or no reward, while correct answers should be rewarded for writing simply. In practice, however, correctness was something the model could improve just by adding more detail. Longer answers could include extra explanations and caveats that made them appear more technically complete, even when that detail was unnecessary. The result was a model that became more verbose instead of simpler. Compared with the SFT baseline, semantic simplicity fell from 85.46% to 77.25%, while average sentence length increased slightly from 15.32 to 15.85 words. Technical adequacy was essentially unchanged at 70.15% vs. 70.50%, and task fulfilment also stayed roughly flat. The multiplicative reward added optimisation pressure without improving correctness, while making the model noticeably less simple. Treating correctness as a constraint The second reward formulation treated correctness as a constraint, rather than an objective: The idea was simple: first be correct enough, then optimise for simplicity. Once a response passed the correctness threshold, adding more technically correct detail no longer increased its reward. The model instead had to compete on simplicity, ASD-STE100 compliance, and the other style objectives. The TensorBoard logs suggested that this produced a much healthier optimisation signal: overall reward improved over training, while correctness, simplicity, ASD-STE100 compliance, and clarity generally improved alongside it. Press enter or click to view image in full size The best model so far Compared with the previous SFT setup, semantic simplicity improved from 85.46% to 95.83%, while technical adequacy improved from 70.15% to 72.42%. Average sentence length fell from 15.32 to 11.80 words, and mean output length dropped from 622 to 331 tokens. Press enter or click to view image in full size The progression is clear. SFT produced the largest initial improvement in simplicity, while GRPO pushed it further and also achieved the highest technical adequacy. The final model wrote much shorter sentences and almost eliminated long sentences. This came with a modest trade-off in task fulfilment and clarity, suggesting that the reward function still has room for refinement. Importantly, the looping problem did not return, even with the GRPO adapter at full strength. We kept the same repetition and presence penalties for consistency, so GRPO alone should not get all the credit. Try it yourself If you wish to reproduce the results or explore the artifacts: GitHub — code and evaluation harness HF Dataset — GRPO training and validation data HF Model — trained GRPO LoRA adapter and model details