Supervised Fine-Tuning: How to Teach a Base Model Good Manners
You recruit a sixteen-year-old with insane aim into your esports roster.
In solo matches, he crushes everyone. But in scrims, it is a mess: he peeks alone, screams into the mic, and ignores callouts. He knows how to shoot, but not how to play with a team.
The coach sits him down and pulls up a match replay: “Hear footsteps? Call two B, drop smoke, and hold the angle. Do it again.”
The kid repeats the drill fifty times until muscle memory takes over.
Every conversational AI goes through that exact bootcamp before it reaches your screen.
The Reality
A base model read the whole internet. It knows physics, history, and code. But it does not know you want help.
To a base model, text is just a game: predict the next word.
Type “Find the bug in my code”, and it might help. Or it might drop a snarky forum rant or an ad. Online, people argue over code far more often than they solve it. The model cannot tell the difference; it just predicts the next token.
To teach the model how to act, engineers use Supervised Fine-Tuning (SFT).
They show it thousands of clean dialogues: a user asks a question, an engineer answers it. The loss is calculated only on the answer. The model picks up the pattern and starts talking like a colleague instead of a forum troll.
The Why
In 2023, Meta researchers published the LIMA study. They showed that just 1,000 curated dialogues were enough to turn a base model into a solid assistant.
The model did not learn physics or law from those 1,000 examples. It already knew them. SFT just gave it a uniform and taught it how to format answers.
Where does this break down?
The imitation trap.
When a model learns by copying humans, it inherits their flaws.
The people who wrote the training answers were paid to be thorough. They almost never wrote “I do not know” or “Your question makes no sense.”
So the model copies the posture of an expert. When it hits a question it cannot answer, it invents facts with that same calm tone. It mimics confidence, not truth.
To get past this ceiling, copying is not enough. The model needs a place where it can test ideas, take penalties, and learn from mistakes. That is where reward models and reinforcement learning take over.
The Takeaway
Imitation teaches an AI manners, but copying humans only takes a system as far as human blind spots.
AI specialists call it: Supervised Fine-Tuning (SFT) / Instruction Tuning / Superficial Alignment Hypothesis
SFT fine-tunes a base model on prompt-response pairs. Loss is calculated only on response tokens: Loss = -Σ log P(y_t | x, y_<t). Under the Superficial Alignment Hypothesis, pre-training provides all knowledge, while SFT merely unlocks tone and persona.
💬 Have you ever copied a mentor so closely that you picked up their mistakes along with their skills?
Part 13 of 15 | #ReinforcementLearningForHumans
#ai_edu Based on Stanford and industry lectures