A Practical Checklist for SFT Dataset Quality
Supervised fine-tuning (SFT) datasets fail in predictable ways. Here are the seven checks we run on every dataset before delivery, and what each one catches.
1. Instruction diversity
Cluster your prompts by embedding and look at the distribution. If 30% of your "coding" dataset is variations of reverse a string, your model will be excellent at reversing strings and mediocre at everything else. Aim for coverage of task types, difficulty levels, and phrasings.
2. Response correctness
Every response should be verified by someone qualified to verify it. For code, that means it runs. For math, the answer checks out. For domain content, a practitioner has reviewed it. "Looks right to a generalist" is not a bar.
3. Format consistency
Inconsistent formatting — sometimes markdown, sometimes not; sometimes with preamble, sometimes without — teaches the model that format is random. Decide on conventions and enforce them mechanically where possible.
4. Length distribution
Check the histogram of response lengths. A dataset dominated by long responses trains verbosity; one dominated by short responses trains laziness. The right distribution depends on the task mix, but you should know what yours is.
5. Refusal balance
If the dataset includes safety refusals, audit both directions: harmful prompts that should be refused, and benign prompts that superficially resemble them and should be answered. Over-refusal is a data problem before it's a model problem.
6. Contamination and duplication
Deduplicate near-identical examples and screen against your eval sets. Nothing invalidates a benchmark faster than training on it.
7. Provenance
For every example, you should be able to answer: who wrote this, under what guidelines, and who reviewed it? Untraceable data is unfixable data — when a quality issue surfaces later, provenance is what lets you scope and repair it.
None of these checks is exotic. The discipline is in running all of them, every time, and treating failures as blocking. If you'd like help building or auditing an SFT dataset, talk to us.