Why Domain Experts Matter for RLHF Data
Reinforcement learning from human feedback (RLHF) is only as good as the humans providing the feedback. That sounds obvious, but its implications are routinely underestimated.
The problem with generic annotation
Ask a generalist annotator to rank two model responses about a contract clause, a differential diagnosis, or a numerical methods bug, and you'll usually get a confident answer. The problem is what the answer is based on: surface features like length, formatting, tone, and fluency.
Models trained on those labels learn exactly those lessons. They get longer, more polished, and more confident — without getting more correct. This is one of the mechanisms behind reward hacking and sycophancy.
Where expertise changes the label
In our work across legal, medical, scientific, and engineering domains, expert and non-expert annotators frequently disagree on preference pairs — and the disagreements cluster in the cases that matter most:
- Plausible-but-wrong answers. The response reads well but contains a subtle factual or reasoning error only an expert catches.
- Dangerous omissions. Both responses are "correct," but one leaves out a caveat that a practitioner would consider essential.
- False equivalence. A generalist marks two responses as tied; an expert sees a clear winner.
These are precisely the labels that teach a model the difference between sounding right and being right.
Making expert annotation work at scale
Expertise alone isn't enough — it has to be operationalized:
- Vetting and calibration. Test experts on gold-standard tasks before they touch production data, then track agreement over time.
- Clear, evolving guidelines. Start with a pilot, mine the disagreements, and turn them into written policy.
- Layered quality control. Combine automated checks, peer review, and audit sampling rather than relying on any single gate.
- Feedback loops. Route reviewer findings back to annotators quickly, so quality improves instead of drifting.
The upfront cost is higher than crowdsourcing. The cost of retraining a model on bad labels is much higher.