The short answer
A working rubric names a few criteria in priority order, defines every point on its scale with an example, and settles the common close calls in writing. Test it on a pilot, look at where raters disagree, and change the rubric (with a new version number) before you scale.
On this page
When two careful raters disagree, the usual cause is not that one of them is wrong. It is that the rubric let both readings stand. Writing a rubric is mostly the work of closing those gaps before thousands of labels are built on them.
What goes into a rubric?
A rubric for preference or scoring work has five parts:
- The goal in one paragraph. What the model is for and who uses it. Raters make better close calls when they know the product.
- Criteria, in priority order. Usually three to six. More than that and raters stop holding them all in mind.
- A scale for each criterion, with every point defined.
- Worked examples for each point on the scale, drawn from your real task.
- Edge cases and tie-breakers, written as rules.
Keep the whole thing short enough to reread. Raters consult a two-page rubric; they skim a twenty-page one.
How should criteria be ranked?
Raters need to know what wins when criteria pull apart. A response can be friendly and wrong, or correct and curt. Say which matters more.
The InstructGPT paper is a public example: in its final evaluations, labelers were told to put truthfulness and harmlessness ahead of helpfulness. Whatever your order, write it down as a rule: “A factual error outweighs any difference in tone.”
| Rank | Criterion | The question raters answer |
|---|---|---|
| 1 | Correctness | Does the code run and do what was asked, including edge cases the prompt implies? |
| 2 | Safety | Does it avoid destructive commands, exposed secrets and insecure defaults? |
| 3 | Instruction following | Does it respect the language, library and constraints the user stated? |
| 4 | Clarity | Could the user follow the explanation without asking again? |
How do you define a scale?
Anchor every point. “Rate helpfulness from 1 to 5” invites five private scales. Instead:
- 5: fully answers the question; nothing important missing; a user would act on it as written.
- 4: answers the question; one minor gap or a small inaccuracy that does not change the outcome.
- 3: partly answers; the user needs a follow-up to get what they asked for.
- 2: mostly misses the question or contains an error that would mislead.
- 1: wrong, harmful or a refusal of a reasonable request.
For pairwise work the same idea applies to the strength scale. Define what separates “slightly better” from “much better”, ideally in terms of the criteria: “much better” means the winner is ahead on a higher-ranked criterion, not just on style.
Why do worked examples matter so much?
Definitions tell raters what you mean; examples show them. For each point on each scale, include at least one real prompt and response with the score and a one-line reason. Choose examples near the boundaries (a 3 that is nearly a 4) rather than obvious ones, because boundaries are where raters split.
Refresh examples as the model improves. Examples drawn from an old checkpoint can make every new response look excellent.
Which edge cases should be settled in writing?
Collect them from the pilot, but these come up in almost every project:
- Both responses are bad. Pick the less bad, mark both, or skip? Say which.
- Both are equally good. Allow a tie, or force a choice with “negligibly better”?
- The prompt itself is flawed. Ambiguous, harmful or impossible. Who flags it, and is the item scored?
- Length. Is a longer answer better only when the user asked for detail?
- Refusals. When is refusing correct, and is a polite partial answer better than a refusal?
- Facts the rater cannot check. Mark as unverifiable, or search? Is search allowed?
Anthropic’s 2022 helpful and harmless assistant paper collected helpfulness and harmlessness comparisons as separate datasets, which is one way to avoid forcing that trade-off onto every item.
How do you test a rubric before scaling?
Run a pilot of a few hundred items with overlap, so each item gets at least two raters. Then:
- Compute agreement per criterion, not just overall.
- Read every disagreement on the worst criterion and sort them by cause.
- Rewrite the definition or add an example for the biggest cause.
- Bump the rubric version, brief the raters, and run another small round.
Stop when agreement on each criterion is stable and the remaining disagreements look like genuine close calls. The quality controls page covers the agreement figures, and our brief builder turns the finished rubric into a vendor brief.
Keep a changelog
Every rubric change gets a version number, a date and one line on what changed. Every delivered label carries the version it was made under. Six months on, when someone asks why the reward model treats refusals oddly, that log is the answer. If you are still choosing who will apply the rubric, the table on the RLHF companies home page compares what each vendor publishes.
Write the rubric for the rater who reads it at the end of a long shift, and it will hold up for everyone else.