Quality controls for human feedback data

RLHF quality controls show whether raters apply your rubric consistently: agreement between raters on shared items, accuracy on gold tasks with known answers, and audits where a reviewer re-scores a sample. Ask any vendor to report these per criterion and per week, from the pilot onward.

How is rater agreement measured?

Give the same item to two or more raters and count how often they match. Raw percent agreement is easy to read but flatters easy tasks: on a binary choice, two raters guessing at random still agree half the time.

Cohen's kappa corrects for that chance agreement. With observed agreement p_o and the agreement expected by chance p_e:

κ = (p_o − p_e) / (1 − p_e)

A kappa of 0 means no better than chance and 1 means perfect agreement. Landis and Koch's widely used bands call 0.41 to 0.60 moderate and 0.61 to 0.80 substantial. For rubric scores from a rotating pool, with gaps and ordinal scales, Krippendorff's alpha is the better fit.

Preference data rarely reaches the agreement levels of object labeling. In OpenAI's InstructGPT work, trained labelers agreed with each other about 73% of the time on which output was better. Low agreement on a criterion is often a rubric problem, not a rater problem.

What are gold tasks?

Gold tasks (also called honeypots or test questions) are items with an answer your team has fixed in advance, mixed into the normal stream without being marked. Accuracy on gold measures whether a rater applies the rubric as you meant it. Vendors use gold to qualify new raters and to remove raters whose accuracy drops.

For preference work, good gold items have a clear winner for a stated reason: a factual error, a broken code block, an instruction ignored. Close calls make poor gold, because they test taste rather than the rubric.

How do rubric audits work?

A senior reviewer, or someone on your team, re-scores a random sample of finished items and records every overturn against the rubric criterion involved. Overturns that cluster on one criterion point to an unclear instruction, and the fix is a rubric change with a new version number and a short calibration session with the raters.

Ask that every delivered record carries the rubric version it was made under. Without that, labels made before and after a change are mixed together and you cannot separate them later.

Which rater-level checks catch bad data?

Position bias: a rater who picks the first-shown response far more than half the time. Randomize order and track it.
Length bias: a steady preference for the longer answer even when the rubric does not ask for detail.
Speed outliers: times per task far below the median, which usually means the response was not read in full.
AI-written judgments: written reasons that read as generated text. If raters must not use AI tools, say so in the contract and ask how it is checked.

What should a vendor report each week?

These are the figures worth asking for in the statement of work. A vendor that cannot produce them during the pilot will not produce them at scale.

Quality figures to request from a human feedback vendor
Measure	What it shows	Ask for
Inter-rater agreement	How often two raters pick the same response, on items given to both	Raw percent and a chance-corrected figure, by criterion and by week
Gold-task accuracy	How often raters match answers your team has fixed in advance	Per rater, with the share of gold items in the stream
Reviewer overturn rate	Share of a sample a senior reviewer re-scores differently	Sample size, and which rubric criteria are overturned most
Position balance	Whether raters favor response A or B whatever the content	Share of picks for the first-shown response, per rater
Time per task	Rushed or stalled work	Median and spread per task type; flag raters far below the median
Rubric version	Which instructions each label was made under	A version id on every delivered record

We do not test or audit vendors. This list is a buyer's checklist, not a rating of anyone.

Questions teams ask

What is a good inter-rater agreement for preference data?

There is no single target, because many comparisons are genuinely close. In the InstructGPT paper, trained labelers agreed with each other about 73% of the time on pairwise judgments. Agree on a baseline in the pilot, then watch whether it holds as volume grows.

Is Cohen's kappa or Krippendorff's alpha better?

Kappa compares two raters on the same items. Alpha handles any number of raters, missing ratings and ordinal scales, which suits rubric scores collected from a rotating pool. Many teams report both alongside raw agreement.

How many items should overlap between raters?

Enough to measure agreement per criterion and per rater with some confidence. A common pattern is to send a fixed share of items, often 5 to 20 percent, to two or more raters, and to raise it during the pilot.

Should we write the gold tasks ourselves?

Yes, where you can. Gold items written by your own researchers test whether raters apply your rubric, not the vendor's reading of it. Refresh them so raters cannot learn them.

Do you audit vendors?

No. We do not test, audit or rate vendors. This page explains the controls to ask for; the vendor table reports only what each vendor publishes.

Sources and dates

Ouyang et al., "Training language models to follow instructions with human feedback" (InstructGPT), arXiv 2203.02155, 2022: labeler agreement figure.
Landis and Koch (1977), Biometrics, vol. 33: the descriptive kappa bands.
Jacob Cohen (1960), Educational and Psychological Measurement, vol. 20: origin of the kappa statistic.