Preference data: how pairwise comparisons are collected

Every preference dataset is a long list of small decisions: this response, not that one. How those decisions are set up decides whether the list teaches your model anything.

Where do the prompts come from?

Prompts are the first design choice and the one vendors control least. There are three usual sources:

• Your own traffic, filtered and stripped of anything identifying. This matches what users really ask, which is why labs prize it.
• Prompts written by the vendor’s raters to a brief: “write a hard question about SQL window functions”, “ask for a refusal-worthy request phrased politely”. Useful for covering gaps.
• Public or synthetic prompt sets, quick to start with but often already familiar to strong base models.

State the mix in the brief, and say which categories matter most. A dataset that is 80 percent casual chat will not move a coding model.

How are the two responses chosen?

The pair is usually sampled from your current model, sometimes alongside an older checkpoint or a different sampling temperature, so the two answers differ in ways worth judging. If the responses are nearly identical, raters guess. If one is obviously broken, the comparison teaches little. The useful zone is two plausible answers with a real difference.

InstructGPT went further: labelers ranked several outputs per prompt at once, and every pair within a ranking became a training comparison. Ranking four responses yields six pairs for about the effort of reading four answers, though long rankings tire raters and agreement drops.

What does the rater actually see?

A good comparison screen shows:

1. The prompt, plus any earlier turns of the conversation.
2. Two responses, labeled neutrally (A and B), with left and right order randomized per item.
3. The rubric, or a link to it, with the version number.
4. The questions: which is better, by how much, and why.

Raters should not see which model produced which answer. Order randomization matters more than it sounds: tired raters drift toward the first-shown option, and without randomization that drift becomes a bias in your reward model.

How is “how much better” recorded?

Meta’s Llama 2 paper is a well-documented example. It collected more than a million binary comparisons, asked annotators to rate the margin on a four-level scale (from significantly better down to negligibly better or unsure), and kept helpfulness and safety as separate judgments trained into separate reward models.

How does a reward model use the pairs?

Most reward models assume the Bradley-Terry model from 1952: the chance that response A is preferred to B depends on the gap between their scores.

P(A ≻ B) = σ(r(A) − r(B))

Here r is the reward model’s score and σ the logistic function. Training pushes r(chosen) above r(rejected) on every pair. DPO uses the same formula but folds the reward into the language model itself, which is why it can train on the pairs without a separate reward model.

The practical consequence: the model only learns differences. If both responses in most pairs share a flaw, nothing in the data teaches the model to avoid it.

What should the delivered record contain?

Ask the vendor to deliver one JSON line per comparison with at least:

• the prompt and full conversation context;
• both responses, with which was chosen;
• the strength score and any tie flag;
• the written reason, if collected;
• a stable, anonymous rater ID;
• the rubric version and a timestamp;
• whether the item was a gold task or overlapped with another rater.

The rater ID and overlap flag look like admin, but they are what let you measure agreement, find a rater whose labels drift, and re-weight or drop their work later. Our quality controls page sets out which figures to compute from them.

How many comparisons do you need?

It depends on the base model, the domain and how far behavior has to move. Published projects range from tens of thousands of comparisons to Llama 2’s million-plus. For a first vendor engagement, size a pilot to answer two questions: do raters agree often enough on your task, and does a reward model trained on the pilot rank held-out pairs better than chance? Scale only after both answers are yes. The vendor table on the RLHF companies home page lists who supplies this kind of work.

A pair is cheap; a pair you can trust is the product.