Write an RLHF vendor brief
A good RLHF brief tells a vendor what raters judge, who the raters must be, how judgments are scored and how many you need. Tick what applies below and the brief writes itself; you can copy it or send it with the quote form.
What vendors ask first
Most scoping calls open with the same four questions. Having answers ready shortens the call and makes quotes comparable.
- How long does one task take? A two-response comparison on a short chat turn is minutes; a rubric review of a long agent trace can take an hour. Time per task sets the price more than anything else.
- Who is qualified to judge? Generalists can judge tone and instruction following. Code, math, law and science need people who can verify the answer, and they cost more.
- What counts as good enough? Name the measure: agreement between raters, accuracy on gold tasks you supply, or both. See quality controls.
- What comes back, and in what format? Pairs with a label, a strength score, a written reason, rewritten responses, or per-criterion scores, delivered as JSONL or through the vendor's platform.
For the reasoning behind each choice, the rubric post walks through criteria, scales and worked examples.
Questions about briefs
What should an RLHF vendor brief include?
The kind of judgment (pairwise preference, rubric score, written demonstration or red teaming), who must make it, the criteria and their priority, task length, volume for a pilot and for production, languages, the quality figure you expect (an agreement target or gold-task accuracy), timing and how data comes back to you.
Should the brief include sample prompts?
Describe them here, and share real samples only with a vendor under an NDA you have signed. Do not send prompts, outputs or data through this site; we never handle them.
How big should a pilot be?
Large enough to measure agreement and turnaround on your own task. Teams often pilot several hundred to around two thousand items before committing; the right number depends on how many criteria and task types you need to test.
Do I need to know the price before I brief a vendor?
No. Vendors price after they see the task, because time per task and rater expertise drive most of the cost. The pricing models page shows the structures they use and the figures some publish.
Get a quote
Sketch the feedback work in about a minute. What comes back is one vendor name, not a round of cold calls.
RLHF Companies introduces teams to vendors, and the vendor can pay us for that introduction. Sending a brief costs your team nothing.
Keep prompts, model outputs, datasets and samples out of this form; describe the work in words instead.
Received. Here is what happens next.
- Your brief is compared against the services and terms each vendor states on its own pages.
- An email names the one vendor that looks like the best fit, with the reason.
- Reply yes and they receive your brief and contact details; reply no and nothing moves. Any pilot, quote or contract is agreed directly between your team and theirs.