Blog
Working notes on buying human feedback: what each training method asks raters to do, how preference pairs are collected, what rater pay signals and how to write a rubric that two people apply the same way.
Rubrics
4 min readHow to write rubrics for a human feedback project
A working rubric names a few criteria in priority order, defines every point on its scale with an example, and settles the common close calls in writing. Test it on a pilot, look at where raters disagree, and change the rubric (with a new version number) before you scale.
Data
4 min readPreference data: how pairwise comparisons are collected
Preference data is collected by showing a rater one prompt and two model responses in random order, and asking which is better against a written rubric. Most projects also record how much better, a short reason and the rater's ID, so the pairs can train a reward model or be used directly with DPO.
Pricing
4 min readRater pay and quality: what the rates tell you
Rater pay tells you who is likely doing your task and how they are pushed to work. The few published figures run from Prolific's $8 hourly minimum for participants to the up-to-$150 hourly figure Alignerr states for experts. Ask any vendor what raters earn, whether pay is per task or per hour, and how quality affects it.
Methods
4 min readRLHF vs DPO vs RLAIF: what are you paying people to do?
RLHF and DPO both need the same raw material from people: judgments of which response is better. RLHF turns those judgments into a reward model and then runs reinforcement learning; DPO trains on the pairs directly. RLAIF swaps most of the human labels for a model's, so the people you pay write principles, build test sets and audit the machine's labels instead.
Get a quote
Sketch the feedback work in about a minute. What comes back is one vendor name, not a round of cold calls.
RLHF Companies introduces teams to vendors, and the vendor can pay us for that introduction. Sending a brief costs your team nothing.
Keep prompts, model outputs, datasets and samples out of this form; describe the work in words instead.
Received. Here is what happens next.
- Your brief is compared against the services and terms each vendor states on its own pages.
- An email names the one vendor that looks like the best fit, with the reason.
- Reply yes and they receive your brief and contact details; reply no and nothing moves. Any pilot, quote or contract is agreed directly between your team and theirs.