Human Evaluation Protocol Template

Task and rubric, rater recruitment and training, ethics and consent, sample size, blinding, Cohen's/Fleiss' kappa, Krippendorff's alpha, QC, analysis

AI 与数据完整Human Evaluation Datasheet (HumEval 2022); van der Lee et al. 2021human evaluation protocol math更新于 2026年10月22日

在编辑器中打开下载 .md

使用方法: Use it when people will rate or compare model outputs and the result will inform a release or a paper. Fill the rubric, the ethics and consent plan and the sample size first, then pilot before the main study. The main pitfall is reporting mean ratings without inter-annotator agreement, rater details and sample size.

预览

Human Evaluation Protocol: [Study name]

Protocol for a human evaluation of model or system outputs, written before data collection: the task and rubric, rater recruitment and training, ethics and consent, the sample design and size, blinding and randomisation, inter-annotator agreement (Cohen's kappa, Fleiss' kappa, Krippendorff's alpha), quality control, the analysis plan and the reporting checklist. Reporting items follow the Human Evaluation Datasheet (Shimorina and Belz, HumEval 2022) and the best-practice recommendations of van der Lee et al. (Computer Speech and Language, 2021), paraphrased in original wording. Replace every [bracketed] value and every row marked Example. Worked examples evaluate answers from two versions of a support assistant, rated by trained contractors.

Document control

Field	Value
Document ID	[HEVAL-PROT-001]
Study name	[Name]
Version	[0.1]
Status	[Draft / In review / Approved / Data collection / Closed]
Owner	[@study-lead]
Approvers / reviewers	[@research-lead], [@ethics-reviewer], [@privacy-officer]
Last updated	YYYY-MM-DD
Classification	[Internal / Confidential]
Systems under test	[System A version], [System B version]
Ethics review	[Board name, reference number, approval or exemption date]
Pre-registration	[Registry link or internal record, date]
Related documents	[Annotation guidelines], [Evaluation plan], [Data protection assessment]

Approvals

Role	Name	Decision	Date
Study lead	[@name]	[Submitted]	YYYY-MM-DD
Research lead	[@name]	[Approved / Changes requested]	YYYY-MM-DD
Ethics reviewer	[@name]	[Decision]	YYYY-MM-DD
Privacy officer	[@name]	[Decision]	YYYY-MM-DD

Revision history

Version	Date	Author	Change summary
0.1	YYYY-MM-DD	[@name]	First draft
0.2	YYYY-MM-DD	[@name]	Example: Rubric criterion "helpfulness" split after pilot
1.0	YYYY-MM-DD	[@name]	Example: Frozen before main data collection

Objective and research questions
Task and rubric
Ethics consent and rater welfare
Rater recruitment and training
Sample design and size
Blinding and randomisation
Inter-annotator agreement
Quality control
Analysis plan
Reporting checklist
Glossary and references

1. Objective and research questions

ID	Research question	Criterion	Comparison	Primary measure
RQ-001 (Example)	Do raters prefer answers from System B over System A?	Overall usefulness	Pairwise, same prompt	Win rate of B, ties excluded
RQ-002 (Example)	Is System B more often factually correct?	Correctness	Absolute rating per answer	Share rated "Correct"
RQ-003 (Example)	Does System B produce more harmful content?	Safety	Absolute rating per answer	Share flagged "Harmful"
[RQ-nnn]	[Question]	[Criterion]	[Comparison]	[Measure]

Primary question: [Example: RQ-001]. Secondary questions are reported as such and corrected for multiple comparisons (section 9).

2. Task and rubric

2.1 What raters see

[Describe the interface: the prompt, the context (if any), one or two outputs, the questions asked, and any tools such as source documents.]

Example: Each screen shows a customer question, the help-centre article the assistant could use, and two answers labelled "Response 1" and "Response 2". Raters answer three questions, then may leave a comment. Expected time: about 90 seconds per screen.

2.2 Rubric

ID	Criterion	Scale	Definition	Anchors
CRT-001 (Example)	Correctness	Correct / Partly correct / Incorrect	Every factual claim is supported by the article or general knowledge	Correct: no unsupported claim; Partly: minor unsupported detail that does not change the action; Incorrect: any claim that would mislead the customer
CRT-002 (Example)	Usefulness (pairwise)	Response 1 much better, Response 1 slightly better, tie, Response 2 slightly better, Response 2 much better (5 points)	Which answer would better resolve the customer's issue	Judge resolution of the issue, not length or tone
CRT-003 (Example)	Harm	Yes / No, with category	Contains unsafe advice, personal data or abusive language	Category list in annotation guidelines section 4
[CRT-nnn]	[Criterion]	[Scale]	[Definition]	[Anchors]

Keep detailed instructions and worked examples in Annotation Guidelines (ai-data/annotation-guidelines) and link the version here.

2.3 Absolute versus pairwise judgements

Design	Strengths	Weaknesses	Use here
Absolute (rate one output)	Gives a level, not only a ranking; reusable across studies	Scale drift between raters	CRT-001, CRT-003
Pairwise (compare two outputs)	Easier and often more consistent judgement	Only relative; ties need a rule	CRT-002
Ranking (order several outputs)	Efficient for many systems	Harder task, slower	Not used

3. Ethics consent and rater welfare

Item	Plan
Ethics review	[Board, reference, date of approval or exemption]
Information sheet and consent	Example: Raters read the purpose, tasks, content warnings, pay, data use and the right to withdraw, and give consent before the first task
Right to withdraw	Example: At any time without penalty; paid for completed work
Pay	Example: Hourly equivalent at or above [local living wage or minimum wage, whichever is higher], checked against pilot task times
Content warnings	Example: Some answers may contain offensive text from adversarial prompts; raters may skip any item
Exposure limits	Example: No more than 2 hours of harm-labelling per day; access to [support resource]
Personal data	Example: Customer questions pseudonymised before rating; raters' identities stored separately from ratings under a random rater ID
Retention	Example: Ratings kept [period]; rater identity data deleted [period] after the study
Data protection basis	[Lawful basis and assessment reference, if applicable]

Shmueli et al. (2021) discuss ethical issues in crowdsourced NLP work beyond pay, including exposure to harmful content and the status of workers as research participants. Check which rules apply in your jurisdiction and institution.

4. Rater recruitment and training

4.1 Rater profile

Item	Plan
Rater pool	Example: Contract raters via [vendor], plus 2 in-house support agents for adjudication
Number of raters	Example: 12 raters for the main study
Required expertise	Example: Fluent in the item language; 6 months or more of customer support experience
Exclusions	Example: No one who worked on either system
Relevant characteristics recorded	Example: Language, years of support experience, region (with consent)

4.2 Training and qualification

Step	Content	Pass rule
Guidelines	Read the annotation guidelines and rubric	Not applicable
Worked examples	20 items with gold answers and explanations	Not applicable
Qualification test	30 items with adjudicated gold labels	Example: At least 80% agreement with gold on CRT-001
Calibration call	Discuss disagreements on qualification items	Attendance

4.3 Pilot

Item	Plan
Size	Example: 50 items, 4 raters
Goals	Check task time, rubric clarity, interface bugs, early agreement
Exit rule	Example: Agreement on CRT-001 at least 0.6 (Krippendorff's alpha) or rubric revised and piloted again
Pilot data	Not used in the main analysis

5. Sample design and size

5.1 Item sampling

Stratum	Source	Share	Items
Typical traffic (Example)	Random sample of production questions, last 30 days	70%	214
Known hard cases (Example)	Escalated tickets	20%	61
Safety probes (Example)	Red-team prompt set	10%	30
Total		100%	305

Report results per stratum and overall; weight the overall estimate by the real traffic mix when the sample shares differ from it.

5.2 Sample size for the primary question

For a pairwise win rate tested against 50% (no preference), the number of non-tied comparisons needed is approximately:

n \approx \left( \frac{z_{1-\alpha/2} \cdot 0.5 + z_{1-\beta} \cdot \sqrt{p_1 (1 - p_1)}}{p_1 - 0.5} \right)^{2}

Example: to detect a true win rate $p_1 = 0.58$ with two-sided $\alpha = 0.05$ ( $z = 1.960$ ) and 80% power ( $z = 0.842$ ): $\sqrt{0.58 \times 0.42} = 0.4936$ ; numerator $1.960 \times 0.5 + 0.842 \times 0.4936 = 0.980 + 0.416 = 1.396$ ; divided by $0.08$ gives $17.44$ ; squared gives about 304, so 305 items. If pilot data show 20% ties, collect about $305 / 0.8 \approx 382$ comparisons, or increase the item count accordingly.

5.3 Ratings per item

Criterion	Raters per item	Reason
CRT-001 (Example)	3	Majority label plus agreement estimate
CRT-002 (Example)	3	Reduces noise in preference
CRT-003 (Example)	3, any "Yes" triggers adjudication	Safety errors are costly to miss

6. Blinding and randomisation

Control	Plan
System identity	Example: Hidden; outputs labelled only "Response 1" and "Response 2"
Position	Example: System A shown first on a random half of screens, balanced per rater
Item order	Example: Random per rater, seed recorded
Item assignment	Example: Balanced incomplete design so each rater sees a similar mix of strata
Formatting cues	Example: Both outputs rendered with the same template; signatures and model-specific phrases removed
Length	Example: Length recorded and checked as a covariate, since raters often prefer longer answers
Assignment key	Held by [@study-lead]; not shared with raters or analysts until the analysis script is frozen

7. Inter-annotator agreement

7.1 Choosing a statistic

Statistic	Raters	Scale	Missing ratings	Notes
Cohen's kappa (Cohen 1960)	Exactly 2, same raters for all items	Nominal (weighted variant for ordinal)	Not supported	Simple; use for adjudication pairs
Fleiss' kappa (Fleiss 1971)	Fixed number per item, raters may differ across items	Nominal	Not supported	Common for crowd designs
Krippendorff's alpha (Krippendorff 2018)	Any number	Nominal, ordinal, interval, ratio	Supported	Most flexible; preferred here

All three compare observed agreement with the agreement expected by chance:

\kappa = \frac{p_o - p_e}{1 - p_e}, \qquad \alpha = 1 - \frac{D_o}{D_e}

where $p_o$ is observed agreement, $p_e$ chance agreement, and $D_o$ , $D_e$ the observed and expected disagreement.

7.2 Worked example: Cohen's kappa

Two raters label 200 answers as acceptable or not acceptable:

	Rater 2 acceptable	Rater 2 not acceptable	Total
Rater 1 acceptable	120	18	138
Rater 1 not acceptable	12	50	62
Total	132	68	200

Example: $p_o = (120 + 50) / 200 = 0.85$ . Chance agreement $p_e = 0.69 \times 0.66 + 0.31 \times 0.34 = 0.4554 + 0.1054 = 0.5608$ . $\kappa = (0.85 - 0.5608) / (1 - 0.5608) = 0.2892 / 0.4392 = 0.66$ .

Raw agreement of 85% looks high, but kappa of 0.66 shows that part of it would happen by chance given how often both raters say "acceptable".

7.3 Targets and actions

Criterion	Statistic	Target (Example)	Action if below
CRT-001	Krippendorff's alpha, ordinal	≥ 0.67	Retrain raters, clarify anchors, re-pilot
CRT-002	Krippendorff's alpha, ordinal	≥ 0.50	Report as a limitation; consider a 3-point scale
CRT-003	Fleiss' kappa	≥ 0.60	Adjudicate every disagreement

Targets are project choices, not universal standards; published labels for kappa ranges are conventions. Report the value with a bootstrap confidence interval over items, and report agreement per stratum. Low agreement can mean a vague rubric, a genuinely subjective criterion, or unreliable raters: section 8 helps tell these apart.

8. Quality control

ID	Check	Rule	Action
QC-001 (Example)	Gold items	10% of screens are gold items with adjudicated answers, mixed in unannounced	Rater below 70% on the gold items seen so far (minimum 5) is paused and retrained
QC-002 (Example)	Attention items	One obviously wrong answer per 50 items	Two misses lead to review
QC-003 (Example)	Time per item	Median under 20 seconds	Review that rater's work
QC-004 (Example)	Position bias	Rater picks position 1 more than 70% of the time	Review and retrain
QC-005 (Example)	Agreement with peers	Rater's alpha with others far below the group	Review, not automatic exclusion
QC-006 (Example)	Comments	Weekly read of free-text comments	Fix rubric or interface issues

Exclusion rules: [Example: exclude a rater's work only if QC-001 fails after retraining; record every exclusion with reason and report the count]. Never exclude raters because they disagree with the expected result.

9. Analysis plan

Question	Estimator	Test	Interval	Notes
RQ-001 (Example)	Share of non-tied comparisons won by B	Two-sided binomial test against 0.5	Bootstrap over items, 10,000 resamples	Ties reported separately; sensitivity analysis counting ties as half a win
RQ-002 (Example)	Share of answers rated "Correct" by majority	Paired test over items (McNemar)	Bootstrap over items	Partly correct analysed separately
RQ-003 (Example)	Share flagged harmful after adjudication	Exact test on paired items	Exact binomial interval	Any increase is reviewed regardless of significance

Item	Plan
Unit of analysis	Example: Item, with ratings aggregated by majority; mixed-effects model with rater and item random effects as a robustness check
Multiple comparisons	Example: Holm correction across RQ-002 and RQ-003; RQ-001 is primary
Missing data	Example: Items with fewer than 2 valid ratings excluded and counted
Covariates	Example: Answer length difference, stratum
Analysis code	Written and frozen before unblinding; [path]

Report a model's human-rated results alongside automatic metrics in the Benchmark Report (ai-data/benchmark-report) or the LLM Evaluation Plan (ai-data/llm-evaluation-plan), and use human labels to calibrate any LLM judge described there.

10. Reporting checklist

Item	Where reported	Done
Systems and versions evaluated	[Section]	[ ]
Item sampling method and strata, with counts	[Section]	[ ]
Exact instructions, rubric and interface screenshots	[Appendix]	[ ]
Rater number, recruitment, expertise, training and pay	[Section]	[ ]
Ethics review and consent process	[Section]	[ ]
Blinding and randomisation procedure	[Section]	[ ]
Ratings per item and aggregation rule	[Section]	[ ]
Agreement statistic per criterion with interval	[Section]	[ ]
Quality control results and exclusions with counts	[Section]	[ ]
Estimates with intervals, tests and corrections	[Section]	[ ]
Deviations from this protocol, with reasons	[Section]	[ ]
Release of anonymised ratings, if permitted	[Link]	[ ]

11. Glossary and references

Term	Meaning
Rubric	The criteria, scales and anchors raters apply
Anchor	A description or example that fixes what a scale point means
Gold item	An item with an agreed correct label, used to check raters
Blinding	Hiding which system produced an output
Inter-annotator agreement	How consistently different raters label the same items, corrected for chance
Adjudication	Resolving disagreements by an expert or discussion
Win rate	Share of pairwise comparisons in which a system is preferred

References:

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20(1), 37 to 46.
Fleiss, J. L. (1971). Measuring nominal scale agreement among many raters. Psychological Bulletin, 76(5), 378 to 382.
Krippendorff, K. (2018). Content Analysis: An Introduction to Its Methodology (4th ed.). SAGE.
van der Lee, C., Gatt, A., van Miltenburg, E. and Krahmer, E. (2021). Human evaluation of automatically generated text: Current trends and best practice guidelines. Computer Speech and Language, 67, 101151.
Howcroft, D. M. et al. (2020). Twenty Years of Confusion in Human Evaluation: NLG Needs Evaluation Sheets and Standardised Definitions. INLG 2020.
Shimorina, A. and Belz, A. (2022). The Human Evaluation Datasheet: A Template for Recording Details of Human Evaluation Experiments in NLP. HumEval 2022.
Shmueli, B., Fell, J., Ray, S. and Ku, L.-W. (2021). Beyond Fair Pay: Ethical Implications of NLP Crowdsourcing. NAACL 2021.
Related templates: Annotation Guidelines (ai-data/annotation-guidelines), LLM Evaluation Plan (ai-data/llm-evaluation-plan), Agent Evaluation Plan (ai-data/agent-evaluation-plan), Benchmark Report (ai-data/benchmark-report), Model Card (ai-data/model-card).

Standard and licence note

Based on established methods for human evaluation and inter-rater reliability and on the published guidance cited above, including the Human Evaluation Datasheet (Shimorina and Belz 2022) and van der Lee et al. (2021); their recommendations are paraphrased in original wording and no text or tables are reproduced. No formal standard governs this document. The worked example and its figures are illustrative. Check your institution's ethics requirements and the applicable data protection law before recruiting raters. Template licence: MIT.