Data Annotation Guidelines: [Task name]

Guidelines for people who label data for a machine learning task: task definition, label taxonomy with definitions and examples, edge cases, decision rules, quality control with gold sets and inter-annotator agreement, annotator training, tooling, privacy and versioning of the guidelines. Based on common data annotation practice. Replace every [bracketed] value and every row marked Example. Worked examples use intent labelling of customer support messages.

1. Task definition

Purpose. Labels train a model that routes customer messages to the right team. A wrong label sends a customer to the wrong queue and adds on average [6] hours to resolution.

Unit of annotation. One customer message (the first message of a conversation), shown with [the product name and channel] but without agent replies.

Output. Exactly one primary intent label from section 2, plus an optional secondary label, plus a confidence flag (Sure / Unsure).

Throughput. Expected [90 to 120] items per hour after training.

2. Label taxonomy

3. Decision rules

Apply these rules in order and stop at the first that matches:

1. If the message asks to access, delete or correct personal data, label L07, even if other intents appear.
2. If the customer wants money back, label L02.
3. If the message reports a fault or damage, label L04.
4. Otherwise choose the label for the main request; if two requests carry equal weight, use the one mentioned first as primary and the other as secondary.
5. If no rule applies, use L09 and set the flag to Unsure.

The same rules as a flow:

4. Edge cases

5. Quality control

5.1 Gold set

A gold set of [300] items labelled by [two senior annotators with adjudication] is mixed into the queue at [5%]. An annotator whose gold accuracy over the last [100] gold items falls below [90%] is paused and retrained.

5.2 Overlap and agreement

[10%] of items are labelled independently by two annotators. Agreement is measured with Cohen's kappa, which corrects observed agreement for agreement expected by chance:

where $p_o$ is the observed share of items on which both annotators agree and $p_e = \sum_k p_{1k}\, p_{2k}$ is the chance agreement from each annotator's label proportions $p_{1k}$ and $p_{2k}$. Example: $p_o = 0.88$ and $p_e = 0.20$ give $\kappa = 0.68 / 0.80 = 0.85$. For more than two annotators use Fleiss' kappa or Krippendorff's alpha.

5.3 Adjudication

Disagreements on overlap items go to an adjudicator, who records the final label and, where the guidelines were unclear, proposes an edge case for section 4.

6. Annotator training

1. Read these guidelines and the edge case list ([30] minutes).
2. Label [50] practice items with immediate feedback.
3. Pass a qualification test of [100] items with accuracy ≥ [90%] and no errors on L07 privacy requests.
4. First [500] production items are reviewed at [20%].
5. Weekly calibration session of [30] minutes on recent disagreements.

7. Tooling

• Tool: [annotation tool and version], project [link].
• Keyboard shortcuts: keys 1 to 9 select L01 to L09; S toggles Unsure; R opens "Report sensitive data".
• Items are served in random order; annotators cannot see others' labels.
• Problems with the tool go to [#annotation-support].

8. Privacy and sensitive content

• Data is pseudonymised before annotation; names and emails are replaced with tokens such as [NAME_1].
• If you see remaining personal data, card numbers or health information, label the item and press "Report sensitive data"; the item is redacted and removed from the dataset.
• Do not copy, screenshot or discuss items outside the tool.
• Annotators may skip distressing content; support is available at [contact].
• Access is limited to annotators who signed [confidentiality agreement] and completed [privacy training].

9. Throughput and payment

Annotators are paid [hourly rate, at least the local living wage], not per item. Expected throughput is a guide, not a target; quality metrics take priority.

10. Versioning of guidelines

Each labelled item stores the guidelines version in field guidelines_version.

Standard and licence note

Based on common industry practice for data annotation and on standard inter-annotator agreement statistics (Cohen's kappa). Original text written for this collection; examples are illustrative. Template licence: MIT.