# Data Annotation Guidelines: [Task name]

> Guidelines for people who label data for a machine learning task: task
> definition, label taxonomy with definitions and examples, edge cases,
> decision rules, quality control with gold sets and inter-annotator
> agreement, annotator training, tooling, privacy and versioning of the
> guidelines. Based on common data annotation practice. Replace every
> `[bracketed]` value and every row marked *Example*. Worked examples use
> intent labelling of customer support messages.

<!--
Guidance: How to use this template
- Write for the annotator, not the data scientist: short sentences,
  concrete examples, one decision per rule.
- Pilot the guidelines on [200] items with at least three annotators
  before large-scale labelling; most ambiguities appear in the pilot.
- Every disagreement that reveals a gap becomes an edge case entry
  (section 4) and a new guideline version (section 10).
- Keep labels stable within a dataset version; changing a definition
  mid-project silently corrupts the data.
-->

| Field | Value |
|---|---|
| Guidelines ID | [ANN-GL-014] |
| Task | [Intent classification of support messages] |
| Guidelines version | [1.0] |
| Status | [Pilot / Active / Retired] |
| Owner | [@annotation-lead] |
| Dataset | [support-intents-v3] |
| Annotation tool | [Tool name and project link] |
| Last updated | YYYY-MM-DD |
| Related documents | [Datasheet for the dataset], [Model card], [Data protection impact assessment] |

## 1. Task definition

<!-- Guidance: Explain what is labelled, why, and what a unit of work is. Annotators who know the purpose make better judgement calls. -->

**Purpose.** Labels train a model that routes customer messages to the
right team. A wrong label sends a customer to the wrong queue and adds
on average [6] hours to resolution.

**Unit of annotation.** One customer message (the first message of a
conversation), shown with [the product name and channel] but without
agent replies.

**Output.** Exactly one primary intent label from section 2, plus an
optional secondary label, plus a confidence flag (Sure / Unsure).

**Throughput.** Expected [90 to 120] items per hour after training.

## 2. Label taxonomy

<!-- Guidance: Give each label an ID, a one-sentence definition, an inclusion rule, an exclusion rule and real-looking examples. Keep labels mutually exclusive at the primary level. -->

| ID | Label | Definition | Include | Exclude | Example |
|---|---|---|---|---|---|
| L01 | Billing question | Customer asks about a charge, invoice or payment | Unknown charge, invoice copy, payment method | Request for money back (use L02) | *Example:* "Why was I charged 14.99 twice this month?" |
| L02 | Refund request | Customer wants money returned | Return refund, cancellation refund | Questions only about a charge (L01) | *Example:* "Please refund my order 55812, it arrived broken." |
| L03 | Delivery status | Customer asks where an order is | Tracking, delay, missing parcel | Damaged item (L04) | *Example:* "My parcel said delivered but it's not here." |
| L04 | Product problem | Item is faulty, damaged or not as described | Broken, wrong size, missing parts | How-to questions (L05) | *Example:* "The charger stopped working after two days." |
| L05 | How-to | Customer asks how to use a product or feature | Setup, settings, account features | Fault reports (L04) | *Example:* "How do I change my delivery address?" |
| L06 | Account access | Login, password, verification issues | Locked account, 2FA codes | Data deletion request (L07) | *Example:* "I'm not getting the verification code." |
| L07 | Privacy request | Access, deletion or correction of personal data | GDPR requests, unsubscribe from all data use | Marketing email opt-out only (L08) | *Example:* "Delete all data you hold about me." |
| L08 | Other | Clear intent not covered above | Feedback, compliments, partnership offers | Unclear or empty messages (L09) | *Example:* "Your new app design is great." |
| L09 | Unclear | No intent can be determined | Greetings only, gibberish, wrong language | Any message with a clear intent | *Example:* "hello??" |

## 3. Decision rules

<!-- Guidance: Turn the most frequent judgement calls into an ordered flow. Annotators apply the first rule that matches. -->

Apply these rules in order and stop at the first that matches:

1. If the message asks to access, delete or correct personal data, label
   **L07**, even if other intents appear.
2. If the customer wants money back, label **L02**.
3. If the message reports a fault or damage, label **L04**.
4. Otherwise choose the label for the main request; if two requests
   carry equal weight, use the one mentioned first as primary and the
   other as secondary.
5. If no rule applies, use **L09** and set the flag to Unsure.

The same rules as a flow:

```mermaid
flowchart TD
  A["Read message"] --> B{"Personal data request?"}
  B -->|yes| L7["L07 Privacy request"]
  B -->|no| C{"Wants money back?"}
  C -->|yes| L2["L02 Refund request"]
  C -->|no| D{"Fault or damage?"}
  D -->|yes| L4["L04 Product problem"]
  D -->|no| E{"Clear main request?"}
  E -->|yes| F["Matching label L01, L03, L05, L06, L08"]
  E -->|no| L9["L09 Unclear"]
```

## 4. Edge cases

<!-- Guidance: Record real ambiguous items and the agreed answer with the reason. This section grows over time; it is the most valuable part of the guidelines. -->

| ID | Situation | Correct label | Reason |
|---|---|---|---|
| EC-01 | *Example:* "Item broken, I want a refund" | L02 primary, L04 secondary | Rule 2 before rule 3: the requested outcome decides |
| EC-02 | *Example:* Message in a language other than [English] | L09, flag Unsure | Out of scope for this dataset |
| EC-03 | *Example:* Sarcasm: "Great, another charge I didn't ask for" | L01 | Label the intent, not the tone |
| EC-04 | *Example:* Message contains card number | Label normally, then use "Report sensitive data" | See section 8 |
| EC-05 | *Example:* "Stop emailing me" | L08 | Marketing opt-out only, not a data deletion request |

## 5. Quality control

<!-- Guidance: Combine gold items, overlap between annotators and review. Report agreement per label, not only overall. -->

### 5.1 Gold set

A gold set of [300] items labelled by [two senior annotators with
adjudication] is mixed into the queue at [5%]. An annotator whose gold
accuracy over the last [100] gold items falls below [90%] is paused and
retrained.

### 5.2 Overlap and agreement

[10%] of items are labelled independently by two annotators. Agreement
is measured with Cohen's kappa, which corrects observed agreement for
agreement expected by chance:

$$
\kappa = \frac{p_o - p_e}{1 - p_e}
$$

where $p_o$ is the observed share of items on which both annotators
agree and $p_e = \sum_k p_{1k}\, p_{2k}$ is the chance agreement from
each annotator's label proportions $p_{1k}$ and $p_{2k}$. *Example:*
$p_o = 0.88$ and $p_e = 0.20$ give $\kappa = 0.68 / 0.80 = 0.85$. For more
than two annotators use Fleiss' kappa or Krippendorff's alpha.

| Measure | Target | Action if below |
|---|---|---|
| Overall kappa | ≥ [0.80] | Review guidelines, retrain |
| Kappa per label | ≥ [0.70] | Add edge cases for that label |
| Gold accuracy per annotator | ≥ [90%] | Pause and retrain annotator |
| Unsure flag rate | ≤ [8%] | Check for missing label or rule |

### 5.3 Adjudication

Disagreements on overlap items go to an adjudicator, who records the
final label and, where the guidelines were unclear, proposes an edge
case for section 4.

## 6. Annotator training

<!-- Guidance: Describe onboarding and the qualification test. -->

1. Read these guidelines and the edge case list ([30] minutes).
2. Label [50] practice items with immediate feedback.
3. Pass a qualification test of [100] items with accuracy ≥ [90%] and no
   errors on L07 privacy requests.
4. First [500] production items are reviewed at [20%].
5. Weekly calibration session of [30] minutes on recent disagreements.

## 7. Tooling

<!-- Guidance: Name the tool, the interface conventions and keyboard shortcuts, and how to report problems. -->

- Tool: [annotation tool and version], project [link].
- Keyboard shortcuts: keys 1 to 9 select L01 to L09; S toggles Unsure;
  R opens "Report sensitive data".
- Items are served in random order; annotators cannot see others'
  labels.
- Problems with the tool go to [#annotation-support].

## 8. Privacy and sensitive content

<!-- Guidance: Annotators often see personal data. State what they may and may not do, and how to escalate. Follow applicable data protection law and your organisation's policy. -->

- Data is pseudonymised before annotation; names and emails are replaced
  with tokens such as `[NAME_1]`.
- If you see remaining personal data, card numbers or health
  information, label the item and press "Report sensitive data"; the
  item is redacted and removed from the dataset.
- Do not copy, screenshot or discuss items outside the tool.
- Annotators may skip distressing content; support is available at
  [contact].
- Access is limited to annotators who signed [confidentiality
  agreement] and completed [privacy training].

## 9. Throughput and payment

<!-- Guidance: If external annotators are used, state fair pay and realistic throughput; speed pressure lowers quality. -->

Annotators are paid [hourly rate, at least the local living wage], not
per item. Expected throughput is a guide, not a target; quality metrics
take priority.

## 10. Versioning of guidelines

<!-- Guidance: Version every change and record which dataset items were labelled under which version. Re-label affected items when a definition changes. -->

| Version | Date | Change | Items affected | Re-label needed |
|---|---|---|---|---|
| 0.9 | YYYY-MM-DD | Pilot version | Pilot 200 | n/a |
| 1.0 | YYYY-MM-DD | *Example:* Added rule 1 for privacy requests; EC-05 | 0 | No |
| 1.1 | YYYY-MM-DD | *Example:* Split "Delivery" into L03 and a future returns label | 4,120 L03 items | Yes, 600 sampled |

Each labelled item stores the guidelines version in field
`guidelines_version`.

## Standard and licence note

Based on common industry practice for data annotation and on standard
inter-annotator agreement statistics (Cohen's kappa). Original text
written for this collection; examples are illustrative. Template
licence: MIT.