Human Oversight Design: [AI system name]
Design document for human oversight of an AI system: oversight
objectives, the decision points matrix (human in the loop, on the loop
and over the loop), approval gates, escalation, override and stop
mechanisms, competence and training of overseers, automation-bias
countermeasures, interface requirements for interpretation, logging
and evidence, and review cadence. Organised on Article 14 of the EU
Artificial Intelligence Act (Regulation (EU) 2024/1689 as amended by
Regulation (EU) 2026/1744) and the deployer duties in Article 26, with
a mapping to the related Annex A controls of ISO/IEC 42001:2023 cited
by number only. Replace every [bracketed] value and every row marked
Example. Under the amended Article 113, the high-risk requirements
that include Article 14 apply from 2 December 2027 for Annex III
systems and from 2 August 2028 for Annex I systems; check the
consolidated text on EUR-Lex. This is a documentation format, not
legal advice.
Document control
| Field | Value |
|---|
| Document ID | [HOD-001] |
| AI system and version | [Name] [v2.3] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Retired] |
| Owner | [@oversight-owner] |
| Approvers/reviewers | [@business-owner], [@risk-management], [@compliance], [@data-protection-officer] |
| Organisation role | [Provider / Deployer / Both] |
| AI Act classification | [High-risk, Annex III point [x] / High-risk, Annex I / Not high-risk] |
| Provider instructions for use | [Document and version] |
| Classification | [Internal / Confidential] |
| Last updated | YYYY-MM-DD |
| Next review | YYYY-MM-DD |
| Related documents | [Risk management file], [FRIA], [DPIA], [AI literacy plan], [Runbook] |
Approvals
| Role | Name | Decision | Date |
|---|
| Business process owner | [@name] | [Approve / Approve with conditions / Reject] | YYYY-MM-DD |
| AI system owner | [@name] | [Decision] | YYYY-MM-DD |
| Risk or compliance | [@name] | [Decision] | YYYY-MM-DD |
| Head of the overseeing team | [@name] | [Decision] | YYYY-MM-DD |
Revision history
| Version | Date | Author | Change summary |
|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft |
| 1.0 | YYYY-MM-DD | [@name] | Example: Approved for pilot with all recommendations reviewed |
| 1.1 | YYYY-MM-DD | [@name] | Example: Added disagreement sampling after automation-bias review |
Contents
1. System and legal context
1.1 System description
[Describe the system, its intended purpose and the decision it supports.]
Example: Recommends a credit limit and a risk band for consumer credit
card applications. Lending officers see the recommendation and decide the
limit. About 1,200 applications a day.
| Item | Value |
|---|
| Intended purpose | Example: Support creditworthiness assessment of natural persons |
| Output | Example: Risk band A to E, recommended limit, top contributing factors |
| Decision supported | Example: Approve, decline or set limit |
| Affected persons | Example: Credit applicants |
| Level of autonomy | Example: Recommendation only; no automatic decisions |
| Context of use | Example: Branch and online back office, business hours |
1.2 Role and applicability
| Question | Answer |
|---|
| Is the system high-risk under Article 6? | Example: Yes, Annex III point 5(b), creditworthiness of natural persons |
| Our role | Example: Deployer; provider is [Vendor] |
| Applicable oversight duties | Example: Article 26(1) use per instructions, 26(2) assign oversight to competent persons, 26(5) monitor and report; provider measures under Article 14 described in the instructions for use (Article 13) |
| Other law | Example: GDPR Article 22 on solely automated decisions; consumer credit rules |
| Application date | Example: 2 December 2027 under the amended Article 113 (check the consolidated text) |
2. Oversight objectives
| ID | Objective | Risk addressed | Risk register ref |
|---|
| HOD-OBJ-001 (Example) | Detect recommendations based on wrong or stale applicant data | Unfair decline from data error | [RISK-014] |
| HOD-OBJ-002 (Example) | Prevent systematic disadvantage to protected groups | Discrimination | [RISK-003] |
| HOD-OBJ-003 (Example) | Stop use quickly if model performance degrades | Wide-scale wrong decisions | [RISK-021] |
| HOD-OBJ-004 (Example) | Ensure officers do not rubber-stamp recommendations | Automation bias | [RISK-009] |
| HOD-OBJ-005 (Example) | Give applicants a meaningful human review on request | Loss of contestability | [RISK-011] |
3. Allocation of oversight measures
| ID | Measure | Built in by provider (14(3)(a)) | Implemented by deployer (14(3)(b)) | Source |
|---|
| HOD-M-001 (Example) | Factor explanation shown with each recommendation | Yes | Configure display | Instructions for use section [x] |
| HOD-M-002 (Example) | Confidence indicator and out-of-distribution flag | Yes | Act on the flag (section 5) | Instructions for use section [x] |
| HOD-M-003 (Example) | Suspend switch per branch and global | Yes | Assign who may use it | Admin guide |
| HOD-M-004 (Example) | Second review for declines | No | Yes, internal procedure | Lending procedure [ref] |
| HOD-M-005 (Example) | Overseer training and certification | No | Yes | AI literacy plan |
| HOD-M-006 (Example) | Disagreement sampling and feedback to provider | Partly: feedback channel | Yes, sampling process | Section 13 |
4. Decision points matrix
| ID | Decision point | Impact if wrong | Reversible? | Mode | Overseer | Time to decide |
|---|
| HOD-DP-001 (Example) | Approve application at or below recommended limit | Moderate | Yes, limit can be lowered | Human in the loop (light review) | Lending officer | Same day |
| HOD-DP-002 (Example) | Decline application | High: affects access to credit | Hard: applicant may go elsewhere | Human in the loop (full review plus second reviewer) | Lending officer and senior officer | 2 working days |
| HOD-DP-003 (Example) | Limit above recommendation | Moderate | Yes | Human in the loop with reason code | Lending officer | Same day |
| HOD-DP-004 (Example) | Daily portfolio of decisions | High in aggregate | Partly | Human on the loop | Credit risk analyst | Daily |
| HOD-DP-005 (Example) | Model version, thresholds and permitted use | High | Yes | Human over the loop | Model risk committee | Monthly and on change |
5. Approval gates
| ID | Gate | Checks required | Evidence shown | Allowed actions | Default if no action |
|---|
| HOD-GATE-001 (Example) | Light review (HOD-DP-001) | Identity and income documents match inputs | Inputs, recommendation, top 3 factors | Approve, adjust, escalate | Application stays pending |
| HOD-GATE-002 (Example) | Decline review (HOD-DP-002) | Independent assessment against lending policy before viewing the recommendation band | Full application, factors, policy checklist | Confirm decline, approve, refer | Not declined |
| HOD-GATE-003 (Example) | Out-of-distribution case | Manual assessment without the recommendation | Inputs only; recommendation hidden until assessment recorded | Decide manually | Pending |
6. Escalation, override and stop
| ID | Mechanism | Who can use it | Effect | Safe state | Target time | Tested |
|---|
| HOD-CTL-001 (Example) | Override recommendation | Any certified officer | Officer's decision replaces output; reason code required | Normal manual process | Immediate | Each release |
| HOD-CTL-002 (Example) | Disregard system for a case | Any certified officer | Case handled manually | Manual process | Immediate | Each release |
| HOD-CTL-003 (Example) | Reverse a decision | Senior officer | Decision reopened, applicant notified | Pending | 1 working day | Quarterly |
| HOD-CTL-004 (Example) | Suspend for one branch | Branch manager | Recommendations hidden; manual process | Manual process | 5 min | Quarterly drill |
| HOD-CTL-005 (Example) | Global stop | Head of credit risk, on-call | System returns no recommendations | Manual process for all | 5 min | Quarterly drill |
6.1 Escalation path
| Trigger | Escalate to | Within |
|---|
| Example: Officer believes a recommendation is systematically wrong | Team lead, then model owner | 1 working day |
| Example: Suspected discrimination pattern | Compliance and model risk committee | 2 working days |
| Example: Serious incident or risk to rights | Provider and, where Article 73 applies, the relevant market surveillance authority | As required by law; check Article 26(5) and Article 73 |
7. Overseer capabilities (Article 14(4))
| Point | Capability (paraphrased) | How it is enabled here | Measures |
|---|
| 14(4)(a) | Understand the system's capacities and limitations and monitor its operation, including detecting anomalies and unexpected performance | Training module on limits; dashboard of drift and override rates | HOD-M-005, section 13 |
| 14(4)(b) | Stay aware of the tendency to over-rely on outputs (automation bias), in particular for recommendations | Countermeasures in section 9 | HOD-AB-001 to HOD-AB-006 |
| 14(4)(c) | Correctly interpret the output, with the interpretation tools available | Factor display, confidence band, glossary | HOD-M-001, HOD-M-002, section 10 |
| 14(4)(d) | Decide not to use the system, or disregard, override or reverse its output | Override, disregard and reverse controls | HOD-CTL-001 to HOD-CTL-003 |
| 14(4)(e) | Intervene or interrupt the system through a stop control or similar procedure that leaves it in a safe state | Branch suspend and global stop | HOD-CTL-004, HOD-CTL-005 |
8. Competence, training and authority
| Role | Number | Competence required | Training | Assessment | Authority |
|---|
| Lending officer (Example) | 140 | Lending policy, reading factors, known failure modes | 3 h module plus 20 supervised cases | Certification quiz and case review | Override, disregard |
| Senior officer (Example) | 18 | As above plus fairness indicators | Additional 2 h module | Case review | Reverse decisions |
| Branch manager (Example) | 30 | When and how to suspend | 30 min module and drill | Drill participation | Suspend branch |
| Credit risk analyst (Example) | 4 | Monitoring metrics and thresholds | Model documentation walkthrough | Peer review | Recommend global stop |
| Head of credit risk (Example) | 1 | Full system and governance | Provider briefing | n/a | Global stop |
- [Example: Officers have time targets that allow full review of declines
(target 25 minutes per decline case).]
- [Example: Overrides are not penalised in performance reviews; override
rates are monitored for quality, not reduced as a target.]
- [Example: Refresher training every 12 months and after each major
model change.]
9. Automation-bias countermeasures
| ID | Countermeasure | Applies to | How effectiveness is measured |
|---|
| HOD-AB-001 (Example) | Officer records an independent view before seeing the band on decline candidates | HOD-GATE-002 | Agreement rate before and after reveal |
| HOD-AB-002 (Example) | Recommendation hidden for out-of-distribution cases | HOD-GATE-003 | Share of flagged cases handled manually |
| HOD-AB-003 (Example) | Reason code required for every decision, not only overrides | All gates | Share of free-text reasons with substance in audit sample |
| HOD-AB-004 (Example) | Seeded test cases with known wrong recommendations | Lending officers | Detection rate per officer per quarter |
| HOD-AB-005 (Example) | Feedback on outcomes of past decisions | Lending officers | Monthly report read rate |
| HOD-AB-006 (Example) | Display of uncertainty rather than a single number | All gates | Officer survey and interpretation quiz |
Signals of over-reliance to watch: [Example: agreement rate above 98%
for an officer over a month; median review time under 60 seconds on
declines; seeded wrong cases approved.]
10. Interface requirements for interpretation
| ID | Requirement | Rationale | Verification |
|---|
| HOD-UI-001 (Example) | Show the inputs the system used, with their source and date | Officers spot stale or wrong data | UI test |
| HOD-UI-002 (Example) | Show the top contributing factors in plain language, with direction | Interpretation | Usability test with 8 officers |
| HOD-UI-003 (Example) | Show a confidence band and an explicit "outside usual cases" flag | Signals when not to rely on the output | UI test |
| HOD-UI-004 (Example) | Make override and disregard as easy to reach as approve | Avoid steering by design | Click-path review |
| HOD-UI-005 (Example) | Link to the system's known limitations from every screen | 14(4)(a) | Content review |
| HOD-UI-006 (Example) | Never pre-select an outcome | Automation bias | UI test |
| HOD-UI-007 (Example) | Stop controls visible to authorised roles with confirmation of the resulting state | 14(4)(e) | Drill |
| HOD-UI-008 (Example) | Accessible to screen readers and keyboard users | Inclusive oversight | Accessibility audit |
11. Separate verification by two persons (Article 14(5))
| Question | Answer |
|---|
| Does the system perform remote biometric identification under Annex III point 1(a)? | Example: No. The system scores credit applications and uses no biometric data |
| If yes: who are the two verifiers and how is independence ensured? | [Roles, separate screens, no shared view of each other's decision until both record] |
| If yes: does an exception apply? | [Legal basis, or "No exception"] |
| Voluntary dual control used anyway | Example: Declines use a second reviewer (HOD-GATE-002) as an internal choice |
12. Logging and evidence
| Evidence | Content | Source | Retention | Owner |
|---|
| Decision log (Example) | Case ID, recommendation, officer decision, reason code, time spent | Lending platform | [Per legal minimum and policy, for example 7 years] | Lending operations |
| System event log (Example) | Automatically generated logs of the AI system | System (Article 12) | At least six months under Article 26(6), unless other Union or national law sets a different period; check | IT |
| Override and stop log (Example) | Who, when, scope, reason, restore time | Admin console | [7 years] | Credit risk |
| Training records (Example) | Completion and assessment per overseer | Learning system | Employment plus [2 years] | HR |
| Drill records (Example) | Stop drills with timings | Runbook records | [5 years] | Credit risk |
| Monitoring reports (Example) | Metrics in section 13 | Analytics | [5 years] | Credit risk analyst |
13. Effectiveness monitoring and review cadence
| ID | Metric | Definition | Threshold for review | Frequency |
|---|
| HOD-KPI-001 (Example) | Override rate | Decisions differing from recommendation divided by all decisions | Below 2% or above 25% per month | Monthly |
| HOD-KPI-002 (Example) | Seeded-case detection | Seeded wrong cases caught divided by seeded cases | Below 80% for an officer | Quarterly |
| HOD-KPI-003 (Example) | Decline review time | Median minutes per decline case | Below 10 minutes | Monthly |
| HOD-KPI-004 (Example) | Outcome gap by group | Difference in approval rates across monitored groups after controls | Above agreed tolerance | Monthly |
| HOD-KPI-005 (Example) | Stop drill time | Time from decision to safe state | Above 5 minutes | Quarterly |
| HOD-KPI-006 (Example) | Training coverage | Certified overseers divided by active overseers | Below 100% | Monthly |
| Review | Cadence | Participants | Output |
|---|
| Operational review | Monthly | Team leads, credit risk analyst | Actions on metrics |
| Design review of this document | Every 12 months and after any major model change | Owner, compliance, overseer representatives | New version |
| Feedback to provider | Quarterly, or at once for serious issues | Model owner | Provider ticket and response |
14. ISO/IEC 42001 Annex A mapping
| Annex A control | Subject (paraphrased) | Where this design supports it |
|---|
| A.3.2 | Defining and allocating AI roles and responsibilities | Sections 4, 8 |
| A.3.3 | Process for reporting concerns about AI systems | Section 6.1 |
| A.4.6 | Human resources and competences for AI | Section 8 |
| A.5.4 | Assessing impacts on individuals and groups | Section 2 and the FRIA |
| A.6.1.3 | Processes for responsible design and development | Section 3 (provider measures) |
| A.6.2.2 | Requirements and specification for the AI system | Section 10 |
| A.6.2.6 | Operation and monitoring of the AI system | Sections 6, 13 |
| A.6.2.8 | Recording of event logs | Section 12 |
| A.8.2 | Documentation and information for users | Sections 7, 10 |
| A.9.2 | Processes for responsible use | Sections 4, 5 |
| A.9.3 | Objectives for responsible use | Section 2 |
| A.9.4 | Use according to the intended use | Section 1 |
Clauses 7.2 (competence) and 7.3 (awareness) of the management system
also apply to section 8.
15. Traceability matrix
| Objective | Decision points | Measures and gates | Controls | Bias countermeasures | Metrics |
|---|
| HOD-OBJ-001 (Example) | HOD-DP-001, HOD-DP-002 | HOD-GATE-001, HOD-UI-001 | HOD-CTL-001 | HOD-AB-003 | HOD-KPI-001 |
| HOD-OBJ-002 (Example) | HOD-DP-002, HOD-DP-004 | HOD-GATE-002, HOD-M-004 | HOD-CTL-003 | HOD-AB-001 | HOD-KPI-004 |
| HOD-OBJ-003 (Example) | HOD-DP-004, HOD-DP-005 | HOD-M-003 | HOD-CTL-004, HOD-CTL-005 | n/a | HOD-KPI-005 |
| HOD-OBJ-004 (Example) | All | HOD-GATE-002, HOD-GATE-003 | HOD-CTL-002 | HOD-AB-001 to HOD-AB-006 | HOD-KPI-001 to HOD-KPI-003 |
| HOD-OBJ-005 (Example) | HOD-DP-002 | HOD-GATE-002 | HOD-CTL-003 | HOD-AB-003 | HOD-KPI-003 |
16. Glossary and references
| Term | Meaning |
|---|
| Human in the loop | A person must act before the system's output takes effect |
| Human on the loop | The output takes effect; a person monitors and can intervene |
| Human over the loop | A person sets the system's limits, reviews its aggregate behaviour and can suspend it |
| Automation bias | Tendency to accept automated output without enough scrutiny |
| Override | Replacing the system's output with a human decision |
| Safe state | A condition in which stopping the system causes no further harm and work continues by other means |
| Provider | The organisation that develops an AI system and places it on the market under its name |
| Deployer | A person or body using an AI system under its authority in a professional activity |
- Regulation (EU) 2024/1689 (Artificial Intelligence Act), as amended by
Regulation (EU) 2026/1744; Articles 4, 6, 12, 13, 14, 26, 73, 113 and
Annex III; consolidated text on EUR-Lex, https://eur-lex.europa.eu/eli/reg/2024/1689/oj
- ISO/IEC 42001:2023 Information technology, Artificial intelligence,
Management system, https://www.iso.org/standard/81230.html
- Regulation (EU) 2016/679 (GDPR), Article 22
- [Provider instructions for use], [Internal lending procedure]
Standard and licence note
Based on Regulation (EU) 2024/1689 (the Artificial Intelligence Act) as
amended by Regulation (EU) 2026/1744, Article 14 on human oversight and
Article 26 on deployer obligations, as published in the Official Journal
of the European Union; EU legislative texts are free to reuse and the
requirements are summarised here in original wording, with no text of
the Regulation reproduced. Structure aligned with ISO/IEC 42001:2023 for
the Annex A mapping, which cites control numbers with original
paraphrases only; obtain the standard for normative requirements. Check
the consolidated text on EUR-Lex and current Commission guidance for the
binding wording and application dates. This is a documentation format,
not legal advice. Template licence: MIT.