# Usability Test Report: [Product or feature name]

> A report of one usability test: what was evaluated and why, the method and procedure, task
> results for effectiveness, efficiency and satisfaction, findings rated by severity with their
> evidence, and recommendations traced to the backlog. The section order follows the content
> elements that ISO/IEC 25066:2016 (Common Industry Format for usability evaluation reports,
> clause 5.2; withdrawn by ISO in April 2026, see references) lists for a user observation report, and the measures use the usability framework
> of ISO 9241-11:2018. Both are paid standards: this template cites clause numbers and uses
> original wording. Replace every `[bracketed]` value and every row marked *Example*. Worked
> examples continue the Quillmere Libraries test planned in design/usability-test-plan
> (Quillmere Libraries is a fictional city library service).

<!--
Guidance: How to use this template
- Start from the approved plan (design/usability-test-plan). Keep its task IDs, measures,
  segments and severity scale so readers can trace every number back to the plan.
- Write the executive summary last, and keep it to one page: decision, top findings, what to do.
- Every finding needs evidence a sceptical reader can check: which participants, which task,
  timestamps or quotes. A finding without evidence is an opinion.
- Separate what you observed (section 6) from what you think it means and what to do about it
  (section 7).
- Common pitfall: presenting percentages from a formative round of 6 to 8 people as if they
  described all users. Report counts ("4 of 8") and confidence intervals, and say what the sample
  cannot show.
-->

## Document control

<!-- Guidance: Link the plan this report executes. Classification matters: reports often contain quotes and clips, so check they are pseudonymised before wider sharing. -->

| Field | Value |
|---|---|
| Document ID | [UTR-001] |
| Version | [1.0] |
| Status | [Draft / In review / Approved / Superseded] |
| Owner (lead researcher) | [@researcher] |
| Approvers / reviewers | [@product-owner], [@design-lead] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal; pseudonymised; no raw recordings attached] |
| Plan executed | [UTP-001, design/usability-test-plan, version 1.0] |
| Object of evaluation | [Product, version or build] |
| Test type | [Formative / Summative], [Moderated / Unmoderated], [Remote / In person] |
| Fieldwork dates | YYYY-MM-DD to YYYY-MM-DD |
| Participants | [Number completed of number recruited] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Lead researcher | [@name] | [Approved] | YYYY-MM-DD |
| Product owner | [@name] | [Accepted, actions agreed] | YYYY-MM-DD |

### Revision history

| Version | Date | Author | Change |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | Topline summary (top issues) |
| 0.2 | YYYY-MM-DD | [@name] | Full results, severity agreed by two raters |
| 1.0 | YYYY-MM-DD | [@name] | Approved after findings workshop |

## Contents

1. Executive summary
2. Object of evaluation
3. Purpose of the evaluation
4. Method
5. Procedure
6. Results
7. Interpretation and recommendations
8. Limitations
9. Traceability and follow-up
10. Glossary and references

## 1. Executive summary

<!-- Guidance: One page for the decision maker. State the decision, the overall result, the top findings by severity and the recommended action. ISO/IEC 25066:2016 clause 5.2.1 treats the summary as optional; most readers only read this section, so write it. -->

[Two to four short paragraphs, then the top findings table.]

*Example:* Eight borrowers tested the Quillmere Libraries app 5.2 beta reservation flow in
moderated remote sessions between YYYY-MM-DD and YYYY-MM-DD. Finding a book and reserving it
(T1) and checking opening hours (T4) worked well. Changing the collection branch of an existing
reservation (T2) failed for 4 of 8 participants, and 2 of them created a duplicate reservation
instead. Both screen reader users could not renew loans (T3) because the renew controls have no
accessible name.

*Example:* Recommendation: do not ship 5.2 as planned. Fix F-001 to F-003 (severity 3 and 4),
retest T1 to T3 with at least five participants including two screen reader users, then decide.

| Finding | Severity | Summary | Recommendation |
|---|---|---|---|
| F-002 *(Example)* | 4 Catastrophe | Renew controls have no accessible name; both screen reader users failed T3 | REC-002 |
| F-001 *(Example)* | 3 Major | Changing the collection branch is hidden; 4 of 8 failed T2 | REC-001 |
| F-003 *(Example)* | 3 Major | Branch picker not reachable with a screen reader; P04 failed T1 | REC-003 |

## 2. Object of evaluation

<!-- Guidance: Describe exactly what participants used, so the findings are tied to a version. ISO/IEC 25066:2016 clause 5.2.2 covers this element. -->

| Item | Value |
|---|---|
| Product and version | [*Example:* Quillmere Libraries app 5.2.0 beta, build 412] |
| Platforms | [*Example:* iOS and Android, participants' own phones] |
| Fidelity | [*Example:* coded beta with test catalogue data and test library cards] |
| Parts evaluated | [*Example:* search, reserve, change collection branch, renew loans, branch opening hours] |
| Parts not evaluated | [*Example:* account registration, fine payments, staff app] |
| Changes since the last test | [*Example:* new reservation detail screen with a "Manage" menu; new renew screen] |
| Intended users and context | [*Example:* adult borrowers on their own phones, often on the move; see plan section 2] |

## 3. Purpose of the evaluation

<!-- Guidance: Restate the decision, the research questions and any targets from the plan. A reader must be able to judge whether the results answer them. Clause 5.2.3 covers this element. -->

- Decision informed: [*Example:* ship the redesigned reservation flow in release 5.2, or iterate].
- Evaluation type: [*Example:* formative user observation with measures; not a summative benchmark].
- Research questions: [*Example:* RQ-1 Can borrowers reserve and redirect a reservation unaided?
  RQ-2 Where do they hesitate or make errors? RQ-3 How do they rate the app overall?]
- Targets set in the plan: [*Example:* completion at least 80% for T1, T3 and T4 and at least 70%
  for T2; T4 median time 60 s or less; SUS mean 68 or higher].

## 4. Method

<!-- Guidance: Describe participants, tasks, environment and measures in enough detail that another team could repeat the test. Clause 5.2.4 lists the method elements; keep anything that identifies a participant out of this section. -->

### 4.1 Participants

| Code | Segment | Device | Assistive technology | Notes |
|---|---|---|---|---|
| P01 *(Example)* | S1 Frequent | iOS | None | |
| P02 *(Example)* | S2 Occasional | Android | None | |
| P03 *(Example)* | S1 Frequent | iOS | None | |
| P04 *(Example)* | S3 Assistive technology | iOS | VoiceOver | |
| P05 *(Example)* | S2 Occasional | Android | None | |
| P06 *(Example)* | S2 Occasional | iOS | None | |
| P07 *(Example)* | S1 Frequent | Android | None | |
| P08 *(Example)* | S3 Assistive technology | Android | TalkBack and font scaling | |

Recruited [*Example:* 9]; completed [*Example:* 8]; [*Example:* 1 no-show, replaced from the
over-recruit list]. All participants received the incentive. No participant withdrew.

### 4.2 Tasks

| Task ID | Goal (short) | Success criterion | Time limit |
|---|---|---|---|
| T1 *(Example)* | Reserve a title for collection at the nearest branch | Reservation exists with the nearest branch | 3 min |
| T2 *(Example)* | Move an existing reservation to the Central branch | Existing reservation shows Central; no duplicate | 5 min |
| T3 *(Example)* | Renew two loans that are due | Both loans renewed | 3 min |
| T4 *(Example)* | Find Saturday closing time of the Central branch | Correct time stated | 1 min |

### 4.3 Context and environment

[*Example:* remote moderated sessions over a video call, participants at home on their own phones
and networks, screen mirrored to the moderator. The commuting context was not reproduced.]

### 4.4 Measures

| Measure | Definition used |
|---|---|
| Completion rate | Successful attempts over attempts; moderator assistance that revealed the path counted as failure; 95% adjusted Wald interval |
| Time on task | Seconds from end of task reading to success, successful attempts only; median |
| Errors | Wrong paths that needed a recovery step |
| SEQ | 7-point post-task ease rating |
| SUS | Brooke (1996) 10-item scale, scored 0 to 100 |
| Severity | 0 to 4 scale as in design/heuristic-evaluation, rated by two researchers independently, then agreed |

## 5. Procedure

<!-- Guidance: Say what actually happened, including deviations from the plan. Clause 5.2.5 covers the procedure element. Deviations are not failures; hiding them is. -->

- Session flow: [*Example:* welcome, consent check, background, practice task, T1 to T4 in fixed
  order with SEQ after each, SUS, debrief; 50 minutes].
- Moderator and note-taker: [*Example:* the same moderator ran all sessions; two note-takers alternated].
- Consent: [*Example:* written consent before each session using design/research-consent-form,
  repeated verbally on the recording; all participants agreed to screen and audio recording,
  6 of 8 agreed to anonymised clips].
- Deviations from the plan: [*Example:* P08's session moved to the participant's preferred time
  and ran 15 minutes longer to allow breaks; build 412 replaced build 410 after the pilot found a
  crash in T3. No task wording changed after the first participant.]

## 6. Results

<!-- Guidance: Report what was observed and measured, without interpretation. Clause 5.2.6 covers results. Give counts with the denominator, and intervals for rates. -->

### 6.1 Task performance

| Task | Success | Completion | 95% CI (adjusted Wald) | Median time (successful) | Target | Met |
|---|---|---|---|---|---|---|
| T1 *(Example)* | 7 of 8 | 87.5% | 51% to 100% | 58 s | 80% | Yes (point estimate) |
| T2 *(Example)* | 4 of 8 | 50.0% | 22% to 78% | 166 s | 70% | No |
| T3 *(Example)* | 6 of 8 | 75.0% | 40% to 94% | 98 s | 80% | No |
| T4 *(Example)* | 8 of 8 | 100.0% | 63% to 100% | 24.5 s | 80%; median 60 s or less | Yes |

Failures by participant: T1 P04; T2 P02, P04, P06, P08; T3 P04, P08. P02 and P06 created a
second reservation at Central instead of moving the existing one.

The intervals are wide because the sample is small. A completion of 7 of 8 is consistent with a
true rate anywhere from about half to almost all users, so "met" is stated for the point estimate only.

### 6.2 Satisfaction

| Participant | SUS score |
|---|---|
| P01 *(Example)* | 80.0 |
| P02 *(Example)* | 60.0 |
| P03 *(Example)* | 90.0 |
| P04 *(Example)* | 40.0 |
| P05 *(Example)* | 77.5 |
| P06 *(Example)* | 65.0 |
| P07 *(Example)* | 90.0 |
| P08 *(Example)* | 52.5 |

*Example:* mean SUS 69.4 (standard deviation 18.1, n = 8), 95% confidence interval 54.3 to 84.5
(t distribution, 7 degrees of freedom); median 71.3. The mean is just above the reference average
of about 68, but the interval is too wide to say whether the app is above or below it. The two
screen reader users gave the two lowest scores (40.0 and 52.5).

### 6.3 Findings

Each finding lists where it occurred and the evidence. Severity uses 0 not a problem, 1
cosmetic, 2 minor, 3 major, 4 catastrophe.

| ID | Finding | Tasks | Participants | Evidence | Severity |
|---|---|---|---|---|---|
| F-001 *(Example)* | The option to change the collection branch sits in an unlabelled "Manage" menu on the reservation detail screen; participants looked for it on the branch name itself | T2 | P02, P04, P06, P08 failed; P05 took 204 s | P02 at 12:40 "I'd expect to tap the branch"; P06 at 14:05 reserved again at Central | 3 |
| F-002 *(Example)* | Renew controls are announced only as "button" by VoiceOver and TalkBack, so screen reader users cannot tell which loan a control renews | T3 | P04, P08 failed | P04 at 22:10 "Button, button... which book is this?"; WCAG 2.2 SC 4.1.2 Name, Role, Value | 4 |
| F-003 *(Example)* | The branch picker in the reservation sheet cannot be reached by swipe navigation; P08 reached it on the third attempt using explore by touch | T1 | P04 failed; P08 slowed | P04 at 08:30; WCAG 2.2 SC 2.1.1 Keyboard (swipe navigation acts as sequential navigation) | 3 |
| F-004 *(Example)* | The collection branch defaults to the last branch used, not the nearest; participants noticed only on the confirmation screen | T1 | P01, P05, P07 | P05 at 07:55 "Oh, that's not my library" and went back | 2 |
| F-005 *(Example)* | Branch page lists weekday hours first; Saturday hours need scrolling | T4 | P03, P06 | Scrolled past twice, still within target time | 1 |
| F-006 *(Example)* | Positive: search suggestions helped participants find the title quickly | T1 | 6 of 8 commented | P03 at 06:10 "That's quick" | 0 (keep) |

### 6.4 Post-task ease and debrief themes

[*Example:* SEQ median per task: T1 6, T2 3, T3 5, T4 7. Debrief themes: confusion about whether
a reservation moved or was duplicated (5 of 8), praise for search (6 of 8), request for a reminder
before the due date (3 of 8). For full thematic analysis see design/research-synthesis.]

## 7. Interpretation and recommendations

<!-- Guidance: Explain what the results mean for the research questions and the decision, then give one or more recommendations per finding. Clause 5.2.7 covers this element. Recommendations say what to change and how to check it worked, not how to design it in detail. -->

Answers to the research questions:

- RQ-1 *(Example)*: reserving works for most sighted participants; redirecting a reservation
  does not (F-001), and both flows have barriers for screen reader users (F-002, F-003).
- RQ-2 *(Example)*: hesitation clustered on the reservation detail screen and the renew screen.
- RQ-3 *(Example)*: overall satisfaction is around the reference average with wide uncertainty,
  and clearly lower for screen reader users.

| ID | Recommendation | Addresses | Priority | Owner | How to verify |
|---|---|---|---|---|---|
| REC-001 *(Example)* | Make the collection branch on the reservation detail screen directly editable, and confirm the change with a message that says the reservation moved | F-001 | Before release | [@designer] | Retest T2; target 70% completion and no duplicates |
| REC-002 *(Example)* | Give every renew control an accessible name that includes the item title | F-002 | Before release | [@developer] | Screen reader check on iOS and Android; retest T3 with two screen reader users |
| REC-003 *(Example)* | Make the branch picker reachable by swipe navigation and in a logical focus order | F-003 | Before release | [@developer] | Screen reader check; retest T1 |
| REC-004 *(Example)* | Default the collection branch to the nearest branch when location is allowed, otherwise ask | F-004 | Next release | [@product-owner] | Analytics on branch changes after reservation |
| REC-005 *(Example)* | Show today's and Saturday's hours at the top of the branch page | F-005 | Backlog | [@designer] | Design review |

Decision recommendation: [*Example:* hold release 5.2 until REC-001 to REC-003 are done and
a retest of T1 to T3 with at least five participants, including two screen reader users,
shows no severity 3 or 4 problems].

## 8. Limitations

<!-- Guidance: List what limits the confidence in the results, including any deviations from the plan's context of use. Readers trust reports that name their limits. -->

- *Example:* eight participants in a formative round; rates and SUS are descriptive only.
- *Example:* only two assistive technology users; screen magnifier and switch users were not included.
- *Example:* remote sessions at home did not reproduce the commuting context.
- *Example:* fixed task order; T1 may have helped participants with T2.
- *Example:* test catalogue data; real reservation queues and wait times were not shown.

## 9. Traceability and follow-up

<!-- Guidance: Link each finding to the backlog item that fixes it and the retest that verifies it. Close the loop in the next report. -->

| Finding | Recommendation | Backlog item | Status | Verified in |
|---|---|---|---|---|
| F-001 *(Example)* | REC-001 | [*Example:* QL-1432] | [Open] | [UTR-002] |
| F-002 *(Example)* | REC-002 | [*Example:* QL-1433] | [Open] | [UTR-002] |
| F-003 *(Example)* | REC-003 | [*Example:* QL-1434] | [Open] | [UTR-002] |
| F-004 *(Example)* | REC-004 | [*Example:* QL-1440] | [Open] | [Release 5.3 analytics review] |
| F-005 *(Example)* | REC-005 | [*Example:* QL-1441] | [Open] | [Design review] |

```mermaid
flowchart LR
  F["Finding F-001 to F-005"] --> R["Recommendation REC-001 to REC-005"]
  R --> B["Backlog item"]
  B --> V["Retest or check"]
  V --> N["Next report UTR-002"]
```

Data retention: [*Example:* raw recordings for P01 to P08 deleted on YYYY-MM-DD as set in the
plan; pseudonymised notes and approved clips kept with this report].

## 10. Glossary and references

<!-- Guidance: Define the terms readers outside the research team need. Cite sources by name and clause, without copying their text. -->

| Term | Meaning in this report |
|---|---|
| Adjusted Wald interval | A confidence interval for a proportion that behaves well with small samples |
| Completion rate | Share of attempts that met the task's success criterion |
| Finding | A usability problem or strength observed in the sessions, with evidence |
| Severity | Rating of a problem's impact from 0 (not a problem) to 4 (catastrophe) |
| SEQ | Single Ease Question, a 7-point post-task rating |
| SUS | System Usability Scale, a 10-item questionnaire scored 0 to 100 |

References:

- ISO/IEC 25066:2016, Systems and software engineering, SQuaRE, Common Industry Format (CIF) for
  usability: evaluation report, first edition, published 2016-06-13. Report content elements in
  clause 5.2 (5.2.1 executive summary to 5.2.8 additional content for conformity assessment);
  Annex A lists required and recommended elements per evaluation type. ISO withdrew this edition
  on 2026-04-24 (stage 95.99) without naming a replacement, as shown on ISO's committee catalogue
  page and in ISO open data, checked 2026-10-30; the structure remains useful, but do not claim
  conformance to a current standard. Official text: ISO, purchase or licensed access (withdrawn
  editions may be available on request): https://www.iso.org/standard/63831.html
- ISO 9241-11:2018, Ergonomics of human-system interaction, Part 11: Usability: definitions and
  concepts, second edition. Clauses 6.2 to 6.4 (effectiveness, efficiency, satisfaction).
  Official text: ISO, purchase or licensed access: https://www.iso.org/standard/63500.html
- W3C, Web Content Accessibility Guidelines (WCAG) 2.2, https://www.w3.org/TR/WCAG22/
- Brooke, J. (1996). SUS: a "quick and dirty" usability scale. In *Usability Evaluation in Industry*. Taylor and Francis.
- Sauro, J. and Lewis, J. R. (2016). *Quantifying the User Experience*, 2nd edition. Morgan Kaufmann.
- Nielsen, J. (1994). Severity ratings for usability problems. Nielsen Norman Group,
  https://www.nngroup.com/articles/how-to-rate-the-severity-of-usability-problems/
- Related templates: design/usability-test-plan, design/research-synthesis,
  design/heuristic-evaluation, design/accessibility-checklist, quality/accessibility-audit.

## Standard and licence note

Structure aligned with the usability evaluation report content elements of ISO/IEC 25066:2016
(clause 5.2) and the usability measures of ISO 9241-11:2018. ISO standards are paid
publications: this template cites names, editions and clause numbers only, and all wording is
original. The SUS is attributed to Brooke (1996) and the severity scale to Nielsen (1994); WCAG
2.2 success criteria are cited by number and short name. Check the current official versions
before relying on them. Template licence: MIT.