# Fine-Tuning Run Report: [Run name or ID]

> Report for one fine-tuning run of a pretrained model: objective, base
> model and licence, data, method and hyperparameters, compute, training
> curves, evaluation against the baseline, safety evaluation, failure
> analysis, decision, artefacts and reproducibility. Based on common
> machine learning practice. Replace every `[bracketed]` value and every
> row marked *Example*. Worked examples use a LoRA fine-tune of an
> open-weight 8B instruction model for support ticket summarisation.

<!--
Guidance: How to use this template
- Write the objective and the success criteria (section 1) before the
  run starts, and do not change them after seeing results.
- Always compare with a baseline: the base model with the best prompt,
  and the current production model if there is one.
- Record exact versions and seeds; a run that cannot be reproduced
  cannot be audited or improved.
- Check the base model and data licences before training, not before
  release.
-->

| Field | Value |
|---|---|
| Run ID | [FT-2026-031] |
| Project | [Ticket summarisation] |
| Author | [@ml-engineer] |
| Reviewer | [@ml-lead] |
| Run dates | YYYY-MM-DD to YYYY-MM-DD |
| Status | [Planned / Running / Completed / Abandoned] |
| Decision | [Promote / Iterate / Reject] |
| Experiment tracking link | [link] |
| Related documents | [LLM evaluation plan], [Model card], [Datasheet for the dataset], [AI red team report] |

## 1. Objective and success criteria

<!-- Guidance: State the behaviour you want to change, why fine-tuning rather than prompting or retrieval, and measurable success criteria. -->

**Objective.** Produce ticket summaries of at most [80] words that keep
the customer's issue, steps already taken and requested outcome, in the
house style, at lower cost than the current [hosted large model].

**Why fine-tuning.** Prompting the base model reached [0.71] on the
rubric score with a [1,900]-token prompt; a smaller tuned model is
expected to match quality at about [one fifth] of the cost.

| ID | Criterion | Target |
|---|---|---|
| SC-01 | Rubric score on held-out set (0 to 1) | ≥ [0.80] |
| SC-02 | Factual consistency (no unsupported claims) | ≥ [97%] of summaries |
| SC-03 | Safety: refusal and toxicity regression vs base | No significant increase |
| SC-04 | Median latency on [GPU type] | ≤ [600] ms |

## 2. Base model and licence

<!-- Guidance: Identify the exact checkpoint and check that its licence allows your use, fine-tuning and distribution of derivatives. -->

| Field | Value |
|---|---|
| Base model | [Model name], [size], [revision hash] |
| Source | [Model hub link] |
| Licence | [Licence name and version] |
| Licence allows commercial use | [Yes / No / With conditions] |
| Licence conditions for derivatives | [Attribution, naming, acceptable use policy, user thresholds] |
| Reviewed by | [@legal or @oss-office], YYYY-MM-DD |

## 3. Data

<!-- Guidance: Describe sources, consent and licence, size, splits and every filter applied. Data issues explain most fine-tuning failures. -->

### 3.1 Sources

| Source | Records | Licence or legal basis | Personal data | Notes |
|---|---|---|---|---|
| *Example:* Resolved tickets 2024 to 2026 with agent-written summaries | 48,200 | Internal, privacy notice covers service improvement | Yes, pseudonymised | Summaries written by tier-2 agents |
| *Example:* Synthetic summaries from larger model, human-checked | 6,000 | Provider terms allow training | No | Covers rare categories |
| [Source] | [n] | [Basis] | [Yes / No] | [Notes] |

### 3.2 Filtering and cleaning

- Removed tickets with fewer than [3] messages or summaries under [10]
  words: [-3,900].
- Removed near-duplicates (MinHash similarity > [0.9]): [-2,100].
- Removed items flagged by PII scanner after pseudonymisation: [-410].
- Decontamination: removed any ticket whose ID or text appears in the
  evaluation sets: [-35].

### 3.3 Size and splits

| Split | Examples | Tokens (approx.) | Selection |
|---|---|---|---|
| Train | [43,000] | [38 M] | Random, stratified by category |
| Validation | [2,400] | [2.1 M] | Random |
| Test (held out) | [2,355] | [2.0 M] | Time-based: last 2 months |

## 4. Method and hyperparameters

<!-- Guidance: Name the method (full fine-tune, LoRA, QLoRA, preference optimisation) and give every setting needed to rerun. -->

Method: [LoRA supervised fine-tuning on chat-formatted pairs, loss on
assistant tokens only].

| Hyperparameter | Value |
|---|---|
| LoRA rank $r$ / alpha / dropout | [16] / [32] / [0.05] |
| Target modules | [q, k, v, o, gate, up, down projections] |
| Trainable parameters | [42 M (0.5%)] |
| Learning rate and schedule | [2e-4], cosine, [100] warm-up steps |
| Effective batch size | [64] sequences |
| Max sequence length | [4,096] tokens |
| Epochs | [3] |
| Optimiser | [AdamW], weight decay [0.0] |
| Precision | [bf16] |
| Seed | [42] |

With LoRA the weight update is a low-rank product $\Delta W = BA$ with
$B \in \mathbb{R}^{d \times r}$ and $A \in \mathbb{R}^{r \times k}$,
scaled by $\alpha / r$, so only $r(d + k)$ parameters per matrix train.

## 5. Compute

<!-- Guidance: Record hardware, time, cost and, where you report it, energy estimate. -->

| Item | Value |
|---|---|
| Hardware | [4 x 80 GB GPUs], [cloud region] |
| Training time | [5 h 40 min] |
| GPU hours | [22.7] |
| Cost | [about 95 EUR] |
| Framework versions | [library and version list or lock file link] |

## 6. Training curves

<!-- Guidance: Describe the curves in words and link the plots: convergence, overfitting, instabilities. -->

[Description and link to plots.] *Example:* "Training loss fell from
1.92 to 0.88 in the first epoch and to 0.71 by the end. Validation loss
reached its minimum 0.83 at step 1,650 (epoch 2.5) and rose slightly to
0.85 at the end, so the checkpoint at step 1,650 was selected. No loss
spikes occurred; gradient norm stayed below 1.0 after warm-up."

## 7. Evaluation against baseline

<!-- Guidance: Use the same test set, decoding settings and judges for all candidates. Report intervals where the sample is small. -->

| Metric | Base model + best prompt | Production model | Fine-tuned (step 1,650) | Target |
|---|---|---|---|---|
| *Example:* Rubric score (LLM judge, calibrated) | 0.71 | 0.82 | 0.84 | ≥ 0.80 |
| *Example:* Factual consistency | 94.1% | 97.6% | 97.9% | ≥ 97% |
| *Example:* Length ≤ 80 words | 61% | 92% | 99% | ≥ 95% |
| *Example:* Human preference vs production (n = 300) | n/a | baseline | 56% win, 31% loss, 13% tie | > 50% win |
| *Example:* Median latency | 410 ms | 1,800 ms | 430 ms | ≤ 600 ms |
| *Example:* Cost per 1,000 summaries | 0.30 EUR | 4.10 EUR | 0.30 EUR | n/a |

## 8. Safety evaluation

<!-- Guidance: Fine-tuning can weaken safety behaviour even on benign data. Re-run safety tests from the base model. -->

| Test | Base model | Fine-tuned | Result |
|---|---|---|---|
| *Example:* Harmful request refusal rate (500 prompts) | 97.2% | 96.8% | No significant change |
| *Example:* Prompt injection inside ticket text (200 cases) | 8.5% success | 4.0% success | Improved |
| *Example:* PII reproduction from training data (canary test, 50 canaries) | n/a | 0 reproduced | Pass |
| *Example:* Toxic language in output | 0.2% | 0.1% | Pass |

## 9. Failure analysis

<!-- Guidance: Look at the worst outputs, group them, and say what would fix each group. -->

| Failure group | Share of test errors | Example | Likely cause | Fix |
|---|---|---|---|---|
| *Example:* Missing requested outcome | 38% | Summary omits "wants replacement" | Training summaries often omit it | Filter or rewrite such examples |
| *Example:* Long multi-issue tickets merged | 27% | Two issues summarised as one | Few multi-issue examples | Add 2,000 multi-issue examples |
| *Example:* Wrong product name | 11% | Model confuses similar products | Pseudonymisation replaced product codes | Keep product codes |

## 10. Decision

<!-- Guidance: State the decision, the reasons, and conditions such as shadow testing. -->

Decision: **[Promote to shadow / Promote / Iterate / Reject]**.
*Example:* "Promote to a 2-week shadow deployment. SC-01, SC-02 and
SC-04 are met; SC-03 shows no regression. Next run will address the
missing-outcome failure group."

| Role | Name | Decision | Date |
|---|---|---|---|
| ML lead | [@name] | [Decision] | YYYY-MM-DD |
| Product owner | [@name] | [Decision] | YYYY-MM-DD |
| Responsible AI | [@name] | [Decision] | YYYY-MM-DD |

## 11. Artefacts and reproducibility

<!-- Guidance: List everything needed to rerun and audit the run. -->

| Artefact | Location | Version or hash |
|---|---|---|
| Training code | [repo link] | [commit] |
| Config file | [path] | [hash] |
| Dataset snapshot | [storage path] | [hash] |
| Adapter weights | [registry link] | [version] |
| Evaluation outputs | [storage path] | [run ID] |
| Environment | [container image] | [digest] |

Rerun command: `[train --config configs/ft-2026-031.yaml --seed 42]`.

## Standard and licence note

Based on common industry practice for documenting fine-tuning
experiments of machine learning models. Original text written for this
collection; example numbers are illustrative. Check the licences of the
base model and the training data for your own use. Template licence: MIT.