# Model Monitoring Plan: [Model name] [version]

> Plan for monitoring a machine learning model in production: what is
> measured (performance, data drift, concept drift, data quality,
> fairness, latency and cost), thresholds and alerts, dashboards,
> retraining triggers, ownership and the link to incident management.
> Based on common MLOps practice. Replace every `[bracketed]` value and
> every row marked *Example*. Worked examples use a fraud detection model
> scoring card transactions.

<!--
Guidance: How to use this template
- Start from the decisions the model supports and the harm a silent
  failure would cause; monitor what would reveal that harm first.
- Many labels arrive late (fraud chargebacks take weeks). Pair delayed
  performance metrics with immediate proxies such as drift and score
  distribution.
- Every alert needs an owner and a runbook entry; an alert nobody acts
  on is noise.
- Review thresholds after the first [8] weeks of real data, then quarterly.
-->

| Field | Value |
|---|---|
| Plan ID | [MON-MDL-0231] |
| Model and version | [Fraud scorer v7.2] |
| Model tier | [Tier 1 high / Tier 2 medium / Tier 3 low] |
| Version | [0.1] |
| Status | [Draft / Approved / Superseded] |
| Owner | [@model-owner] |
| On-call team | [@ml-platform-oncall] |
| Last updated | YYYY-MM-DD |
| Related documents | [Model card], [Model validation report], [ML system design], [AI incident report template], [Data pipeline spec] |

## 1. Model and monitoring objectives

<!-- Guidance: State what the model does, the decision it drives and what the monitoring must protect against. -->

[Model name] scores [every card-not-present transaction, about 3.2
million per day] and blocks transactions above score [0.92] or sends
them to review between [0.75] and [0.92]. Monitoring must detect, within
[24 hours], any change that would raise missed fraud or false declines
by more than [10%] relative to the validated baseline.

| Objective | Failure it guards against |
|---|---|
| Keep fraud capture at the validated level | Concept drift as fraud patterns change |
| Avoid excessive false declines | Score inflation after an upstream change |
| Protect fairness across customer groups | Uneven error rates by region or card type |
| Keep the service within latency and cost budget | Timeouts that default to "allow" |

## 2. Metrics

<!-- Guidance: List each metric with definition, data source, granularity and label delay. Separate leading indicators (drift, data quality) from lagging ones (performance). -->

### 2.1 Performance

| Metric | Definition | Label delay | Granularity |
|---|---|---|---|
| Recall at operating point | Share of confirmed fraud scored above block or review cut-off | 30 to 60 days | Weekly cohort |
| Precision of blocks | Share of blocked transactions confirmed fraudulent | 30 days | Weekly cohort |
| False decline rate | Blocked transactions later confirmed genuine per 10,000 | 7 days (customer contact) | Daily |
| AUC-PR | Area under precision-recall curve | 60 days | Monthly |

### 2.2 Data drift

Distribution shift of inputs and scores is measured with the population
stability index (PSI). For a variable split into bins $i$, with expected
(baseline) share $e_i$ and actual (current) share $a_i$:

$$
PSI = \sum_i (a_i - e_i)\ln\left(\frac{a_i}{e_i}\right)
$$

Use [10] bins from baseline deciles and add a small constant (for example
$10^{-4}$) to empty bins. *Example:* if one bin moves from $e = 0.10$ to
$a = 0.16$, its term is $0.06 \times \ln(1.6) \approx 0.028$.

| Monitored distribution | Baseline | Frequency |
|---|---|---|
| Model score | Validation sample, 2026-Q2 | Hourly and daily |
| Top 15 features by importance | Same | Daily |
| Categorical features (merchant category, country) | Same | Daily, with chi-square test |

### 2.3 Concept drift

Concept drift is a change in the relationship between inputs and the
outcome. Detect it with performance on matured labels, with calibration
by score band, and with early signals such as the fraud team's manual
case findings and the share of chargebacks on low-score transactions.

### 2.4 Data quality

| Check | Rule | Frequency |
|---|---|---|
| Missing values | Share of nulls per feature within ±2 points of baseline | Hourly |
| Schema | Types and allowed values match data contract | Every batch |
| Freshness | Feature store lag ≤ [5] minutes | Every minute |
| Volume | Transactions scored within ±25% of same hour last week | Hourly |
| Range | Amounts, ages and counts within validated bounds | Hourly |

### 2.5 Fairness

| Metric | Groups | Rule |
|---|---|---|
| False decline rate ratio | Customer region, card type, age band (where lawful) | Ratio of highest to lowest ≤ [1.25] |
| Recall gap | Same | Gap ≤ [5] points |

### 2.6 Latency and cost

| Metric | Target |
|---|---|
| p99 scoring latency | ≤ [40] ms |
| Error or timeout rate | ≤ [0.05%] |
| Cost per 1,000 scores | ≤ [0.02] EUR |

## 3. Thresholds and alerts

<!-- Guidance: Use two levels: warning (investigate in working hours) and critical (page on-call). Avoid thresholds so tight that they fire weekly. -->

| Metric | Warning | Critical | Route | Runbook |
|---|---|---|---|---|
| Score PSI (daily) | > 0.10 | > 0.25 | Slack #fraud-model / PagerDuty | RB-01 |
| Feature PSI, any top-15 feature | > 0.10 | > 0.25 | Slack / ticket | RB-02 |
| False decline rate | > 12 per 10,000 | > 20 per 10,000 | PagerDuty | RB-03 |
| Weekly recall (matured) | < 0.80 | < 0.72 | Ticket to model owner | RB-04 |
| Missing values, any feature | +2 points | +10 points | PagerDuty | RB-05 |
| p99 latency | > 40 ms | > 80 ms | PagerDuty | RB-06 |
| False decline ratio across groups | > 1.25 | > 1.5 | Ticket to model owner and fairness lead | RB-07 |

## 4. Dashboards

<!-- Guidance: Name dashboards, audiences and what each must show at a glance. -->

| Dashboard | Audience | Content | Link |
|---|---|---|---|
| Operational health | On-call | Volume, latency, errors, data quality | [link] |
| Model behaviour | Model owner | Score distribution, PSI, block and review rates | [link] |
| Performance and fairness | Model owner, risk | Matured recall and precision, group metrics | [link] |
| Executive summary | Risk committee | Monthly trend, open alerts, retraining status | [link] |

## 5. Retraining triggers

<!-- Guidance: Define when to retrain, recalibrate or roll back, and who decides. Retraining without validation is a change like any other. -->

| Trigger | Action | Decision by |
|---|---|---|
| Matured recall below critical for 2 consecutive weeks | Retrain with last [6] months of labels | Model owner, with validation sign-off |
| Score PSI > 0.25 with stable fraud rate | Investigate upstream; recalibrate if cause is genuine shift | Model owner |
| New fraud pattern confirmed by fraud team | Add features or rules; retrain | Model owner and fraud lead |
| Scheduled | Retrain every [quarter] | Model owner |
| Upstream breaking change | Roll back to last good feature version or model | On-call, then model owner |

Every retrained model passes the [model validation] change process and a
shadow period of [7] days before promotion.

## 6. Ownership

<!-- Guidance: Use a RACI so that alerts, investigations and decisions are clearly owned. -->

| Activity | Responsible | Accountable | Consulted | Informed |
|---|---|---|---|---|
| Alert response | [@ml-platform-oncall] | [@ml-platform-lead] | [@model-owner] | [#fraud-model] |
| Drift investigation | [@data-scientist] | [@model-owner] | [@data-eng] | [Risk] |
| Retraining decision | [@model-owner] | [@head-of-fraud-analytics] | [Model validation] | [Risk committee] |
| Threshold review | [@model-owner] | [@head-of-fraud-analytics] | [Model validation] | [On-call] |

## 7. Incident linkage

<!-- Guidance: Say when a monitoring signal becomes an incident and which process applies. -->

A critical alert that persists for more than [2] hours, any confirmed
harm to customers, or a fairness ratio above [1.5] opens an incident
using the [AI incident report] template and the [incident management
procedure]. The incident lead decides on containment, such as routing
all scores between [0.75] and [0.98] to manual review or rolling back
the model. Post-incident actions update this plan.

## 8. Review and change log

<!-- Guidance: Record threshold and metric changes with reasons. -->

| Version | Date | Change | Approved by |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | First draft | [@name] |
| *Example:* 0.2 | YYYY-MM-DD | Raised score PSI warning to 0.10 after 8 weeks of noise at 0.05 | [@model-owner] |

## Standard and licence note

Based on common industry practice for production machine learning
monitoring (MLOps), including the population stability index as a
standard drift statistic. Original text written for this collection;
example numbers are illustrative. Template licence: MIT.