Model Monitoring Plan: [Model name] [version]

Plan for monitoring a machine learning model in production: what is measured (performance, data drift, concept drift, data quality, fairness, latency and cost), thresholds and alerts, dashboards, retraining triggers, ownership and the link to incident management. Based on common MLOps practice. Replace every [bracketed] value and every row marked Example. Worked examples use a fraud detection model scoring card transactions.

1. Model and monitoring objectives

[Model name] scores [every card-not-present transaction, about 3.2 million per day] and blocks transactions above score [0.92] or sends them to review between [0.75] and [0.92]. Monitoring must detect, within [24 hours], any change that would raise missed fraud or false declines by more than [10%] relative to the validated baseline.

2. Metrics

2.1 Performance

2.2 Data drift

Distribution shift of inputs and scores is measured with the population stability index (PSI). For a variable split into bins $i$, with expected (baseline) share $e_i$ and actual (current) share $a_i$:

Use [10] bins from baseline deciles and add a small constant (for example $10^{-4}$) to empty bins. Example: if one bin moves from $e = 0.10$ to $a = 0.16$, its term is $0.06 \times \ln(1.6) \approx 0.028$.

2.3 Concept drift

Concept drift is a change in the relationship between inputs and the outcome. Detect it with performance on matured labels, with calibration by score band, and with early signals such as the fraud team's manual case findings and the share of chargebacks on low-score transactions.

2.4 Data quality

2.5 Fairness

2.6 Latency and cost

3. Thresholds and alerts

4. Dashboards

5. Retraining triggers

Every retrained model passes the [model validation] change process and a shadow period of [7] days before promotion.

6. Ownership

7. Incident linkage

A critical alert that persists for more than [2] hours, any confirmed harm to customers, or a fairness ratio above [1.5] opens an incident using the [AI incident report] template and the [incident management procedure]. The incident lead decides on containment, such as routing all scores between [0.75] and [0.98] to manual review or rolling back the model. Post-incident actions update this plan.

8. Review and change log

Standard and licence note

Based on common industry practice for production machine learning monitoring (MLOps), including the population stability index as a standard drift statistic. Original text written for this collection; example numbers are illustrative. Template licence: MIT.