Model Validation Report Template (SR 26-2 / SR 11-7)

Independent model validation: soundness, data, outcomes and back-testing, benchmarking, stress tests, findings, conclusion

AI 与数据完整Federal Reserve SR 26-2 Supervisory Guidance on Model Risk Management (2026), superseding SR 11-7model validation report math更新于 2026年10月20日

在编辑器中打开下载 .md

使用方法: Use it when an independent validator reviews a quantitative or ML model before use, after change or periodically. Fill scope, independence and materiality first, then test against thresholds agreed in advance. The main pitfall is a conclusion without evidence-backed findings and conditions of use.

预览

Model Validation Report: [Model name] [version]

Independent validation report for a quantitative or machine learning model: model overview and use, materiality, conceptual soundness, data quality, outcomes analysis and back-testing, benchmarking, sensitivity and stress testing, implementation checks, ongoing monitoring, limitations, findings and the validation conclusion. Structure follows the components of model validation in the US banking agencies' Supervisory Guidance on Model Risk Management attached to Federal Reserve SR letter 26-2 (17 April 2026), which supersedes SR 11-7 (2011). Replace every [bracketed] value and every row marked Example. Worked examples use a retail credit probability of default (PD) scorecard.

Document control

Field	Value
Report ID	[MV-2026-014]
Model ID in inventory	[MDL-0231]
Model name and version	[Retail PD scorecard v4.0]
Validation type	[Initial / Periodic / Change / Targeted]
Model tier (materiality)	[Tier 1 high / Tier 2 medium / Tier 3 low]
Version	[0.1]
Status	[Draft / In review / Final]
Lead validator	[@validator]
Model owner	[@model-owner]
Model developer	[@developer]
Classification	[Confidential]
Last updated	YYYY-MM-DD
Related documents	[Model development document], [Model card], [ML system design], [Monitoring plan]

Approvals

Role	Name	Decision	Date
Lead validator	[@name]	[Final]	YYYY-MM-DD
Head of model validation	[@name]	[Approved / Changes requested]	YYYY-MM-DD
Model owner (acknowledges findings)	[@name]	[Acknowledged]	YYYY-MM-DD
Model risk committee	[Committee]	[Approved for use / Conditional / Not approved]	YYYY-MM-DD

Revision history

Version	Date	Author	Change summary
0.1	YYYY-MM-DD	[@name]	Draft findings
0.2	YYYY-MM-DD	[@name]	Example: Developer responses added; MV-F-03 downgraded after new evidence
1.0	YYYY-MM-DD	[@name]	Final report

Executive summary
Validation scope and approach
Model overview and use
Conceptual soundness
Data quality review
Outcomes analysis and back-testing
Benchmarking
Sensitivity and stress testing
Implementation and process verification
Ongoing monitoring plan
Limitations and compensating controls
Third-party components
Findings
Validation conclusion and conditions
Sign-off
Glossary and references

1. Executive summary

Item	Summary
Model purpose	[PD estimation for unsecured personal loans, used in approval and pricing]
Conclusion	[Fit for purpose / Fit for purpose with conditions / Not fit for purpose]
Findings	[0] critical, [1] high, [3] medium, [2] low
Key conditions	[Conditions of use and deadlines]
Next validation	YYYY-MM-DD

Example: "The scorecard is conceptually sound and discriminates well (out-of-time Gini 0.58). Calibration is weak for applicants with thin credit files (MV-F-01, high). The model is approved for use with a conservative overlay for thin-file applicants until recalibration by 31 March 2027."

2. Validation scope and approach

2.1 Scope

In scope: [model methodology, development data, code, implementation in the decision engine, monitoring reports]. Out of scope: [upstream bureau data quality controls, covered by DQ review DQ-2026-07].

2.2 Evidence reviewed

Evidence	Version	Provided by
Model development document	v4.0, YYYY-MM-DD	[@developer]
Development and test datasets	[snapshot IDs]	[@data-owner]
Model code repository	[commit]	[@developer]
Production scoring logs	[period]	[@ops]

2.3 Independence and approach

The validation team reports to [Chief Risk Officer] and had no role in development. Work comprised document review, replication of key results from code and data, independent tests and interviews with developers and users over [6] weeks.

2.4 Limitations of this validation

State anything that limited the work, so readers can weigh the conclusion. Example: "Default outcomes for the 2025 cohort are not yet mature, so out-of-time testing uses 2024 applications only. The vendor did not provide the bureau score methodology; we relied on outcomes analysis on our own portfolio instead. Developers were available for interviews throughout, and no requested evidence was refused."

3. Model overview and use

3.1 Purpose and intended use

[Model name] estimates the 12-month probability of default for [applicants for unsecured loans up to 25,000 EUR]. Outputs are used for [approval cut-offs, risk-based pricing and expected credit loss input].

3.2 Model description

Attribute	Value
Method	[Logistic regression on WoE-binned variables / gradient boosted trees]
Inputs	[14 variables: bureau score, utilisation, months on book]
Output	[PD in [0, 1], mapped to 10 risk grades]
Development sample	[210,000 applications, 2019-01 to 2023-12]
Default definition	[90+ days past due within 12 months]

3.3 Materiality

Materiality reflects inherent risk (complexity, assumptions, data limits) and model exposure and purpose.

Factor	Assessment	Rating
Exposure	Example: 1.8 bn EUR portfolio, 9,000 decisions per week	High
Purpose	Example: Credit decisions and financial reporting input	High
Inherent risk	Example: Moderate complexity, stable data	Medium
Overall tier		Tier 1

4. Conceptual soundness

Area	Validator assessment	Rating
Choice of method	Example: Logistic regression appropriate for interpretable PD; alternatives considered	Satisfactory
Variable selection	Example: Economic rationale given for 13 of 14 variables; "device type" lacks rationale	Needs improvement
Key assumptions	Example: Stable relationship between bureau score and default through the cycle; partly tested	Needs improvement
Developmental testing	Example: Out-of-sample and out-of-time tests performed	Satisfactory
Interpretability	Example: Monotonic WoE bins; reason codes available	Satisfactory
Expert judgement and overlays	Example: Two manual bin merges documented	Satisfactory

5. Data quality review

Check	Result	Comment
Completeness	Example: 2.1% missing utilisation, imputed as separate bin	Acceptable
Accuracy	Example: 500-record sample reconciled to source systems, 0 breaks	Acceptable
Representativeness	Example: Thin-file applicants 6% of sample vs 14% of current flow	Weakness, see MV-F-01
Default definition consistency	Example: Same definition in development and production	Acceptable
Data lineage	Example: Documented in data pipeline spec v2	Acceptable

6. Outcomes analysis and back-testing

6.1 Metrics and thresholds

Discrimination is measured with the Gini coefficient derived from the area under the ROC curve:

\text{Gini} = 2 \cdot \text{AUC} - 1

Calibration per risk grade $g$ is tested with a binomial test of the observed defaults $d_g$ among $n_g$ accounts against the predicted mean PD $\bar{p}_g$ :

z_g = \frac{d_g - n_g \bar{p}_g}{\sqrt{n_g \bar{p}_g (1 - \bar{p}_g)}}

Overall calibration error is summarised by the Brier score over $N$ accounts with predictions $p_i$ and outcomes $y_i \in \{0, 1\}$ :

\text{BS} = \frac{1}{N} \sum_{i=1}^{N} (p_i - y_i)^2

Metric	Green	Amber	Red
Gini (out-of-time)	≥ 0.50	0.40 to 0.50	< 0.40
Grades failing binomial test at 95%	0 to 1	2	≥ 3
Population stability index	< 0.10	0.10 to 0.25	> 0.25

6.2 Results

Sample	N	Defaults	Gini	Brier	Grades failing	Status
Example: Development	147,000	5,880	0.62	0.036	0	Green
Example: Out-of-sample	63,000	2,490	0.60	0.037	1	Green
Example: Out-of-time 2024	58,400	2,610	0.58	0.041	2	Amber
Example: Thin-file segment (OOT)	8,100	610	0.41	0.071	3	Red

Example: For grade 7 in the out-of-time sample, $n_g = 4{,}200$ , $\bar{p}_g = 0.050$ and $d_g = 252$ give $z_g \approx 3.0$ , so observed defaults exceed prediction at the 95% level.

7. Benchmarking

Benchmark	Gini (OOT)	Comment
Example: Current production v3.2	0.54	New model improves rank ordering
Example: Validator challenger (gradient boosting)	0.61	Small gain, lower interpretability
Example: Bureau generic score alone	0.49	Model adds information

8. Sensitivity and stress testing

Test	Change applied	Effect on portfolio mean PD	Assessment
Example: Input shock	Bureau score -20 points for all	+38%	Plausible, monotonic
Example: Missing data	Utilisation missing for 20%	+4%	Robust
Example: Macroeconomic stress	Unemployment +3 points via overlay	+55%	Overlay documented; within range
Example: Variable removal	Drop "device type"	Gini -0.004	Variable not needed (see MV-F-02)

9. Implementation and process verification

Check	Result
Example: Replication of 10,000 production scores from logged inputs	100% match to 6 decimals
Example: Version in decision engine equals validated version	Confirmed, artefact hash recorded
Example: Change control for cut-offs	Cut-off changes require committee approval; confirmed
Example: Input data feeds	Two feeds lack reconciliation alerts (MV-F-04)

10. Ongoing monitoring plan

Metric	Frequency	Threshold	Action on breach	Owner
Gini on matured cohorts	Quarterly	< 0.50 amber, < 0.40 red	Investigate; red triggers revalidation	[@model-owner]
Calibration by grade	Quarterly	≥ 2 grades failing	Recalibration review	[@model-owner]
PSI of score distribution	Monthly	> 0.10 amber, > 0.25 red	Root cause analysis	[@ml-ops]
Override rate	Monthly	> 8%	Review with credit policy	[@credit-policy]
Thin-file segment Gini	Quarterly	< 0.45	Keep overlay; escalate	[@model-owner]

11. Limitations and compensating controls

Limitation	Impact	Compensating control
Example: Under-predicts PD for thin-file applicants	Under-pricing, higher losses	+1 risk grade overlay; manual review above 10,000 EUR
Example: Development data mostly benign economy	Stress behaviour uncertain	Macroeconomic overlay; annual stress test
Example: Not validated for small business loans	Misuse risk	Inventory restriction; decision engine blocks other products

12. Third-party components

Component	Vendor	Evidence obtained	Validator testing	Gap
Example: Bureau generic score	[Bureau]	Vendor validation summary 2026	Outcomes on own portfolio	Methodology proprietary
[Component]	[Vendor]	[Evidence]	[Tests]	[Gap]

13. Findings

Severity scale: Critical (model not fit for purpose), High (material weakness, condition of use required), Medium (weakness to fix within [6] months), Low (improvement).

ID	Area	Finding	Severity	Required action	Owner	Due	Status
Example: MV-F-01	Outcomes	Poor calibration and discrimination for thin-file applicants	High	Overlay now; recalibrate with 2025 data	[@developer]	YYYY-MM-DD	Open
Example: MV-F-02	Conceptual	"Device type" lacks economic rationale and adds no lift	Medium	Remove or justify	[@developer]	YYYY-MM-DD	Open
Example: MV-F-03	Data	Under-representation of 2020 to 2021 period not discussed	Low	Document effect	[@developer]	YYYY-MM-DD	Open
Example: MV-F-04	Implementation	Two input feeds lack reconciliation alerts	Medium	Add alerts	[@ml-ops]	YYYY-MM-DD	Open

14. Validation conclusion and conditions

Conclusion: [Fit for purpose with conditions].

Conditions of use:

[Thin-file overlay applied from go-live, confirmed by @credit-policy.]
[MV-F-01 closed by YYYY-MM-DD; otherwise use for thin-file applicants stops.]
[Quarterly monitoring report submitted to the model risk committee.]

Next full validation: [YYYY-MM-DD, or earlier on material change].

15. Sign-off

Role	Name	Statement	Date
Lead validator	[@name]	I performed this validation independently of development	YYYY-MM-DD
Head of model validation	[@name]	Conclusion endorsed	YYYY-MM-DD
Model owner	[@name]	Findings acknowledged; actions accepted	YYYY-MM-DD
Model risk committee chair	[@name]	[Decision]	YYYY-MM-DD

16. Glossary and references

Term	Meaning
Effective challenge	Critical analysis by objective, competent people with the standing to effect change
Outcomes analysis	Comparison of model outputs with actual outcomes
Back-testing	Outcomes analysis on historical periods not used for development
Benchmarking	Comparison with alternative models or estimates
Materiality	Importance of a model, from its exposure and purpose
PSI	Population stability index, a measure of distribution shift

References: Board of Governors of the Federal Reserve System, SR 26-2 "Revised Guidance on Model Risk Management" (17 April 2026), https://www.federalreserve.gov/supervisionreg/srletters/SR2602.pdf; [model development document]; [model monitoring plan].

Standard and licence note

Based on the Supervisory Guidance on Model Risk Management issued by the Federal Reserve, FDIC and OCC with SR letter 26-2 (17 April 2026), which supersedes SR letter 11-7 (4 April 2011); US government work in the public domain. Headings follow its components of validation (conceptual soundness, outcomes analysis, ongoing monitoring, vendor products) in original wording; no text is reproduced. The guidance is non-binding and aimed at banking organisations; check current supervisory expectations for your jurisdiction. Template licence: MIT.