# System Design: [Fraud platform name] real-time payment fraud detection

> Reference design for scoring card and account-to-account payments for
> fraud at authorisation time. It covers the decision API and its latency
> budget, features computed from the transaction and from streaming
> aggregates, the online and offline feature stores, rules combined with a
> machine-learning score, decisions (allow, challenge, review, block),
> manual review queues, label delay, feedback loops and selection bias, and
> model monitoring. It follows the structure of the System Design Document
> template and is pre-filled with a worked design to adapt. Replace every
> `[bracketed]` value and every row marked *Example*; values marked
> *Example* are assumptions, not benchmarks. This is an engineering format,
> not legal, card-network or supervisory advice.

<!--
Guidance: How to use this template
- Start from the decision, not the model: what happens to a payment at
  each score band, who pays for a missed fraud, who pays for a false
  decline. Write those costs in 2.2 before choosing thresholds.
- Fix the latency budget early. The score must return inside the
  authorisation path; every feature has to be available within that
  budget or be dropped from the online model.
- Plan for labels that arrive weeks later. Training, evaluation and
  monitoring all depend on how you handle the delay.
- Model governance (validation, monitoring, documentation) lives in
  Model Monitoring Plan (`ai-data/model-monitoring-plan`) and, for banking
  organisations, Model Validation Report (`ai-data/model-validation-report`).
  This template covers the system that runs the model.
-->

## Document control

<!-- Guidance: A design is a proposal until it is approved; record who decided and when, including the risk owner who accepts the loss and decline targets. -->

| Field | Value |
|---|---|
| Document ID | [SDD-FRD-001] |
| System / feature | [Real-time fraud scoring] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Owner | [@owner] |
| Author(s) | [@author] |
| Approvers / reviewers | [@reviewer-1], [@reviewer-2] |
| Last updated | YYYY-MM-DD |
| Classification | [Confidential] |
| Payment types in scope | [Card not present / card present / bank transfer / wallet] |
| Model risk tier | [High / Medium / Low per internal policy] |
| Related | [Model card], [Model monitoring plan], [Data contract], [DPIA], [Threat model] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Tech lead | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Data science lead | [@name] | [Decision] | YYYY-MM-DD |
| Fraud risk owner | [@name] | [Decision] | YYYY-MM-DD |
| Fraud operations (review) | [@name] | [Decision] | YYYY-MM-DD |
| Privacy | [@name] | [Decision] | YYYY-MM-DD |
| SRE / operations | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Note what changed in the design, not the wording. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft for design review |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Moved velocity features to the streaming pipeline |
| 0.3 | YYYY-MM-DD | [@name] | *Example:* Added exploration sample for blocked traffic |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved |

## Contents

<!-- Guidance: Keep aligned with the headings. -->

1. [Overview](#1-overview)
2. [Requirements](#2-requirements)
3. [Capacity estimation](#3-capacity-estimation)
4. [API design](#4-api-design)
5. [Data model](#5-data-model)
6. [High-level design](#6-high-level-design)
7. [Detailed design](#7-detailed-design)
8. [Scalability and performance](#8-scalability-and-performance)
9. [Reliability, security and operations](#9-reliability-security-and-operations)
10. [Trade-offs and alternatives](#10-trade-offs-and-alternatives)
11. [Delivery plan and risks](#11-delivery-plan-and-risks)
12. [Appendices](#12-appendices)

## 1. Overview

<!-- Guidance: One page that a busy reviewer can read alone: problem, proposed solution, decisions needed and the interview path. -->

### 1.1 Problem statement

[*Example:* We process payments for 120 000 merchants. Fraud losses rose
to 0.11% of volume last year while our false decline rate is estimated at
2%, which merchants report as lost sales. Today's controls are 900 static
rules evaluated in the payment service; nobody knows which rules still
work. Features such as "cards used on this device in the last hour" are
computed by a nightly batch, so a fraud ring testing stolen cards is
visible only the next morning. The review team receives 40 000 cases a
day and works them in arrival order.]

### 1.2 Proposed solution

[*Example:* A fraud decision service is called synchronously in the
authorisation path with a 100 ms p99 budget. It joins request features with
aggregates kept fresh by a streaming pipeline in an online feature store,
runs a gradient-boosted model and a versioned rule set, and returns a
decision with reasons. Thresholds map scores to allow, 3-D Secure
challenge, review or block per merchant risk profile. Review cases are
prioritised by expected loss. Labels from chargebacks, fraud reports and
review outcomes flow back into an offline store with point-in-time
correct features; models retrain on a fixed cadence and ship through
shadow and canary stages under monitoring.]

### 1.3 Decisions requested from reviewers

| ID | Question | Recommended option | Section |
|---|---|---|---|
| Q-01 | *Example:* Behaviour when the fraud service times out? | Fail open below *Example* USD 500, fail to challenge above | 7.1, 9.1 |
| Q-02 | *Example:* Streaming or batch velocity features? | Streaming for windows up to 24 h; batch for longer | 7.2 |
| Q-03 | *Example:* Rules and model together or separately? | Rules for hard policy and fast response to attacks; model for ranking risk | 7.3 |
| Q-04 | *Example:* How to measure fraud in blocked traffic? | Small random exploration sample with loss cap | 7.5 |
| Q-05 | *Example:* Review queue order? | Expected loss (score x amount), with SLA per merchant tier | 7.4 |
| [Q-nn] | [Question] | [Option] | [Section] |

### 1.4 45-minute interview path

<!-- Guidance: The interesting parts are the latency budget, fresh features and delayed labels. Do not spend the deep-dive time on model architecture. -->

| Minute | Cover | Notes |
|---|---|---|
| 0 to 5 | Scope: payment types, who decides (us or merchant), latency budget | Name the cost of a miss and of a false decline |
| 5 to 10 | Core requirements; below the line (AML, account opening, chargeback disputes) | |
| 10 to 15 | Estimates with consequences (3.5) | Feature reads per second and review capacity |
| 15 to 20 | Decision API and data model with access patterns | Entities: payment, card, device, account, merchant |
| 20 to 30 | High-level design, then deep dive 1: streaming features and train-serve parity (7.2) | Point-in-time joins |
| 30 to 40 | Deep dive 2: labels and feedback loops (7.5) or rules plus model (7.3) | Pick 7.5 to show maturity |
| 40 to 45 | Failure modes, monitoring, 10x and 0.1x | End on fail-open versus fail-closed |

## 2. Requirements

<!-- Guidance: Split the core from what is explicitly below the line, and attach a number or a cost to every quality. -->

### 2.1 Core functional requirements

| ID | Requirement | Priority |
|---|---|---|
| FR-01 | *Example:* Score every payment before authorisation and return a decision with reason codes | Must |
| FR-02 | *Example:* Compute velocity and aggregate features per card, device, email, IP and merchant over sliding windows | Must |
| FR-03 | *Example:* Evaluate versioned rules written by fraud analysts, with backtest before activation | Must |
| FR-04 | *Example:* Map scores to decisions with per-merchant thresholds | Must |
| FR-05 | *Example:* Send payments in the review band to a prioritised review queue | Must |
| FR-06 | *Example:* Ingest labels from chargebacks, issuer fraud reports, customer reports and review outcomes | Must |
| FR-07 | *Example:* Retrain, evaluate and deploy models through shadow and canary stages | Must |
| FR-08 | *Example:* Explain a decision to a merchant or customer support agent in plain reason codes | Should |
| [FR-nn] | [Requirement] | [Must / Should / Could] |

### 2.2 Non-functional requirements and decision economics

| ID | Quality | Target | Measured as |
|---|---|---|---|
| NFR-01 | Decision latency | *Example:* p99 at or below 100 ms, p50 at or below 30 ms | Service timing |
| NFR-02 | Availability | *Example:* 99.99% of payments receive a decision or a defined fallback | SLO |
| NFR-03 | Feature freshness | *Example:* Velocity features reflect events older than 2 s | Event-to-store lag |
| NFR-04 | Fraud loss | *Example:* Below [x] basis points of volume | Mature labels (7.5) |
| NFR-05 | False decline rate | *Example:* Below [y]% estimated from exploration sample | 7.5 |
| NFR-06 | Review SLA | *Example:* 95% of review cases decided within 2 h | Queue metrics |
| NFR-07 | Train-serve parity | *Example:* Online and offline feature values agree for 99.9% of sampled requests | Parity job |
| [NFR-nn] | [Quality] | [Target] | [Measurement] |

Decision economics *(Example)*: a missed fraud costs the amount plus a
chargeback fee and network penalties; a false decline costs the merchant
margin and a share of customer lifetime value. The threshold for blocking
sits where the expected fraud cost of allowing equals the expected cost of
declining a good payment; a challenge band sits between allow and block
where 3-D Secure shifts liability or adds friction.

### 2.3 Below the line (out of scope)

- [*Example:* anti-money-laundering transaction monitoring (separate system, different labels)]
- [*Example:* account opening and identity verification]
- [*Example:* chargeback dispute management]
- [*Example:* model training research and feature discovery notebooks]
- [*Example:* authorisation routing and the payment ledger (see Payment Ledger System Design (`system-design/design-payment-ledger`))]
- [Item deferred, with the release that will take it]

## 3. Capacity estimation

<!-- Guidance: Every number must end in a consequence for the design. -->

### 3.1 Assumptions

| Input | Value | Source |
|---|---|---|
| Peak payments per second | *Example:* 5 000 | Volume forecast |
| Peak to average ratio | *Example:* 3 | Daily curve |
| Entity keys per payment (card, device, email, IP, account, merchant, and pairs) | *Example:* 12 | Feature catalogue |
| Aggregates updated per payment | *Example:* 40 | Feature catalogue |
| Features per model input | *Example:* 300, 4 bytes each | Model spec |
| Review band share | *Example:* 0.02% of payments | Threshold policy |
| Analyst minutes per case | *Example:* 2 | Ops data |
| Label maturity window | *Example:* 90 days for most chargebacks | Historical label curves; check network rules |

### 3.2 Throughput

| Quantity | Calculation | Result | Consequence |
|---|---|---|---|
| Online feature reads | 5 000 x 12 keys | 60 000 key reads/s | Batched multi-get to an in-memory store, co-located with scorer |
| Aggregate updates | 5 000 x 40 | 200 000 updates/s | Streaming job partitioned by entity key, not per-request writes from the scorer |
| Average payments/s | 5 000 / 3 | about 1 670/s | |
| Payments per day | 1 670 x 86 400 | about 144 million | |

### 3.3 Storage

| Quantity | Calculation | Result | Consequence |
|---|---|---|---|
| Feature vector per payment | 300 x 4 B | 1.2 KB | |
| Logged vectors per day | 144 million x 1.2 KB | about 173 GB/day | Log every served vector for parity and training; columnar storage |
| Logged vectors per maturity window | 173 GB x 90 | about 15.6 TB | Training sets are large; sample negatives (good payments) for training, keep all positives |

### 3.4 Review capacity

| Quantity | Calculation | Result | Consequence |
|---|---|---|---|
| Cases per day | 144 million x 0.0002 | 28 800 | |
| Analyst hours | 28 800 x 2 min / 60 | 960 h/day | About 120 analysts on 8-hour shifts: the review band width is set by staffing, not by the model |

### 3.5 Summary

| Quantity | Estimate | Design consequence |
|---|---|---|
| Decision budget | 100 ms p99 | In-memory features, one model call, no remote joins in the hot path |
| Feature reads | 60 000/s | Online store with batched reads |
| Aggregate updates | 200 000/s | Dedicated stream processor |
| Label delay | about 90 days | Training lags reality; use early labels and monitor drift without labels |
| Review | 28 800 cases/day | Prioritise by expected loss; cap the band to capacity |

## 4. API design

<!-- Guidance: Define the decision contract: inputs, decision, reasons, timeouts and the fallback the caller must apply. -->

### 4.1 Endpoints

| Method and path | Purpose | Idempotent |
|---|---|---|
| `POST /v1/decisions` | Score a payment and return a decision | Yes, keyed by `payment_id` |
| `POST /v1/decisions/{id}/outcome` | Report authorisation result and 3-D Secure result | Yes |
| `POST /v1/labels` | Ingest chargeback, fraud report or review outcome | Yes, keyed by source reference |
| `GET /v1/decisions/{id}` | Decision, model and rule versions, reasons | Yes (read) |
| `POST /v1/rules/{id}/backtest` | Run a rule against historical data | Yes |

### 4.2 Request and response examples

```http
POST /v1/decisions
Content-Type: application/json
X-Request-Deadline-Ms: 90

{"payment_id": "pay_6f31", "merchant_id": "m_2201", "amount": {"value": 18900, "currency": "USD"}, "card_fingerprint": "cf_a91e", "device_id": "dv_77c0", "ip": "[ip]", "email_hash": "eh_19b2", "channel": "ecommerce"}
```

```http
HTTP/1.1 200 OK
Content-Type: application/json

{"decision": "challenge", "score": 0.62, "reasons": ["card_new_device", "ip_country_mismatch"], "model_version": "gbm-2026-10-01", "rule_set_version": "rs-418", "fallback": false}
```

- If the deadline passes, the caller applies its configured fallback (7.1)
  and records `fallback: true` with the later `outcome` call.
- Reason codes are stable identifiers documented for merchants; they never
  reveal thresholds or rule logic.

## 5. Data model

<!-- Guidance: Separate the online store (latest values, fast) from the offline store (history, point-in-time). Show the access-pattern table. -->

### 5.1 Entities

Entities are the things fraud attaches to: cards, devices, emails, IP
addresses, accounts and merchants. Features are aggregates keyed by one
entity or a pair of entities over a window.

```mermaid
erDiagram
  PAYMENT ||--|| DECISION : "scored by"
  DECISION ||--o{ REASON : has
  PAYMENT }o--|| CARD : uses
  PAYMENT }o--|| DEVICE : from
  PAYMENT }o--|| MERCHANT : at
  PAYMENT ||--o{ LABEL : "labelled by"
  DECISION ||--o| REVIEW_CASE : "may open"
  PAYMENT {
    string payment_id PK
    bigint amount
    string currency
    datetime event_time
  }
  DECISION {
    string payment_id PK
    float score
    string decision
    string model_version
    string rule_set_version
    bool fallback
  }
  LABEL {
    string payment_id PK
    string source PK
    string label
    datetime label_time
  }
  REVIEW_CASE {
    string case_id PK
    string payment_id FK
    float expected_loss
    string status
    string analyst
  }
  CARD {
    string card_fingerprint PK
  }
  DEVICE {
    string device_id PK
  }
  MERCHANT {
    string merchant_id PK
    string risk_profile
  }
  REASON {
    string payment_id PK
    string code PK
  }
```

### 5.2 Storage choices

| Data | Store | Key | Notes |
|---|---|---|---|
| Online aggregates | *Example:* In-memory key-value store with replication | `entity_type:entity_id:feature:window` | Latest value only; TTL equal to longest window |
| Served feature vectors | Columnar store or data lake | `payment_id`, date partition | Exactly what the model saw |
| Offline feature history | Data lake with event-time columns | entity, event time | For point-in-time joins |
| Decisions and labels | Relational or wide-column store | `payment_id` | Joined for training and monitoring |
| Rules | Versioned configuration repository | `rule_id`, version | Change review and backtest required |
| Review cases | Case management database | `case_id` | Queue ordered by expected loss |

### 5.3 Access patterns

| ID | Query | Served by | Partition key rationale |
|---|---|---|---|
| AP-01 | *Example:* Fetch 12 entities' aggregates for one payment | Batched multi-get on online store | Keys spread across nodes; one round trip |
| AP-02 | *Example:* Update card velocity on a new event | Stream processor keyed by `card_fingerprint` | One partition owns each key, so updates are ordered |
| AP-03 | *Example:* Build training set as of each payment's time | Point-in-time join of offline history on entity and event time | Prevents label leakage from future events |
| AP-04 | *Example:* Next case for an analyst | Priority index on `expected_loss` within queue | Highest value first |
| AP-05 | *Example:* Fraud rate by model version and score band | Decisions joined with labels, partitioned by date | Monitoring and threshold tuning |

## 6. High-level design

<!-- Guidance: One diagram with the synchronous decision path and the asynchronous learning loop clearly separated. -->

```mermaid
flowchart LR
  pay[Payment service] -->|decision request| dec[Decision service]
  dec --> ofs[(Online feature store)]
  dec --> model[Model scorer]
  dec --> rules[Rule engine]
  dec -->|decision| pay
  pay --> bus[[Payment event log]]
  bus --> stream[Stream feature jobs]
  stream --> ofs
  stream --> hist[(Offline feature history)]
  dec --> served[(Served vectors and decisions)]
  dec --> queue[Review queue]
  queue --> analysts([Fraud analysts])
  analysts --> labels[Label service]
  cb[[Chargebacks and fraud reports]] --> labels
  labels --> lake[(Labels)]
  lake --> train[Training pipeline]
  hist --> train
  served --> train
  train --> registry[(Model registry)]
  registry --> model
  served --> mon[Monitoring]
  lake --> mon
```

| Component | Responsibility | Scales by |
|---|---|---|
| Decision service | Assemble features, call model and rules, apply thresholds, log | Horizontally, stateless |
| Online feature store | Latest aggregates by entity | Partitions and replicas |
| Stream feature jobs | Sliding-window aggregates from payment events | Partitions of the event log |
| Rule engine | Evaluate versioned rules in process | With decision service |
| Review queue | Prioritised cases, analyst tooling | Case volume |
| Training pipeline | Point-in-time datasets, training, evaluation, registry | Batch compute |

## 7. Detailed design

<!-- Guidance: Two to four deep dives on the hard problems, each with options, choice and reason. -->

### 7.1 Latency budget and fallback

| Step | Budget (p99) |
|---|---|
| Request parsing and request-only features | 5 ms |
| Online feature multi-get | 20 ms |
| Model inference (in process) | 15 ms |
| Rule evaluation | 10 ms |
| Threshold mapping, logging (asynchronous) | 5 ms |
| Network both ways | 15 ms |
| Headroom | 30 ms |
| **Total** | **100 ms** |

**Fallback options:** fail open (allow), fail closed (block), fail to
challenge, or fall back to rules only. **Choice:** rules-only path inside
the service when features time out; caller-side fail open below a small
amount and fail to challenge above it when the service itself is down.
**Reason:** blocking all payments during an outage costs more than a short
window of weaker screening, but large amounts deserve friction.

### 7.2 Deep dive: streaming features and train-serve parity

| Option | Freshness | Cost | Parity risk |
|---|---|---|---|
| Batch aggregates (hourly or nightly) | Hours | Low | Low, but stale features miss attacks in progress |
| Compute on request from raw events | Exact | High read cost at 5 000/s | Medium; queries differ from training code |
| Streaming aggregates into online store | Seconds | Medium | Medium; needs shared feature definitions |
| Streaming plus logging of served vectors | Seconds | Medium plus storage | Low; train on what was served |

**Choice:** streaming aggregates for windows up to 24 hours, batch for
longer windows, and log every served vector. **Reason:** card-testing and
account-takeover attacks unfold in minutes, so velocity features must be
fresh; logging served vectors removes the most common parity bug, where
training recomputes features differently from serving.

- Feature definitions live in one registry and generate both the stream
  job and the offline backfill.
- Offline training joins use event time, never processing time, so a
  payment only sees aggregates of events that happened before it.
- Late events update aggregates but are flagged; the parity job compares a
  sample of served values against recomputed values daily (NFR-07).
- Every feature has an owner and a Data Contract (`ai-data/data-contract`)
  with the upstream producer.

### 7.3 Deep dive: rules plus model

| Option | Strength | Weakness |
|---|---|---|
| Rules only | Transparent, fast to change | Do not rank risk; rot without measurement |
| Model only | Ranks risk across many weak signals | Slow to respond to a brand-new attack; harder to explain |
| Rules as features of the model | One score | Analysts lose fast override |
| Model score plus a rule layer | Model ranks; rules enforce policy and react within minutes | Two systems to govern |

**Choice:** model score plus a rule layer. Rules have three kinds: policy
(always block sanctioned or blocked instruments), overrides (merchant
allow-lists with expiry) and tactical (temporary rules against an active
attack, expiring in *Example* 14 days unless renewed). **Reason:** the
model handles ranking; humans need a lever that works before the next
retrain. Every rule shows its backtest and its live hit and precision
rate, and rules with no measured value are retired.

```mermaid
flowchart TD
  req[Payment] --> pol{Policy rule hit?}
  pol -- yes --> block[Block]
  pol -- no --> score[Model score]
  score --> tac{Tactical or override rule?}
  tac -- override allow --> allow[Allow]
  tac -- tactical block --> block
  tac -- none --> band{Score band for merchant profile}
  band -- low --> allow
  band -- middle --> chal[3-D Secure challenge]
  band -- high and amount over review floor --> review[Review]
  band -- high and amount under review floor --> chal
  band -- very high --> block
```

> **How real systems do it** *(paraphrased from a public write-up; not copied)*
>
> Stripe's guide to machine learning for fraud describes how its Radar
> product turns each payment into a vector of features drawn from the
> payment and from activity seen across many businesses, trains models on
> labelled history, and judges them with precision, recall and score
> distributions rather than a single accuracy number. It treats the choice
> of threshold as a business trade-off between blocking fraud and turning
> away good customers, lets users add their own rules and see how a rule
> would have performed on past payments before turning it on, and uses
> manual review as a way to handle the uncertain middle. It also stresses
> retraining as fraud patterns change. This design mirrors that split:
> model for ranking, rules with backtests for policy, review for the middle
> band.
>
> Source: Stripe, "A primer on machine learning for fraud detection",
> Stripe Guides, last updated 15 December 2021,
> https://stripe.com/guides/primer-on-machine-learning-for-fraud-protection

### 7.4 Manual review queues

- Cases enter only from the review band and only above an amount floor;
  the band is widened or narrowed so the forecast case count matches
  staffed capacity (3.4).
- Priority is expected loss: score times amount, adjusted by merchant tier.
  Cases older than the SLA escalate.
- The payment is held (authorised, not captured, or held in a pending
  state) while under review. If the SLA expires, the default action is set
  per merchant (release or cancel).
- Analysts see the reasons, the entity graph (other payments sharing card,
  device or email) and prior decisions. Their verdict becomes a label with
  source `review`.
- A random sample of decided cases is re-reviewed weekly to measure
  analyst agreement; low agreement points to unclear guidelines, not bad
  analysts.

### 7.5 Deep dive: label delay, feedback loops and selection bias

Labels arrive late and only for some payments:

| Label source | Typical delay *(Example)* | Covers |
|---|---|---|
| Review verdict | Hours | Review band only |
| 3-D Secure failure | Seconds | Challenged payments only |
| Customer fraud report | Days | Allowed payments |
| Chargeback | Weeks to months | Allowed and captured payments |

Problems and choices:

- **Delay:** a model trained today learns from payments at least one
  maturity window old. **Choice:** train on matured data for the main
  model; monitor recent weeks with early labels and with label-free signals
  (score distribution, feature drift, decline rate). **Reason:** waiting
  for full labels is the only unbiased option; early labels catch
  breakage sooner.
- **Selection bias:** blocked payments never get chargebacks, so their
  labels are missing and the next model learns from a filtered world.
  **Options:** ignore (bias grows with each retrain); infer labels for
  blocked payments (circular); allow a small random exploration sample of
  would-be-blocked payments below an amount cap and weight it in training
  and evaluation. **Choice:** exploration sample with a strict loss budget
  approved by the risk owner (*Example:* 0.1% of blocks, amount at most
  USD 50). **Reason:** it is the only source of unbiased estimates of false
  decline rate (NFR-05) and of the block band's true fraud rate.
- **Feedback loops with rules:** a tactical rule that blocks an attack
  removes those payments from future training labels; record which rule
  fired so training can include or exclude them deliberately.
- **Adversarial adaptation:** fraudsters probe thresholds. Do not expose
  scores to merchants at full precision and rotate tactical rules.

### 7.6 Model lifecycle and monitoring

| Stage | Exit criterion *(Example)* |
|---|---|
| Offline evaluation | Better precision at fixed recall in the business band than the champion on the latest matured month |
| Shadow | Scores logged for 7 days without effect; latency and score distribution within limits |
| Canary | 5% of traffic for 7 days; decline and challenge rates within agreed bounds |
| Full rollout | Risk owner approval; champion kept for instant rollback |

Monitoring signals, owners and thresholds belong in Model Monitoring Plan
(`ai-data/model-monitoring-plan`); describe the model in a Model Card
(`ai-data/model-card`). If the operator is a US banking organisation, the
fraud model may fall under supervisory model risk management guidance (SR
letter 26-2, "Revised Guidance on Model Risk Management", 17 April 2026);
confirm scope with model risk management and record validation in Model
Validation Report (`ai-data/model-validation-report`).

## 8. Scalability and performance

<!-- Guidance: Show how the design meets the numbers in section 3 and where it will break first. -->

| Concern | Approach | Limit before redesign |
|---|---|---|
| Feature reads | Multi-get, replicas in each zone, hot-key caching for merchants | *Example:* 200 000 reads/s per cluster |
| Hot entities | Merchant-level aggregates pre-aggregated per second | Merchant event rate |
| Stream updates | Partition by entity key; scale partitions | Partition count of the event log |
| Model inference | In-process model, no network hop | Model size and CPU per request |
| Training data | Sample negatives, keep positives | Lake query cost |

## 9. Reliability, security and operations

<!-- Guidance: Name the failure modes, the threats and the signals operators watch. -->

### 9.1 Failure modes

| ID | Component | Failure | Detection | Blast radius | Mitigation | RTO |
|---|---|---|---|---|---|---|
| FM-01 | Decision service | Down or over budget | Caller timeouts | All payments | Caller fallback (7.1) | *Example:* 5 min |
| FM-02 | Online feature store | Partition lost | Read errors, missing features rate | Payments with keys on that partition | Rules-only path; rebuild from stream replay | *Example:* 15 min |
| FM-03 | Stream jobs | Lag | Event-to-store lag | Stale velocity features | Alert at 30 s; scale consumers | *Example:* 10 min |
| FM-04 | Model | Bad release | Decline rate jump, score shift | Share of traffic on canary or all | Automatic rollback to champion on guardrail breach | *Example:* 5 min |
| FM-05 | Label feed | Chargeback file missing | Feed freshness | Monitoring and next training | Hold retrain; contact provider | *Example:* 2 days |
| FM-06 | Rule change | Over-broad tactical rule | Hit rate spike | Merchants matching rule | Rule hit caps and automatic disable above cap | *Example:* 5 min |
| [FM-nn] | [Component] | [Failure] | [Signal] | [Scope] | [Mitigation] | [RTO] |

### 9.2 Security and privacy

| ID | Threat or duty | Control |
|---|---|---|
| SEC-01 | Threshold probing | Coarse reason codes, no raw score to merchants, rate limits |
| SEC-02 | Insider weakens rules | Rule changes reviewed by a second analyst; audit log |
| SEC-03 | Personal data in features | Hash identifiers; minimise raw fields; retention limits; DPIA (`compliance/dpia`) |
| SEC-04 | Poisoned labels | Label sources authenticated; anomaly checks on label volume per merchant |
| SEC-05 | Automated decision rights | Record logic and human review route where the law requires it; this is a format, not legal advice |
| [SEC-nn] | [Threat] | [Control] |

### 9.3 Observability

| Signal | Metric | Alert threshold |
|---|---|---|
| Latency | Decision p99 | Over 100 ms for 5 min |
| Fallback rate | Share of decisions with fallback | Above 0.5% |
| Score shift | Population stability of score distribution | *Example:* above 0.2 week over week |
| Decline rate | Blocks plus challenges by merchant tier | Change above 30% day over day |
| Review backlog | Cases past SLA | Above 5% |
| Parity | Online versus recomputed feature mismatch | Above 0.1% |

## 10. Trade-offs and alternatives

<!-- Guidance: List options, the cost of each choice, and what changes at 10x and 0.1x. -->

| Decision | Options | Chosen | Why | What we give up |
|---|---|---|---|---|
| Feature freshness | Batch; streaming | Streaming up to 24 h windows | Attacks unfold in minutes | Stream infrastructure to run |
| Decision logic | Rules; model; both | Model plus rule layer | Ranking plus fast reaction | Two governance processes |
| Outage behaviour | Open; closed; challenge | Amount-based fallback | Keeps sales flowing | Short window of weaker screening |
| Blocked-traffic labels | Ignore; infer; explore | Exploration sample | Unbiased estimates | Small, capped fraud loss |
| Model family | Deep model; gradient-boosted trees | Gradient-boosted trees | Strong on tabular features, fast inference | Less benefit from raw sequences |
| **At 10x (50 000 payments/s)** | Bigger store; regional cells | Regional cells, each with its own feature store and scorer; global features by asynchronous merge | One cluster's read rate and cross-region latency | Global velocity features become seconds stale |
| **At 0.1x (500 payments/s)** | Full platform; managed service plus rules | Rules plus a vendor or managed score, batch features hourly | Too little data to train well in-house | Less control and explainability |
| [Decision] | [Options] | [Chosen] | [Why] | [Cost] |

## 11. Delivery plan and risks

<!-- Guidance: How the design reaches production safely, and what could stop it. -->

### 11.1 Milestones

| Milestone | Scope | Exit criterion | Target date |
|---|---|---|---|
| M1 | Decision service with current rules, served-vector logging | Latency within budget at peak load | YYYY-MM-DD |
| M2 | Streaming features and online store | Parity above 99.9% for 14 days | YYYY-MM-DD |
| M3 | Model in shadow, then canary | Canary guardrails met for 7 days | YYYY-MM-DD |
| M4 | Review queue by expected loss | SLA met at forecast volume | YYYY-MM-DD |
| M5 | Exploration sample and bias-corrected evaluation | Risk owner approval of loss budget | YYYY-MM-DD |

### 11.2 Risks

| ID | Risk | Likelihood | Impact | Mitigation | Owner |
|---|---|---|---|---|---|
| R-01 | *Example:* Label definitions differ between chargeback reason codes and our "fraud" label | High | Medium | Label taxonomy agreed with ops; mapping table versioned | [@name] |
| R-02 | *Example:* Merchants reject higher challenge rate | Medium | Medium | Per-merchant thresholds; reporting on recovered sales | [@name] |
| R-03 | *Example:* Exploration sample seen as deliberate loss by auditors | Low | Medium | Documented loss budget, approval, and purpose | [@name] |
| [R-nn] | [Risk] | [L / M / H] | [L / M / H] | [Mitigation] | [@name] |

## 12. Appendices

<!-- Guidance: Supporting detail, traceability, glossary and references. -->

### 12.1 Traceability

| Requirement | Design element | Verification |
|---|---|---|
| NFR-01 | 7.1 budget | Load test at peak |
| NFR-02 | 7.1 fallback | Fault injection on decision service |
| NFR-03 | 7.2 streaming | Lag dashboard |
| NFR-05 | 7.5 exploration | Monthly estimate with confidence interval |
| NFR-07 | 7.2 served vectors, parity job | Daily parity report |
| [ID] | [Element] | [Test] |

### 12.2 Open questions

| ID | Question | Owner | Due |
|---|---|---|---|
| OQ-01 | *Example:* Do we share network-level signals across merchants under our contracts? | [@name] | YYYY-MM-DD |
| OQ-02 | *Example:* Which label wins when review and chargeback disagree? | [@name] | YYYY-MM-DD |
| [OQ-nn] | [Question] | [@name] | YYYY-MM-DD |

### 12.3 Glossary

| Term | Meaning |
|---|---|
| Velocity feature | Count or sum of events for an entity over a recent window |
| Point-in-time join | Joining features as they were at the moment of each event |
| Label maturity | Time after which most labels for a payment have arrived |
| Selection bias | Distortion caused by learning only from payments the system let through |
| Exploration sample | Small random set of payments allowed despite a block decision, to measure truth |
| Challenge | Step-up authentication such as 3-D Secure |
| Champion and challenger | Current production model and its candidate replacement |

### 12.4 References

| Reference | Use |
|---|---|
| Stripe, "A primer on machine learning for fraud detection", Stripe Guides (last updated 15 December 2021) | Rules, features, thresholds, review (7.3); paraphrased |
| Board of Governors of the Federal Reserve System, SR 26-2 "Revised Guidance on Model Risk Management", 17 April 2026, https://www.federalreserve.gov/supervisionreg/srletters/SR2602.htm | Model governance for banking organisations (7.6) |
| Model Monitoring Plan (`ai-data/model-monitoring-plan`) | Monitoring signals and owners |
| ML System Design Document (`ai-data/ml-system-design`) | General ML system structure |
| Data Contract (`ai-data/data-contract`) | Feature and label feed contracts |
| Payment Ledger System Design (`system-design/design-payment-ledger`) | Payment states around the decision |
| Digital Wallet System Design (`system-design/design-digital-wallet`) | Risk hooks for wallet transfers |

## Standard and licence note

Based on common industry practice for payment fraud scoring and on the
public references in 12.4. Company guides were used as inspiration only;
the text, diagrams and numbers are original and illustrative. SR letter
26-2 is US supervisory guidance aimed at banking organisations; check its
current text and scope before relying on it. Card-network and data
protection rules vary and change; this is an engineering format, not legal
advice. Template licence: MIT.