# System Design: [Digital wallet name]

> Reference design for a stored-value digital wallet: customer balances,
> peer-to-peer and merchant transfers, top-ups and withdrawals. It covers a
> double-entry ledger as the source of truth, cross-shard transfers with a
> saga versus two-phase commit versus event sourcing, idempotency keys,
> hot-account contention, reconciliation with banks and payment providers,
> limits and KYC (know your customer) hooks, and the operational controls a
> regulated wallet needs. It follows the structure of the System Design
> Document template and is pre-filled with a worked design to adapt. Replace
> every `[bracketed]` value and every row marked *Example*; values marked
> *Example* are assumptions, not benchmarks. Regulatory notes describe a
> documentation format only. This is a documentation format, not legal advice.

<!--
Guidance: How to use this template
- Write the money invariants in 2.2 first: "the sum of all ledger entries
  is zero", "no customer balance goes below zero unless a credit line
  allows it", "every external movement of funds is matched in
  reconciliation". Every storage and transfer decision exists to protect
  them.
- Decide early whether the wallet holds funds itself (stored value under
  an e-money or money transmission licence) or only fronts a partner bank.
  The ledger design is the same; the regulatory duties and the
  reconciliation counterparties are not.
- Keep the payment ledger and the wallet ledger apart only if they are
  owned by different teams. Payment Ledger System Design
  (`system-design/design-payment-ledger`) covers card acquiring; this
  template covers internal balances and transfers between them.
- Treat every number here as an assumption to replace with your own
  traffic forecast.
-->

## Document control

<!-- Guidance: A design is a proposal until it is approved; record who decided and when, and which compliance owner signed off the limits and KYC hooks. -->

| Field | Value |
|---|---|
| Document ID | [SDD-WAL-001] |
| System / feature | [Stored-value wallet and transfers] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Owner | [@owner] |
| Author(s) | [@author] |
| Approvers / reviewers | [@reviewer-1], [@reviewer-2] |
| Last updated | YYYY-MM-DD |
| Classification | [Confidential] |
| Licence or regulatory basis | [E-money / money transmission / partner-bank programme; jurisdiction] |
| Related | [PRD], [Payment ledger design], [Fraud design], [Threat model], [DPIA] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Tech lead | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Staff / principal engineer | [@name] | [Decision] | YYYY-MM-DD |
| Finance and treasury | [@name] | [Decision] | YYYY-MM-DD |
| Compliance (AML and KYC) | [@name] | [Decision] | YYYY-MM-DD |
| Security | [@name] | [Decision] | YYYY-MM-DD |
| SRE / operations | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Note what changed in the design, not the wording. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft for design review |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Replaced two-phase commit with a saga through an in-flight account |
| 0.3 | YYYY-MM-DD | [@name] | *Example:* Added sub-balances for merchant hot accounts |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved |

## Contents

<!-- Guidance: Keep aligned with the headings. -->

1. [Overview](#1-overview)
2. [Requirements](#2-requirements)
3. [Capacity estimation](#3-capacity-estimation)
4. [API design](#4-api-design)
5. [Data model](#5-data-model)
6. [High-level design](#6-high-level-design)
7. [Detailed design](#7-detailed-design)
8. [Scalability and performance](#8-scalability-and-performance)
9. [Reliability, security and operations](#9-reliability-security-and-operations)
10. [Trade-offs and alternatives](#10-trade-offs-and-alternatives)
11. [Delivery plan and risks](#11-delivery-plan-and-risks)
12. [Appendices](#12-appendices)

## 1. Overview

<!-- Guidance: One page that a busy reviewer can read alone: problem, proposed solution in a paragraph, the decisions that need their attention, and the short path for an interview. -->

### 1.1 Problem statement

[*Example:* Our app lets 50 million customers hold a balance, pay each other
and pay 300 000 merchants. Balances live in a single relational table that
is updated in place. Last quarter a retry storm after a database failover
applied 1 900 transfers twice, and finance needed 11 days to find them
because there was no ledger to replay. The largest merchant receives so many
credits during lunch hours that its balance row became a lock queue and
transfer latency reached 8 seconds. Compliance cannot show which limits
applied to a given transfer at the time it ran.]

### 1.2 Proposed solution

[*Example:* An append-only double-entry ledger becomes the source of truth.
A balance is a derived, cached projection of ledger entries per account,
updated in the same transaction as the entries. Accounts are sharded by
account ID. A transfer between two shards is a saga: debit the sender into
an in-flight (suspense) account on the sender shard, then credit the
receiver from the in-flight account on the receiver shard, with an outbox
carrying the second step. Every request carries an idempotency key that is
stored with the result. Merchant hot accounts are split into sub-balances.
A limits and risk service runs before the debit and records the rule
version it applied. Reconciliation runs continuously against internal
invariants and daily against bank and provider statements.]

### 1.3 Decisions requested from reviewers

| ID | Question | Recommended option | Section |
|---|---|---|---|
| Q-01 | *Example:* Cross-shard transfer mechanism? | Saga through an in-flight account, not two-phase commit | 7.2, 10 |
| Q-02 | *Example:* Is the balance stored or computed? | Stored projection updated atomically with entries; recomputable from entries | 7.1 |
| Q-03 | *Example:* Hot merchant accounts? | Sub-balances (N buckets) summed on read, swept hourly | 7.4 |
| Q-04 | *Example:* Idempotency key retention? | 7 days for transfers, keyed per client and endpoint | 7.3 |
| Q-05 | *Example:* Limit checks synchronous? | Yes for hard limits; velocity scoring asynchronous with hold-for-review | 7.6 |
| [Q-nn] | [Question] | [Option] | [Section] |

### 1.4 45-minute interview path

<!-- Guidance: The same document serves a real review and an interview. In an interview, spend time where the hard problem is: here it is correctness of money movement, not throughput. -->

| Minute | Cover | Notes |
|---|---|---|
| 0 to 5 | Clarify scope: P2P only or merchants too, top-up and withdrawal, currencies | State the invariants in one sentence each (2.2) |
| 5 to 10 | Core requirements and below-the-line list | Push FX, cards and credit lines below the line |
| 10 to 15 | Estimates with consequences (3.5) | Show why one database is not enough and why hot accounts exist |
| 15 to 20 | API with idempotency key; ledger data model with the access-pattern table | Write the entry and account rows on the board |
| 20 to 30 | High-level diagram, then the main deep dive: cross-shard transfer (7.2) | Compare 2PC, saga and event sourcing; pick one and defend it |
| 30 to 40 | Second deep dive: idempotency (7.3) or hot accounts (7.4) | Choose hot accounts if the interviewer pushes on scale |
| 40 to 45 | Reconciliation, failure modes, what changes at 10x | End on how you would know money is wrong |

Pick 7.2 as the default deep dive. Mention 7.5 (reconciliation) even if there
is no time to detail it: a wallet without reconciliation is not credible.

## 2. Requirements

<!-- Guidance: Split the core from what is explicitly below the line, and attach a number to every quality. -->

### 2.1 Core functional requirements

| ID | Requirement | Priority |
|---|---|---|
| FR-01 | *Example:* Show a customer's available and pending balance per currency | Must |
| FR-02 | *Example:* Transfer funds between two wallets (P2P) with a note | Must |
| FR-03 | *Example:* Pay a merchant wallet from a customer wallet | Must |
| FR-04 | *Example:* Top up from a linked bank account or card through a provider | Must |
| FR-05 | *Example:* Withdraw to a linked bank account | Must |
| FR-06 | *Example:* Show a statement of transactions with running balance | Must |
| FR-07 | *Example:* Apply per-tier limits (single transfer, daily, monthly, balance cap) tied to KYC level | Must |
| FR-08 | *Example:* Reverse a completed transfer through a new compensating transfer, never by editing history | Must |
| FR-09 | *Example:* Hold funds for review when risk scoring requests it | Should |
| [FR-nn] | [Requirement] | [Must / Should / Could] |

### 2.2 Non-functional requirements and invariants

| ID | Quality | Target | Measured as |
|---|---|---|---|
| NFR-01 | Ledger balance | *Example:* Sum of all entries per currency equals zero at every commit | Continuous invariant check |
| NFR-02 | No negative balance | *Example:* No account below its allowed floor (0 unless a credit line exists) | Constraint in the debit predicate; nightly audit |
| NFR-03 | Exactly-once effect | *Example:* Zero transfers applied twice for the same idempotency key | Duplicate-key audit |
| NFR-04 | Transfer latency | *Example:* p99 at or below 400 ms for same-shard, 1.5 s for cross-shard completion | Service timing |
| NFR-05 | Availability | *Example:* 99.95% monthly for transfers; balance reads 99.99% | SLO |
| NFR-06 | Durability | *Example:* RPO 0 for committed ledger entries | Synchronous replication |
| NFR-07 | Reconciliation | *Example:* Every external movement matched within 1 business day; breaks investigated within 2 | Reconciliation dashboard |
| NFR-08 | Auditability | *Example:* For each transfer, the limit set, rule version and KYC tier in force are retrievable for [retention period] | Audit query |
| [NFR-nn] | [Quality] | [Target] | [Measurement] |

### 2.3 Below the line (out of scope)

- [*Example:* foreign exchange between wallet currencies (separate FX design)]
- [*Example:* card issuing linked to the wallet]
- [*Example:* credit lines and overdrafts]
- [*Example:* interest or yield on balances]
- [*Example:* the card acquiring ledger (see Payment Ledger System Design (`system-design/design-payment-ledger`))]
- [*Example:* fraud model internals (see Payment Fraud Detection System Design (`system-design/design-fraud-detection`))]
- [Item deferred, with the release that will take it]

## 3. Capacity estimation

<!-- Guidance: Every number must end in a consequence for the design. If a number changes nothing, drop it. -->

### 3.1 Assumptions

| Input | Value | Source |
|---|---|---|
| Active wallets | *Example:* 50 million | Product forecast |
| Transfers per active wallet per day | *Example:* 4 | Current usage |
| Peak to average ratio | *Example:* 5 | Lunch and payday peaks |
| Ledger entries per transfer | *Example:* 4 (two legs plus fee and in-flight legs on average) | 7.1 and 7.2 |
| Entry row size including indexes | *Example:* 250 bytes | Schema in 5.1 |
| Share of transfers crossing shards | *Example:* about 0.98 with 64 shards (63/64) | Random account placement |
| Credits per second to the largest merchant at peak | *Example:* 2 000 | Merchant analytics |

### 3.2 Throughput

| Quantity | Calculation | Result | Consequence |
|---|---|---|---|
| Transfers per day | 50 million x 4 | 200 million | Above what one primary handles comfortably for writes with indexes |
| Average transfers per second | 200 million / 86 400 | about 2 300/s | Baseline for normal provisioning |
| Peak transfers per second | 2 300 x 5 | about 11 600/s, round to 12 000/s | Shard the ledger from day one |
| Peak entry writes | 12 000 x 4 | 48 000 entries/s | With *Example* 3 000 entries/s per shard at safe utilisation, at least 16 shards; choose 64 for growth |

### 3.3 Storage

| Quantity | Calculation | Result | Consequence |
|---|---|---|---|
| Entries per day | 200 million x 4 | 800 million | Partition entry tables by account and by month |
| Bytes per day | 800 million x 250 B | 200 GB/day | Hot storage for 13 months, then archive |
| Bytes per year | 200 GB x 365 | about 73 TB/year | Archive tier with retrieval for audits; statements read from monthly snapshots |

### 3.4 Contention

The largest merchant at 2 000 credits/s would serialise on one balance row.
If a row update with commit takes *Example* 2 ms under lock, one row sustains
at most about 500 updates/s, so the merchant needs at least 4 sub-balances;
choose 16 to absorb bursts (7.4).

### 3.5 Summary

| Quantity | Estimate | Design consequence |
|---|---|---|
| Peak transfers | 12 000/s | Sharded ledger, 64 shards |
| Cross-shard share | about 98% | Cross-shard transfer is the normal path, not an edge case; design it first (7.2) |
| Ledger growth | 73 TB/year | Partition by month; monthly balance snapshots for statements |
| Hot merchant | 2 000 credits/s | Sub-balances, not one row |

## 4. API design

<!-- Guidance: Define the contract first. For money, idempotency semantics and error types matter more than paths. -->

### 4.1 Endpoints

| Method and path | Purpose | Idempotent |
|---|---|---|
| `GET /v1/wallets/{id}/balance` | Available and pending balance per currency | Yes (read) |
| `GET /v1/wallets/{id}/transactions?cursor=` | Statement with running balance | Yes (read) |
| `POST /v1/transfers` | Transfer between wallets | Yes, with `Idempotency-Key` |
| `GET /v1/transfers/{id}` | Transfer status (`pending`, `completed`, `held`, `failed`, `reversed`) | Yes (read) |
| `POST /v1/transfers/{id}/reverse` | Compensating transfer, by an operator or under policy | Yes, with `Idempotency-Key` |
| `POST /v1/topups` | Pull funds from a linked instrument through a provider | Yes, with `Idempotency-Key` |
| `POST /v1/withdrawals` | Push funds to a linked bank account | Yes, with `Idempotency-Key` |

### 4.2 Request and response examples

```http
POST /v1/transfers
Idempotency-Key: 4b7e0a3c-1f2d-4c8e-9a51-6d2b8e0f7c33
Authorization: Bearer [token]
Content-Type: application/json

{"from_wallet": "wal_81aa", "to_wallet": "wal_19fe", "amount": {"value": 2500, "currency": "EUR"}, "note": "Dinner"}
```

```http
HTTP/1.1 202 Accepted
Content-Type: application/json

{"transfer_id": "trf_5c02", "status": "pending", "amount": {"value": 2500, "currency": "EUR"}, "created_at": "YYYY-MM-DDThh:mm:ssZ"}
```

Amounts are integers in the minor unit of the currency. The API never
accepts floating-point amounts. A same-shard transfer may return `201` with
`completed`; a cross-shard transfer returns `202` and completes
asynchronously (7.2).

### 4.3 Idempotency and errors

- The client generates a unique key per logical operation and reuses it on
  every retry. The server stores the key, a hash of the request body, the
  owning client and the final response for 7 days.
- The same key with the same body returns the stored response. The same key
  with a different body returns 422. A request that arrives while the first
  is still running returns 409 `request-in-progress`.
- Errors use RFC 9457 problem details with a stable `type` URI.

| Status | Problem type | Meaning |
|---|---|---|
| 402 | `insufficient-funds` | Available balance below the amount |
| 403 | `limit-exceeded` | A tier limit blocks the transfer; body names the limit, not the threshold |
| 403 | `kyc-required` | The action needs a higher verification tier |
| 409 | `request-in-progress` | Same key is being processed |
| 422 | `idempotency-key-reused` | Same key, different request |
| 423 | `wallet-frozen` | Wallet is frozen by compliance or the customer |

## 5. Data model

<!-- Guidance: Choose storage by access pattern. Show entities, keys, partitioning, the access-pattern table and the invariant each structure enforces. -->

### 5.1 Entities

```mermaid
erDiagram
  ACCOUNT ||--o{ ENTRY : has
  TRANSFER ||--|{ ENTRY : "posts"
  ACCOUNT ||--o{ SUB_BALANCE : "split into"
  WALLET ||--|{ ACCOUNT : owns
  TRANSFER ||--o| IDEMPOTENCY : "keyed by"
  WALLET {
    string wallet_id PK
    string customer_id
    string kyc_tier
    string status
  }
  ACCOUNT {
    string account_id PK
    string wallet_id FK
    string type
    string currency
    bigint balance
    bigint version
  }
  SUB_BALANCE {
    string account_id PK
    int bucket PK
    bigint balance
  }
  TRANSFER {
    string transfer_id PK
    string status
    string limit_rule_version
    datetime created_at
  }
  ENTRY {
    string entry_id PK
    string transfer_id FK
    string account_id FK
    bigint amount
    string direction
    datetime posted_at
  }
  IDEMPOTENCY {
    string client_id PK
    string key PK
    string request_hash
    string response
    datetime expires_at
  }
```

Account types: `customer`, `merchant`, `in_flight` (one per shard and
currency), `fees`, `bank_settlement` (mirrors the safeguarding or partner
bank account), `provider_clearing` (one per top-up and withdrawal provider).

### 5.2 Storage choices and invariants

| Data | Store | Partition key | Invariant enforced by |
|---|---|---|---|
| Accounts, entries, sub-balances | *Example:* Sharded relational database, 64 shards, synchronous replica per shard | `account_id` | Debit predicate `balance - amount >= floor`; entries and balance in one transaction |
| Transfers | Same shard as the sender account | `from_account_id` | State machine in 7.2 |
| Idempotency records | Same shard as the sender account | `client_id`, `key` | Unique constraint, written in the debit transaction |
| Outbox | Same shard as the writer | none | Written in the same transaction as entries |
| Statements | Read replica plus monthly snapshot table | `account_id`, month | Snapshot plus entries since snapshot equals balance |
| Limits counters | In-memory store with durable backing | `wallet_id`, window | Rebuilt from entries if lost |

### 5.3 Access patterns

| ID | Query | Served by | Partition key rationale |
|---|---|---|---|
| AP-01 | *Example:* Current balance of a wallet | `account` row by `account_id` (plus sub-balances for merchants) | One-shard read; hottest read path |
| AP-02 | *Example:* Debit sender and record entries | `account` row lock or conditional update, insert `entry` | Sender account and its entries co-located so the debit is one local transaction |
| AP-03 | *Example:* Statement page for a month | `entry` index on `(account_id, posted_at)` | Entries for one account live on one shard |
| AP-04 | *Example:* Look up a transfer by ID | `transfer` primary key; transfer ID embeds shard number | Avoids a global index |
| AP-05 | *Example:* Check idempotency key | Unique index `(client_id, key)` on the sender shard | Same transaction as the debit |
| AP-06 | *Example:* Daily sum of entries per settlement account | `entry` index on `(account_id, posted_at)` for settlement accounts | Few accounts; feeds reconciliation |
| AP-07 | *Example:* Daily spend for limits | Limits counter by `wallet_id` and day | Fast path; ledger is the fallback |

## 6. High-level design

<!-- Guidance: One diagram that shows every component on the request paths and where state lives. -->

```mermaid
flowchart LR
  app([Customer or merchant app]) --> gw[API gateway and auth]
  gw --> tapi[Transfer API]
  tapi --> lim[Limits and KYC service]
  tapi --> risk[Risk scoring]
  tapi --> led[Ledger service]
  led --> shards[(Ledger shards 0..63)]
  shards --> outbox[[Outbox relay]]
  outbox --> bus[[Event log]]
  bus --> led
  bus --> notif[Notifications]
  bus --> recon[Reconciliation]
  tapi --> ext[External funds service]
  ext --> prov[[Top-up and payout providers]]
  ext --> bank[[Safeguarding or partner bank]]
  recon --> stmt[(Bank and provider statements)]
  recon --> ops[Operations console]
```

| Component | Responsibility | Scales by |
|---|---|---|
| Transfer API | Validate, check idempotency, call limits and risk, start transfer | Horizontally |
| Limits and KYC service | Evaluate tier limits, record rule version applied | Horizontally, counters partitioned by wallet |
| Ledger service | Post balanced entries within one shard; run saga steps | Shards |
| Outbox relay | Publish committed events exactly once per event ID | One relay per shard |
| External funds service | Top-ups and withdrawals through providers; owns clearing accounts | Horizontally; provider limits apply |
| Reconciliation | Internal invariants continuously, external statements daily | Batch and streaming jobs |

## 7. Detailed design

<!-- Guidance: Go deep on two to four hard problems. For each, list options, the choice and the reason. -->

### 7.1 Double-entry ledger and balances

Every movement of value posts at least two entries whose amounts sum to zero
in one currency. A transfer of EUR 25.00 from Alice to Bob on the same shard:

| Entry | Account | Debit | Credit |
|---|---|---|---|
| E1 *(Example)* | customer:alice | 2 500 | |
| E2 *(Example)* | customer:bob | | 2 500 |

Rules:

- Entries are immutable. Corrections are new transfers that reference the
  original (FR-08).
- The balance column is a projection maintained in the same transaction as
  the entries. A nightly job recomputes balances from entries and alerts on
  any difference (NFR-01).
- A pending balance (holds for review, unsettled top-ups) is a separate
  account type, not a flag on entries, so available balance is always one
  read.

**Options for the balance:** compute on read from entries (always correct,
slow for long histories); store a projection updated with entries (fast,
needs a recompute check); store only snapshots plus recent entries (bounded
read cost). **Choice:** stored projection plus monthly snapshots. **Reason:**
AP-01 is the hottest path and must be one row read, while snapshots keep
statements and audits cheap.

### 7.2 Deep dive: cross-shard transfers

About 98 percent of transfers cross shards (3.5). The sender debit and the
receiver credit cannot share a local transaction.

| Option | How it works | Strength | Weakness |
|---|---|---|---|
| Two-phase commit (2PC) | Coordinator prepares both shards, then commits | Atomic across shards | Blocking on coordinator failure; locks held across network round trips; poor at 12 000/s |
| Try-confirm-cancel (TC/C) | Reserve on sender, reserve on receiver, then confirm both or cancel both | No distributed lock; explicit compensation | Two phases of application logic per participant; more states |
| Saga through an in-flight account | Local transaction 1 debits sender and credits in-flight; outbox event triggers local transaction 2 that debits in-flight and credits receiver | Each step is a local ACID transaction; ledger stays balanced at every instant | Money is visibly in flight for a short time; needs compensation if step 2 cannot apply |
| Event sourcing with deterministic replay | Transfer commands appended to a partitioned log; state machines per partition apply them in order; snapshots for speed | Full history, replayable, high throughput | Harder queries, harder schema evolution, a new operating model for the team |

**Choice:** saga through an in-flight account. **Reason:** it keeps the
ledger balanced at every commit (the in-flight account carries the value),
uses only local transactions, and fits the team's relational operating
model. Event sourcing remains the path if throughput grows beyond what
sharding can absorb (10).

```mermaid
sequenceDiagram
  participant API as Transfer API
  participant S1 as Sender shard
  participant OB as Outbox and event log
  participant S2 as Receiver shard
  API->>S1: T1: check key, debit sender, credit in-flight, write outbox
  S1-->>API: committed (status pending)
  S1->>OB: transfer.debited (event ID = transfer ID + step)
  OB->>S2: deliver
  S2->>S2: T2: dedupe event, debit in-flight, credit receiver
  alt receiver can accept
    S2->>OB: transfer.credited
    OB->>S1: mark transfer completed
  else receiver frozen or closed
    S2->>OB: transfer.rejected
    OB->>S1: T3: debit in-flight, credit sender (compensation)
  end
```

- Step T2 deduplicates on event ID, so redelivery is harmless.
- The in-flight account per shard must net to zero once all sagas settle; a
  non-zero in-flight balance older than *Example* 5 minutes is an alert.
- The sender sees the debit immediately; the receiver sees the credit when
  T2 commits. The UI shows "sent" after T1 and "received" after T2.

### 7.3 Deep dive: idempotency

| Option | Strength | Weakness |
|---|---|---|
| Client-generated key stored with response | Simple, standard, survives client retries | Requires storage and a retention policy |
| Natural key (client reference) unique per account | No extra table | Couples API to client data shape |
| Deduplicate in a cache only | Fast | Cache loss means double spend |

**Choice:** client-generated key stored in the sender shard in the same
transaction as T1. **Reason:** the key and the debit commit or fail
together, so a retry after a timeout finds either nothing (safe to apply) or
the stored response (return it). Internal saga steps reuse
`transfer_id:step` as their key, and calls to external providers pass a
derived key so providers also deduplicate.

> **How real systems do it** *(paraphrased from a public write-up; not copied)*
>
> Stripe's engineering post on idempotency describes the client side of the
> contract: a client creates a unique key for each operation it wants to
> happen once, attaches it to every attempt, and on a network failure
> retries with the same key, so the server can answer with the result of
> the first attempt instead of acting again. The post also recommends
> retrying with exponential backoff plus random jitter so that many clients
> recovering from the same outage do not hit the server in lockstep. This
> design adopts both ideas: the wallet SDK owns key generation and the
> retry schedule, and the server treats the key as part of the transaction.
>
> Source: Brandur Leach, "Designing robust and predictable APIs with
> idempotency", Stripe blog, 22 February 2017,
> https://stripe.com/blog/idempotency

### 7.4 Deep dive: hot accounts

| Option | How it works | Use here |
|---|---|---|
| One row, row lock | Every credit locks the merchant balance | Fails above about 500 updates/s (3.4) |
| Sub-balances | Merchant account split into N buckets; a credit picks a bucket by hash of transfer ID; balance read sums buckets | Chosen for merchants above a threshold |
| Batched credits | Credits accumulate in a queue and post every second as one entry pair per batch | Use for payout-only merchants that do not read balance in real time |
| Credit-only append | No balance update on credit, only entries; balance computed by periodic roll-up | Fallback for extreme bursts |

**Choice:** sub-balances with *Example* 16 buckets for accounts flagged hot.
**Reason:** credits never need to lock the whole account, debits (payouts)
are rare and can lock all buckets briefly, and the ledger remains exact.
An hourly sweep moves bucket balances into bucket 0 to keep reads cheap.

### 7.5 Reconciliation

| Level | Compares | Frequency | Break action |
|---|---|---|---|
| L1 internal | Sum of entries per currency is zero; projections equal recomputed balances | Continuous and nightly | Page on any non-zero |
| L2 saga | In-flight accounts net to zero after settlement window | Every 5 minutes | Investigate stuck sagas |
| L3 provider | Provider clearing account entries against provider settlement files | Daily | Match by provider reference; open a break ticket |
| L4 bank | Sum of customer balances against safeguarding or partner bank balance | Daily | Treasury review; regulatory report if required |

Every break has an owner, an age and a resolution code. Resolution is always
a new adjusting transfer approved by two people, never an edit.

### 7.6 Limits and KYC hooks

- Each wallet has a KYC tier. Each tier maps to a limit set (single
  transfer, daily, monthly, maximum balance). Limit sets are versioned
  configuration with an approver.
- The limits service is called synchronously before T1 with the wallet,
  amount and counterparty. It returns `allow`, `deny(limit_id)` or
  `step_up(required_tier)`. The transfer records the limit set version
  (NFR-08).
- Risk scoring runs in parallel; a `review` verdict sends funds to a
  pending account instead of the receiver and opens a case.
- Sanctions screening and suspicious activity reporting are owned by
  compliance systems. The wallet emits events and accepts freeze orders
  (status `frozen`, error 423) through an authenticated internal API.

### 7.7 Regulatory notes (format only)

Record, for your jurisdiction, in your own words and with links to the
current consolidated texts: the licence or programme under which balances
are held; the safeguarding method for customer funds; KYC tiers and the
evidence each needs; record retention periods; complaint and dispute
handling duties; and the reports the reconciliation in 7.5 must feed. This
section is a placeholder for that analysis, not the analysis itself.

| Obligation area | Source (jurisdiction, article or section) | Design element | Owner |
|---|---|---|---|
| [Safeguarding of funds] | [Law, article] | 7.5 L4 | [@name] |
| [Customer due diligence tiers] | [Law, article] | 7.6 | [@name] |
| [Record retention] | [Law, article] | 5.2, 9.3 | [@name] |

## 8. Scalability and performance

<!-- Guidance: Show how the design meets the numbers in section 3 and where it will break first. -->

| Concern | Approach | Limit before redesign |
|---|---|---|
| Ledger write rate | 64 shards; add shards by splitting ranges of `account_id` | *Example:* 3 000 entries/s per shard at 60% utilisation |
| Cross-shard latency | Outbox relay polling at 50 ms or log-based change capture | Relay lag p99 under 500 ms |
| Hot accounts | Sub-balances, sweep | Bucket count per account |
| Statement reads | Read replicas, monthly snapshots | Replica lag under 1 s |
| Limits checks | In-memory counters, rebuilt from ledger | Counter store memory |

Latency budget for a same-shard transfer (p99 target 400 ms):

| Step | Budget |
|---|---|
| Auth and validation | 20 ms |
| Limits service (40 ms) and risk scoring (60 ms) in parallel | 60 ms |
| T1 transaction with idempotency and outbox | 60 ms |
| Network and response | 40 ms |
| Headroom for retries | 220 ms |

## 9. Reliability, security and operations

<!-- Guidance: Name the failure modes, the threats and the signals operators watch. Each mitigation should be testable. -->

### 9.1 Failure modes

| ID | Component | Failure | Detection | Blast radius | Mitigation | RTO |
|---|---|---|---|---|---|---|
| FM-01 | Ledger shard primary | Crash or failover | Health check, replication alarms | Wallets on one shard (1/64) | Synchronous replica promotion; idempotency keys absorb client retries | *Example:* 2 min |
| FM-02 | Outbox relay | Stops publishing | Outbox age | Cross-shard transfers from one shard stay pending | Restart relay; T2 dedupes on replay | *Example:* 5 min |
| FM-03 | Limits service | Unavailable | Error rate | All transfers | Fail closed for transfers above *Example* EUR 100; fail open with post-check below | *Example:* 5 min |
| FM-04 | Provider | Timeout on top-up | Step timeout | Top-ups via that provider | Query by idempotency key before retry; second provider | *Example:* 15 min |
| FM-05 | Reconciliation job | Fails or lags | Job heartbeat | Detection only, not movement | Rerun; freeze withdrawals if L4 is over 2 days stale | *Example:* 4 h |
| FM-06 | Clock skew | Entries posted with wrong time | Skew monitor | Statements and limits windows | Server time from one source; windows on posting sequence | *Example:* 30 min |
| [FM-nn] | [Component] | [Failure] | [Signal] | [Scope] | [Mitigation] | [RTO] |

Score these in a Software FMEA (`ops/fmea-software`) and rehearse FM-01 and
FM-02 with a Chaos Experiment Plan (`ops/chaos-experiment`).

### 9.2 Security and abuse

| ID | Threat | Control |
|---|---|---|
| SEC-01 | Account takeover drains a wallet | Step-up authentication for new device, new payee and large amounts; cooling-off on new payees |
| SEC-02 | Insider edits a balance | No update path for balances outside the ledger service; adjustments need two approvers; database access audited |
| SEC-03 | Replay of a signed transfer request | Idempotency key plus request timestamp window |
| SEC-04 | Money mule networks | Risk scoring on graph features; limits by tier; case management |
| SEC-05 | Enumeration of wallet IDs | Opaque IDs; payee lookup rate-limited and requires contact match |
| [SEC-nn] | [Threat] | [Control] |

Model the rest in a Threat Model (`security/threat-model`).

### 9.3 Observability

| Signal | Metric | Alert threshold |
|---|---|---|
| Ledger balance | Sum of entries per currency | Any non-zero |
| In-flight age | Oldest unsettled in-flight entry | Over 5 min |
| Duplicate keys | Same key, different body (422) rate | Doubling vs baseline |
| Reconciliation breaks | Open breaks by age | Any older than 2 business days |
| Transfer latency | p99 by same-shard and cross-shard | Over NFR-04 for 10 min |

### 9.4 Cost estimate

| Item | Unit cost | Quantity | Monthly cost |
|---|---|---|---|
| Ledger shards with replicas | [USD per node-month] | *Example:* 128 nodes | [USD] |
| Event log | [USD per broker-month] | *Example:* 6 brokers | [USD] |
| Archive storage | [USD per TB-month] | *Example:* 73 TB per year | [USD] |
| **Total** | | | **[USD]** |

## 10. Trade-offs and alternatives

<!-- Guidance: For each major decision list the options, what each costs and why you chose one. Add how the design changes at 10x and 0.1x. -->

| Decision | Options | Chosen | Why | What we give up |
|---|---|---|---|---|
| Cross-shard transfer | 2PC; TC/C; saga; event sourcing | Saga through in-flight account | Local transactions only, ledger always balanced | Short visible in-flight state |
| Balance storage | Computed; projection; snapshots | Projection plus snapshots | One-row reads | Recompute job and drift alert |
| Hot accounts | Row lock; buckets; batching | Buckets | Exact and parallel | Reads sum N rows |
| Idempotency store | Cache; database | Database, same transaction | No double spend on cache loss | Storage cost |
| Limits failure | Fail open; fail closed | Threshold-based | Small payments keep working | Some risk below threshold |
| **At 10x (120 000 transfers/s)** | More shards; event-sourced ledger | Move to event-sourced partitions with deterministic apply | Shard count and relay fan-out become the limit | Rewrite of ledger service and tooling |
| **At 0.1x (1 200 transfers/s)** | One primary; shards | One primary with replica, same schema | Cross-shard saga unnecessary; keep in-flight pattern for external flows | Re-sharding later is a migration |
| [Decision] | [Options] | [Chosen] | [Why] | [Cost] |

## 11. Delivery plan and risks

<!-- Guidance: How the design reaches production safely, and what could stop it. -->

### 11.1 Milestones

| Milestone | Scope | Exit criterion | Target date |
|---|---|---|---|
| M1 | Ledger service, single shard, idempotency | L1 reconciliation clean for 30 days in shadow mode | YYYY-MM-DD |
| M2 | Sharding and cross-shard saga | Chaos test of FM-01 and FM-02 with zero imbalance | YYYY-MM-DD |
| M3 | Limits and KYC hooks with versioned rules | Compliance sign-off on audit query | YYYY-MM-DD |
| M4 | Provider and bank reconciliation | 14 consecutive days with breaks under target | YYYY-MM-DD |
| M5 | Hot-account buckets | Load test at 2x largest merchant peak | YYYY-MM-DD |

### 11.2 Migration and rollback

[*Example:* Run the new ledger in shadow mode, posting the same transfers as
the old balance table and comparing balances hourly. Cut over wallets in
cohorts by shard. Rollback for a cohort before cutover is a routing switch;
after cutover, the old table is rebuilt from new ledger entries.]

### 11.3 Risks

| ID | Risk | Likelihood | Impact | Mitigation | Owner |
|---|---|---|---|---|---|
| R-01 | *Example:* Migration mismatch between old balances and opening entries | Medium | High | Opening balance transfer per wallet signed off by finance | [@name] |
| R-02 | *Example:* Regulator questions in-flight funds in safeguarding calculation | Low | High | Treat in-flight as customer funds in L4 | [@name] |
| R-03 | *Example:* Provider settlement file format changes | Medium | Medium | Schema checks, Data Contract (`ai-data/data-contract`) with provider feeds | [@name] |
| [R-nn] | [Risk] | [L / M / H] | [L / M / H] | [Mitigation] | [@name] |

## 12. Appendices

<!-- Guidance: Supporting detail, traceability, glossary and references. -->

### 12.1 Traceability

| Requirement | Design element | Verification |
|---|---|---|
| NFR-01 | 7.1 balanced entries, 7.5 L1 | Continuous invariant check |
| NFR-02 | 5.2 debit predicate | Property-based tests; nightly audit |
| NFR-03 | 7.3 idempotency | Fault injection with client retries |
| NFR-07 | 7.5 L3 and L4 | Reconciliation dashboard |
| NFR-08 | 7.6 rule version on transfer | Audit query test |
| [ID] | [Element] | [Test] |

### 12.2 Open questions

| ID | Question | Owner | Due |
|---|---|---|---|
| OQ-01 | *Example:* Do we allow transfers to unregistered phone numbers with a claim window? | [@name] | YYYY-MM-DD |
| OQ-02 | *Example:* Bucket count per merchant: static or adaptive? | [@name] | YYYY-MM-DD |
| [OQ-nn] | [Question] | [@name] | YYYY-MM-DD |

### 12.3 Glossary

| Term | Meaning |
|---|---|
| Double-entry | Every movement posts entries that sum to zero |
| In-flight account | Internal account that holds value between the steps of a cross-shard transfer |
| Hot account | Account receiving more updates than one row can serialise |
| Idempotency key | Client-generated identifier that makes retries safe |
| Reconciliation break | A difference between two records that should agree |
| Safeguarding | Keeping customer funds separate from the operator's own funds |
| KYC tier | Level of customer verification that unlocks a limit set |

### 12.4 References

| Reference | Use |
|---|---|
| Brandur Leach, "Designing robust and predictable APIs with idempotency", Stripe blog, 2017 | Idempotency contract and retry behaviour (7.3) |
| Alex Xu and Sahn Lam, *System Design Interview: An Insider's Guide, Volume 2*, ByteByteGo, 2022, chapter "Digital Wallet" | Inspiration for problem framing only; no text or numbers reused |
| RFC 9457 Problem Details for HTTP APIs | Error format (4.3) |
| IETF draft "The Idempotency-Key HTTP Header Field" (draft-ietf-httpapi-idempotency-key-header-07, expired) | Header semantics; not a standard |
| Garcia-Molina and Salem, "Sagas", ACM SIGMOD 1987 | Saga pattern (7.2) |
| Payment Ledger System Design (`system-design/design-payment-ledger`) | Card acquiring ledger and PSP states |
| Payment Fraud Detection System Design (`system-design/design-fraud-detection`) | Risk scoring called in 7.6 |
| Distributed Message Queue System Design (`system-design/design-message-queue`) | Event log and delivery semantics behind the outbox |
| Event Sourcing and CQRS Design (`architecture/event-sourcing-cqrs`) | The 10x alternative |

## Standard and licence note

Based on common industry practice for double-entry ledgers and stored-value
wallets and on the public references in 12.4. Books and company blogs were
used as inspiration only; the text, diagrams and numbers are original and
illustrative. The Idempotency-Key header field is an IETF draft, not a
published RFC. Regulatory content is a documentation format, not legal
advice: check the current consolidated texts and licence conditions for your
jurisdiction. Template licence: MIT.