# System Design: [Configuration and feature flag service name]

> Reference design for a dynamic configuration and feature flag service:
> the system that lets engineers change runtime behaviour (flags, limits,
> thresholds, targeting rules) across thousands of servers and millions of
> client devices without a deploy. It covers configuration as code,
> validation, targeting rules, staged and percentage rollout, consistent
> bucketing, fan-out distribution and SDK caching, kill switches, audit and
> the consistency of evaluation. The SDK interface follows the concepts of
> the OpenFeature specification (release v0.9.0, July 2026; sections carry
> their own stability status). It follows the structure of the System
> Design Document template and is pre-filled with a worked design to adapt.
> Replace every `[bracketed]` value and every row marked *Example*; all
> numbers are illustrative assumptions, not benchmarks.

<!--
Guidance: How to use this template
- Treat a configuration change as a code deploy that skips most of your
  safety checks. The design must put those checks back: review,
  validation, canary, staged rollout and fast rollback (7.3).
- Decide early which consumers evaluate rules locally (servers) and which
  receive evaluated values (browsers and mobile apps). This choice drives
  privacy, cost and propagation (7.1).
- Define the consistency promise explicitly: a user must not flip between
  variants within a session, and one request must not see two versions of
  the same flag (7.4).
- Keep flag hygiene in scope: every flag needs an owner and an expiry, or
  the system becomes a museum of dead code paths.
-->

## Document control

<!-- Guidance: A design is a proposal until it is approved; record who decided and when. -->

| Field | Value |
|---|---|
| Document ID | [SDD-CFG-001] |
| System / feature | [Dynamic configuration and feature flags] |
| Version | [0.1] |
| Status | [Draft / In review / Approved / Implemented / Superseded] |
| Owner | [@owner] |
| Author(s) | [@author] |
| Reviewers | [@reviewer-1], [@reviewer-2] |
| Last updated | YYYY-MM-DD |
| Classification | [Internal] |
| Consumers in scope | [Backend services, web app, iOS, Android, batch jobs] |
| SDK interface | [OpenFeature-compatible providers / proprietary] |
| Related | [Experimentation platform design], API Design Guidelines Template (`architecture/api-design-guidelines`), Release Checklist Template (`dev/release-checklist`) |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Tech lead, developer platform | [@name] | [Approved / Changes requested] | YYYY-MM-DD |
| Staff / principal engineer | [@name] | [Decision] | YYYY-MM-DD |
| SRE / operations | [@name] | [Decision] | YYYY-MM-DD |
| Security and privacy | [@name] | [Decision] | YYYY-MM-DD |
| Mobile platform | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: Note what changed in the design, not the wording. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | First draft for design review |
| 0.2 | YYYY-MM-DD | [@name] | *Example:* Moved client apps from local rule evaluation to remote evaluation to stop leaking targeting rules |
| 0.3 | YYYY-MM-DD | [@name] | *Example:* Added automatic rollback on health regression during staged rollout |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved |

## Contents

<!-- Guidance: Keep aligned with the headings. -->

1. [Overview](#1-overview)
2. [Requirements](#2-requirements)
3. [Capacity estimation](#3-capacity-estimation)
4. [API design](#4-api-design)
5. [Data model](#5-data-model)
6. [High-level design](#6-high-level-design)
7. [Detailed design](#7-detailed-design)
8. [Scalability and performance](#8-scalability-and-performance)
9. [Reliability, security and operations](#9-reliability-security-and-operations)
10. [Trade-offs and alternatives](#10-trade-offs-and-alternatives)
11. [Delivery plan and risks](#11-delivery-plan-and-risks)
12. [Appendices](#12-appendices)

## 1. Overview

<!-- Guidance: One page that a busy reviewer can read alone: problem, proposed solution in a paragraph, and the decisions that need their attention. -->

### 1.1 Problem statement

[*Example:* Teams use four different mechanisms for runtime switches:
environment variables, a key-value store read on every request, a vendor
SDK on mobile and hard-coded constants. A typo in a JSON value last quarter
disabled checkout in one region for 47 minutes because nothing validated
the change and it reached every server at once. Nobody can answer "what
changed in the last hour" across systems, and 61% of flags have no owner.]

### 1.2 Proposed solution

[*Example:* One service stores all configuration and flag definitions as
versioned, schema-validated objects in a repository. Every change goes
through review, automated validation and a staged rollout that starts on a
canary population and can roll back automatically. A distribution tier
pushes changes to an agent on each host, which caches them on disk; server
SDKs read from the local agent and evaluate rules in process. Browsers and
mobile apps call a stateless evaluation endpoint that returns already
evaluated values for the caller's context, so targeting rules never leave
the server side. Percentage rollouts use deterministic hashing so a user
stays in the same bucket as the percentage grows. Every change is audited
and every flag has an owner and an expiry date.]

### 1.3 Decisions requested from reviewers

| ID | Question | Recommended option | Section |
|---|---|---|---|
| Q-01 | *Example:* Push or poll for server distribution? | Push through a host agent, with polling as a fallback | 7.1 |
| Q-02 | *Example:* Evaluate on clients or on the server for mobile and web? | Server-side remote evaluation for clients; local evaluation for servers | 7.1, 10 |
| Q-03 | *Example:* How are percentage rollouts made sticky? | Hash of flag salt and targeting key into 10,000 buckets | 7.2 |
| Q-04 | *Example:* Do config changes roll out in stages like code? | Yes, by default; emergency kill switches bypass bake time | 7.3 |
| [Q-nn] | [Question] | [Option] | [Section] |

### 1.4 45-minute interview path

<!-- Guidance: Use this order in an interview or a time-boxed review. Spend the most time on one deep dive rather than touching all of them. -->

| Minutes | Cover | Key point to land |
|---|---|---|
| 0 to 5 | Scope: flags and dynamic config, who consumes them (servers, clients) | Servers and clients need different evaluation models |
| 5 to 10 | Core requirements: propagation time, evaluation latency, safety | Evaluation must be local and never block a request |
| 10 to 15 | Estimates: object count, subscribers, change rate, client polling rate | Fan-out, not storage, is the scaling problem |
| 15 to 20 | Data model: flag, variant, rule, segment, version, audit | Immutable versions make rollback trivial |
| 20 to 25 | High-level diagram: authoring, validation, store, distribution, SDKs | Read path never touches the authoring database |
| 25 to 38 | One deep dive: distribution and propagation (7.1) is the classic; consistent bucketing (7.2) or safe rollout (7.3) are good alternatives | Bounded propagation time with no thundering herd |
| 38 to 42 | Failure modes: bad config, distribution outage, SDK cold start | Last-known-good on disk; defaults in code |
| 42 to 45 | Trade-offs and what changes at 10x | Where the design breaks first |

## 2. Requirements

<!-- Guidance: Separate core requirements (the design must meet them) from below-the-line items (explicitly out of scope). Non-functional targets need a number and a way to measure it. -->

### 2.1 Core functional requirements

| ID | Requirement | Priority |
|---|---|---|
| FR-01 | *Example:* Define boolean, string, number and structured (JSON) flags with typed variants | Must |
| FR-02 | *Example:* Define dynamic configuration objects with a schema per namespace | Must |
| FR-03 | *Example:* Target by attributes (country, app version, plan), by named segment and by percentage | Must |
| FR-04 | *Example:* Roll out changes in stages with health checks and automatic rollback | Must |
| FR-05 | *Example:* Kill switch: turn a feature off everywhere within 10 seconds | Must |
| FR-06 | *Example:* Record who changed what, when and why, and show diffs | Must |
| FR-07 | *Example:* Server SDKs for [Go, Java, Python, Node.js]; client SDKs for web, iOS and Android | Must |
| FR-08 | *Example:* Report which variant each flag served (counts per flag and variant) | Should |
| FR-09 | *Example:* Owner and expiry on every flag, with stale-flag reports | Should |
| [FR-nn] | [Requirement] | [Must / Should / Could] |

### 2.2 Non-functional requirements

| ID | Quality | Target | Measured as |
|---|---|---|---|
| NFR-01 | Evaluation latency (server SDK) | *Example:* p99 below 10 microseconds, no network call | SDK benchmark |
| NFR-02 | Propagation time to servers | *Example:* p99 below 30 s from approval; kill switch below 10 s | Canary flag probe |
| NFR-03 | Read-path availability | *Example:* Flag reads never fail; stale reads allowed during outages | Chaos test with distribution down |
| NFR-04 | Consistency | *Example:* Same user gets the same variant for the same flag version on every server and client | Bucketing conformance test |
| NFR-05 | Auditability | *Example:* 100% of changes attributed and retained for [7 years] | Audit sampling |
| NFR-06 | Client privacy | *Example:* Targeting rules and other users' attributes never shipped to clients | Payload review |
| [NFR-nn] | [Quality] | [Target] | [Measurement] |

### 2.3 Below the line (out of scope)

- [*Example:* experiment analysis and statistics (experimentation platform consumes exposure data from this service)]
- [*Example:* secrets such as API keys and passwords (secret manager; this service must refuse values that look like secrets)]
- [*Example:* infrastructure configuration such as Kubernetes manifests (GitOps pipeline)]
- [*Example:* distribution of large binary artefacts such as ML models above 1 MB (artefact store; this service distributes a pointer and checksum)]

## 3. Capacity estimation

<!-- Guidance: Every estimate must end in a design consequence. If a number changes nothing, drop it. -->

### 3.1 Assumptions

| Input | Value | Source |
|---|---|---|
| Config objects and flags | *Example:* 60,000 objects, average 2 KB | Inventory |
| Server processes subscribing | *Example:* 500,000 on 25,000 hosts (20 per host) | Fleet inventory |
| Average share of processes subscribed to one namespace | *Example:* 5% | Namespace analysis |
| Approved changes | *Example:* 5,000 per day, peak 50 per minute | Change history |
| Concurrently active client devices | *Example:* 20 million | Product analytics |
| Client refresh interval | *Example:* 300 s while in foreground | Product decision |
| Flags evaluated per client refresh | *Example:* 200 | Client flag inventory |
| Server requests per second (all services) | *Example:* 2 million, 10 flag evaluations each | Service metrics |

### 3.2 Storage

60,000 objects x 2 KB = 120 MB for the full current state.

**Consequence:** the whole state fits in memory of the distribution tier, but
not sensibly in every process; servers subscribe by namespace (*Example:*
average 500 objects, about 1 MB per process).

### 3.3 Fan-out

One change reaches 5% of 500,000 processes = 25,000 processes. At 2 KB each
that is 50 MB per change; at the peak of 50 changes per minute it is
2.5 GB per minute, about 42 MB/s.

**Consequence:** pushing from one origin to 500,000 processes is the
problem. Push once per host to a local agent (25,000 agents, so 1,250
agents per change instead of 25,000 processes) through regional
distribution proxies; processes read from the agent over a local socket or
a memory-mapped file.

### 3.4 Client refresh rate

20 million devices / 300 s = about 66,700 requests/s. Each request
evaluates 200 flags; at *Example:* 2 microseconds per evaluation that is
66,700 x 200 x 2 microseconds = 26.7 CPU seconds per second, about 27 cores.

**Consequence:** remote evaluation for clients is affordable (about 54
cores with 2x headroom across regions), so there is no need to ship rules to
devices. Responses are personalised, so a CDN cannot cache them; instead the
endpoint returns a hash of the evaluated result and clients send it back in
a `known_hash` field, so an unchanged result returns a short "not modified"
body. (Do not rely on `If-None-Match` here: on a POST, RFC 9110 answers a
failed condition with 412, not 304.)

### 3.5 Server evaluations and exposure data

2 million requests/s x 10 evaluations = 20 million evaluations/s, all in
process.

**Consequence:** evaluation CPU is negligible, but logging every evaluation
is not (20 million events/s). SDKs aggregate counts per flag, variant and
minute, and emit individual exposure events only for flags linked to an
experiment.

### 3.6 Summary

| Quantity | Estimate | Design consequence |
|---|---|---|
| Total state | 120 MB | Namespaced subscriptions |
| Fan-out per change | 25,000 processes, 1,250 hosts | Host agent plus regional proxies |
| Client refresh | about 66,700 requests/s | Stateless evaluation service, result hash |
| Evaluations | 20 million/s | In-process evaluation; aggregated exposure counts |
| Audit | 5,000 changes per day | Single relational store, retained per NFR-05 |

## 4. API design

<!-- Guidance: Define the authoring API, the distribution protocol and the SDK interface separately; they have different consumers and stability needs. -->

### 4.1 Interfaces

| Interface | Operation | Purpose |
|---|---|---|
| Authoring API | `POST /v1/namespaces/{ns}/changes` | Propose a change (creates a reviewable change request) |
| Authoring API | `POST /v1/changes/{id}/approve` | Approve; starts the rollout plan |
| Authoring API | `POST /v1/flags/{key}/kill` | Kill switch: set the off variant everywhere, fast path |
| Authoring API | `POST /v1/changes/{id}/rollback` | Roll back to the previous version |
| Distribution | `WATCH /v1/subscriptions` (streaming) | Agents receive versioned deltas for subscribed namespaces |
| Client evaluation | `POST /v1/evaluate` | Evaluate all client-visible flags for a context |
| SDK | `getBooleanValue(key, default, context)` and typed variants | In-process evaluation |
| SDK | `getBooleanDetails(key, default, context)` | Value plus variant, reason and error code |

### 4.2 SDK contract

Server SDKs expose an OpenFeature-style interface: an evaluation API with
typed getters, an evaluation context (targeting key plus attributes),
providers that implement the actual lookup, and hooks that run before and
after evaluation. The detailed result carries the flag key, value, variant,
reason and, when evaluation fails, an error code. Use the specification's
reason values (for example `STATIC`, `DEFAULT`, `TARGETING_MATCH`, `SPLIT`,
`CACHED`, `DISABLED`, `STALE`, `ERROR`) so dashboards can explain each
result. The default value is always passed by the caller and returned on
any error; evaluation never throws into application code.

```go
// Example: server-side use through an OpenFeature-style client
ctx := openfeature.NewEvaluationContext("user-8d1f", map[string]any{
    "country": "DE", "plan": "pro", "app_version": "5.12.0",
})
details, _ := client.BooleanValueDetails(reqCtx, "checkout.new-tax-engine", false, ctx)
// details.Value, details.Variant, details.Reason (e.g. "SPLIT"), details.ErrorCode
```

### 4.3 Client evaluation request

```http
POST /v1/evaluate HTTP/1.1
Content-Type: application/json

{"context": {"targetingKey": "user-8d1f", "platform": "ios", "app_version": "5.12.0", "country": "DE"}, "flags": ["*client"], "known_hash": "ev-3f9a"}
```

The response lists key, value and variant per flag, plus the snapshot
version used, so a client can report which version it saw. The OpenFeature
project also publishes a remote evaluation protocol (OFREP), defined in
the open-feature/protocol repository and described in Appendix C of the
specification; consider adopting it for this endpoint so standard
client providers can call it.

### 4.4 Errors

Authoring API errors use Problem Details (RFC 9457). Validation failures
return 422 with one entry per failed rule, so the author sees every problem
at once.

## 5. Data model

<!-- Guidance: Use immutable versions. A rollback should be "make version N-1 current", never "edit back by hand". -->

### 5.1 Entities

Definitions are immutable once approved; a change creates a new version and
a rollout that moves populations from the old version to the new one.

```mermaid
erDiagram
  NAMESPACE {
    string ns_id PK
    string owner_team
    string schema_ref
  }
  FLAG {
    string flag_key PK
    string ns_id FK
    string type
    string owner
    date expires_on
    string salt
  }
  FLAG_VERSION {
    string flag_key PK
    int version PK
    string variants_json
    string rules_json
    string default_variant
    string off_variant
    datetime approved_at
  }
  SEGMENT {
    string segment_id PK
    string definition_json
    int version
  }
  CHANGE {
    string change_id PK
    string author
    string approver
    string reason
    string diff
    datetime created_at
  }
  ROLLOUT {
    string rollout_id PK
    string change_id FK
    string stage
    string state
  }
  NAMESPACE ||--o{ FLAG : contains
  FLAG ||--o{ FLAG_VERSION : "versioned as"
  FLAG_VERSION }o--o{ SEGMENT : references
  CHANGE ||--|{ FLAG_VERSION : creates
  CHANGE ||--o| ROLLOUT : "rolled out by"
```

### 5.2 Storage choices

| Data | Store | Why |
|---|---|---|
| Definitions, versions, changes | Git repository as source of truth, mirrored to a relational database for queries | Review, diff and blame; transactional queries for the UI |
| Current snapshot per namespace | In memory on distribution proxies, persisted in an object store | Fast fan-out; cold start from object store |
| Host agent cache | Local disk file plus memory map | Processes start with last-known-good even when the network is down |
| Exposure counts | Event pipeline to the analytics store | High volume, append only |
| Audit log | Append-only table, replicated | Compliance retention |

### 5.3 Access patterns

| Query | Served by | Key or index | Partition rationale |
|---|---|---|---|
| Evaluate flag X for context C (server) | In-process snapshot | Flag key hash map | No partition: the namespace snapshot is small (*Example:* 1 MB) |
| Evaluate all client flags for context C | Evaluation service memory | Client-visible flag list | Stateless service; snapshot replicated to every instance |
| Changes since version V for namespace N | Distribution proxy | (`ns_id`, `version`) | Partition proxies by namespace hash so one hot namespace does not load all proxies |
| Who changed flag X in the last 24 h | Audit table | (`flag_key`, `created_at`) | Single store; low volume |
| Flags past expiry, by owner | Relational mirror | (`owner`, `expires_on`) | Low volume |
| Which variant did user U see | Exposure events for experiment flags | (`flag_key`, `targeting_key`, time) | Partition by flag key in the analytics store |

## 6. High-level design

<!-- Guidance: Show authoring (write path), distribution (fan-out) and evaluation (read path) as separate lanes, and make clear that the read path does not depend on the write path being up. -->

```mermaid
flowchart LR
  subgraph author["Authoring"]
    ui[UI and CLI] --> api[Authoring API]
    api --> repo[(Git repo)]
    repo --> val[Validation and compile]
    val --> ro[Rollout controller]
  end
  ro --> snap[(Snapshot store)]
  subgraph dist["Distribution per region"]
    snap --> px1[Proxy shard A]
    snap --> px2[Proxy shard B]
  end
  px1 --> ag1[Host agent]
  px2 --> ag2[Host agent]
  ag1 --> p1[Service process, server SDK]
  ag2 --> p2[Service process, server SDK]
  snap --> evs[Client evaluation service]
  apps([Web and mobile clients]) --> evs
  p1 --> exp[[Exposure counts]]
  evs --> exp
  health[[Service health metrics]] --> ro
```

| Component | Responsibility | Scales by |
|---|---|---|
| Authoring API and UI | Propose, review, approve, kill, roll back | Low traffic; one deployment |
| Validation and compile | Schema checks, rule linting, conflict and reference checks | CI workers |
| Rollout controller | Advance stages, watch health, roll back | One active leader per region |
| Snapshot store | Durable compiled snapshots per namespace and version | Object store |
| Distribution proxies | Hold snapshots, stream deltas to agents | Shards by namespace hash |
| Host agent | Subscribe once per host, write local cache | One per host |
| Server SDK | Evaluate locally from the agent's cache | With each process |
| Client evaluation service | Evaluate client flags per context | Stateless instances |

## 7. Detailed design

<!-- Guidance: Go deep on the parts that are new, risky or contested. For each deep dive list the options, the choice and the reason. -->

### 7.1 Deep dive 1: distribution and propagation

| Option | Propagation | Load on origin | Failure behaviour |
|---|---|---|---|
| A. Every process polls a central API | Poll interval (for example 30 s) | 500,000 / 30 s = about 16,700 requests/s, mostly unchanged | Thundering herd after an outage |
| B. Every process holds a stream to the service | Seconds | 500,000 open streams | Reconnect storms |
| C. Host agent streams from regional proxies; processes read locally | Seconds | 25,000 streams spread across proxies | Agent serves last-known-good from disk |

**Choice:** C. **Reason:** it cuts the fan-out by the number of processes
per host, removes the network from evaluation entirely and gives every host
a durable cache. Agents also poll every *Example:* 5 minutes with the last
version they hold, so a missed stream message is repaired without manual
action. Reconnects use jittered exponential backoff.

The sequence for one approved change:

```mermaid
sequenceDiagram
  participant A as Author
  participant API as Authoring API
  participant V as Validation
  participant RC as Rollout controller
  participant S as Snapshot store
  participant P as Proxy
  participant H as Host agent
  participant SDK as Server SDK
  A->>API: propose change (diff, reason)
  API->>V: validate schema, rules, references
  V-->>API: ok
  API->>RC: approved change 812
  RC->>S: write snapshot ns=checkout v=213
  RC->>P: publish v=213 for stage 1 hosts
  P->>H: delta v=212 to v=213
  H->>H: write cache file atomically (rename)
  H-->>SDK: change notification
  SDK->>SDK: swap snapshot pointer
  RC->>RC: watch health for bake time
```

Large values (above *Example:* 1 MB) are not pushed inline: the snapshot
holds a URL and checksum, and agents fetch the blob from the artefact store
or from peers.

### 7.2 Deep dive 2: targeting and consistent bucketing

Rules are evaluated top to bottom; the first match wins:

1. If the flag is killed, return the off variant (reason `DISABLED`).
2. Individual targets (allow lists for testers).
3. Segment and attribute rules (`country in [DE, FR] and app_version >= 5.12.0`).
4. Percentage split for the remaining population.
5. Default variant.

| Option for percentage splits | Sticky as percentage grows | Same answer on every SDK | Verdict |
|---|---|---|---|
| A. Random per evaluation | No | No | Never for user-facing features |
| B. Stored assignment per user | Yes | Yes, with a lookup | Needs a database on the hot path |
| C. Deterministic hash of salt and targeting key | Yes | Yes, if every SDK implements the same hash | Chosen |

**Choice:** C. **Reason:** no storage and no network, and increasing a
rollout from 10% to 20% keeps the first 10% in. Bucket rule:

```text
bucket = hash32(flag.salt + "." + context.targetingKey) mod 10000
in_rollout = bucket < percentage * 100      # 25.5% -> bucket < 2550
```

- Use one hash function across all SDKs (*Example:* MurmurHash3 32-bit) and
  publish conformance vectors so every language agrees.
- Give each flag its own salt so the same users are not always first into
  every rollout; reuse a salt deliberately when two flags must align.
- If the targeting key is missing, return the default with error code
  `TARGETING_KEY_MISSING` rather than bucketing on an empty string, which
  would put every anonymous user in one bucket.

### 7.3 Deep dive 3: safe change and kill switches

| Option | Safety | Speed | Verdict |
|---|---|---|---|
| A. Apply instantly everywhere | Low | Fastest | Only for kill switches |
| B. Manual staged rollout | Medium | Slow, depends on people watching | Error-prone at night |
| C. Automated staged rollout with health gates | High | Minutes | Chosen as default |

**Choice:** C, with A reserved for kill switches. **Reason:** the outage in
1.1 was a valid-looking change applied everywhere at once; staged exposure
plus automatic rollback limits a bad change to the canary population.

Stages (*Example*): internal hosts, 1% of hosts in one region, one region,
all regions. Each stage bakes for *Example:* 5 minutes. The controller
compares error rate, latency and crash rate of hosts on the new version
against hosts still on the old one and rolls back automatically on a
regression above threshold.

Validation before any stage:

- JSON schema per namespace (types, ranges, required fields).
- Rule linting: unreachable rules, overlapping segments, percentage totals.
- Reference checks: segments and variants that exist.
- Secret scanning: reject values that look like credentials.
- Optional custom validators owned by the consuming team, run in CI.

Kill switches are pre-declared off variants. A kill sets the flag version to
"off" through a priority channel that skips bake time, is limited to users
with the kill permission, and is always audited.

### 7.4 Deep dive 4: consistency of evaluation

| Concern | Rule |
|---|---|
| Within one request | The SDK pins one snapshot per request (or per unit of work) so two reads of the same flag cannot disagree |
| Across flags | Flags in one namespace change atomically with the namespace version; cross-namespace atomicity is not promised, so dependent flags belong in one namespace |
| Across servers | Eventually consistent within the propagation target (NFR-02); version numbers in logs make skew visible |
| Across server and client | Both use the same bucketing; the client sees the snapshot version that was current when it last refreshed |
| Monotonic versions | Agents and SDKs refuse a version lower than the one they hold, except when the rollback command explicitly asks |

**Choice:** per-namespace snapshot versions with request-level pinning.
**Reason:** global strong consistency would put a consensus round on every
change and would not stop clients from holding stale values anyway; making
staleness bounded and visible is enough for flags.

> **How real systems do it** (paraphrased from Tang et al., "Holistic
> Configuration Management at Facebook", SOSP 2015; see 12.4). The paper
> describes a stack in which configuration is written as code, compiled and
> validated before it is committed, and stored in version control, so every
> change is reviewed and reversible. Changes are tested automatically on a
> small set of servers before wide release. A separate gating tool
> evaluates rollout and targeting rules inside the application, and mobile
> apps receive configuration through their own mechanism. Distribution is
> layered: a replicated store feeds proxies on each server, and large
> configurations travel through a hybrid of subscription and peer-to-peer
> transfer. The authors report that configuration errors were a major
> cause of incidents, which motivated treating config changes with the same
> rigour as code. Lesson for this design: invest first in validation,
> canary testing and per-host caching, not in a faster database.

### 7.5 Flag lifecycle

| State | Entry | Exit |
|---|---|---|
| Draft | Created by owner | Approved |
| Rolling out | Stage 1 started | Reached 100% or rolled back |
| Fully on or off | Stable for 14 days | Code cleanup removes references |
| Expired | `expires_on` passed | Owner extends with reason or removes |
| Archived | No references in code for 30 days | Kept for audit only |

A weekly report lists expired flags by owner; the code scanner flags
references to archived keys in pull requests.

## 8. Scalability and performance

<!-- Guidance: Show how the design meets the numbers in section 3 and where it breaks first. -->

| Concern | Approach | Limit before redesign |
|---|---|---|
| Fan-out | Host agents, proxies sharded by namespace | *Example:* about 5,000 agent streams per proxy instance |
| Snapshot size per namespace | Split large namespaces; blobs by reference | *Example:* 5 MB per namespace before delta push latency grows |
| Client evaluation | Stateless, horizontally scaled | Evaluation CPU, linear |
| Rule complexity | Compile rules to predicates; cap rules per flag | *Example:* 200 rules per flag |
| Change rate | Batch changes per namespace into one version per second | *Example:* about 100 changes per minute per namespace |

Propagation budget for a normal change (target p99 30 s after the stage
starts):

| Step | Budget |
|---|---|
| Compile and write snapshot | 3 s |
| Proxy fan-out to agents | 5 s |
| Agent write and SDK swap | 1 s |
| Stream reconnect worst case for a lagging agent | 15 s |
| Headroom | 6 s |

## 9. Reliability, security and operations

<!-- Guidance: Name the failure modes, the threats and the signals operators watch. The read path must survive the loss of everything upstream of the host. -->

### 9.1 Failure modes

| Component | Failure | Detection | Blast radius | Mitigation | RTO |
|---|---|---|---|---|---|
| Change content | Valid-looking but harmful value | Health gate in canary stage | Canary population | Automatic rollback | *Example:* 5 min |
| Authoring API | Down | Synthetic probe | No new changes; kill switch unavailable | Break-glass CLI writes snapshot directly with audit | *Example:* 15 min |
| Distribution proxies | Shard down | Agent stream errors | Hosts on that shard stop receiving changes | Agents fail over to another shard; serve last-known-good | *Example:* 1 min |
| Host agent | Crashed | Agent heartbeat | Processes on that host | SDK reads cache file directly; supervisor restarts agent | *Example:* 30 s |
| Process cold start without cache | No snapshot available | SDK reason `DEFAULT`, error `PROVIDER_NOT_READY` | That process | Code defaults; bootstrap file in the image | *Example:* immediate (degraded) |
| Client evaluation service | Down | Error rate | Client apps | Clients keep last cached values; ship defaults in the app | *Example:* 10 min |
| Bucketing bug in one SDK | Users see different variants per platform | Cross-SDK conformance test, variant mismatch metric | Users of that SDK | Pin SDK version; fix and publish vectors | Release cycle |
| [Component] | [Failure] | [Signal] | [Scope] | [Mitigation] | [RTO] |

### 9.2 Security and privacy

| Threat | Control |
|---|---|
| Unauthorised change to a critical flag | Namespace-level permissions; two-person approval for namespaces marked critical |
| Insider abuse of kill switch | Separate kill permission, audited, alert to namespace owners |
| Leaking targeting rules or segment membership to clients | Remote evaluation for clients; client responses carry only values and variants |
| Personal data in evaluation context | Allow-list of context attributes; hash or drop identifiers in exposure events |
| Secrets stored as config | Secret scanning in validation; reject on match |
| Tampered snapshot in transit | Signed snapshots verified by agents; TLS between tiers |
| [Threat] | [Control] |

Review the privacy impact with the Privacy Threat Model Template
(`security/privacy-threat-model`) when context attributes include personal
data.

### 9.3 Observability

| Signal | Source | Alert threshold |
|---|---|---|
| Propagation lag | Canary flag changed every minute, read back on sampled hosts | *Example:* p99 above 30 s |
| Version skew | Active version per namespace across agents | Hosts more than 2 versions behind for 10 min |
| Evaluation errors by code | SDK hook counters | *Example:* `ERROR` reason above 0.1% for a flag |
| Default-served rate | SDK counters | Sudden rise after a change |
| Stale flags | Weekly lifecycle report | Expired flags per owner |
| Change-related incidents | Incident tags | Reviewed monthly |

### 9.4 Cost estimate

| Item | Unit cost | Quantity | Monthly cost |
|---|---|---|---|
| Distribution proxies | [USD per instance-month] | *Example:* 6 per region x 3 | [USD] |
| Client evaluation service | [USD per core-month] | *Example:* 54 cores across regions | [USD] |
| Snapshot and audit storage | [USD per GB-month] | *Example:* under 50 GB | [USD] |
| Exposure pipeline | [USD per GB ingested] | [GB] | [USD] |
| **Total** | | | **[USD]** |

## 10. Trade-offs and alternatives

<!-- Guidance: For each major decision list the options, what each costs and why you chose one. Include how the design changes at ten times and one tenth of the scale. -->

| Decision | Options | Chosen | Why | What we give up |
|---|---|---|---|---|
| Source of truth | Database only; Git plus database mirror | Git plus mirror | Review, diff, blame, revert | Two stores to keep in sync |
| Server evaluation | Remote call; local | Local | No network on the hot path | Rules ship to servers (acceptable inside the trust boundary) |
| Client evaluation | Local rules on device; remote | Remote | Rules and segments stay private | Per-request server cost; offline uses cached values |
| Distribution | Poll; per-process stream; host agent | Host agent | Fan-out reduced by processes per host | One more daemon on every host |
| Rollout of config | Instant; staged | Staged with kill-switch bypass | Limits blast radius | Normal changes take minutes, not seconds |
| Build or buy | Vendor service; open-source; build | [Decide] | [Reason] | [Cost] |
| What changes at 10x | 5 million processes, 200 million devices | Second-level proxy tier, delta compression, regional evaluation clusters close to clients | Proxy stream count and client QPS bind first | More tiers to monitor |
| What changes at 0.1x | 50,000 processes, 2 million devices | Drop host agents; processes stream directly; a vendor or open-source flag service is likely cheaper | Team cost dominates | Less control over propagation and audit |

## 11. Delivery plan and risks

<!-- Guidance: How the design reaches production safely, and what could stop it. -->

### 11.1 Milestones

| Milestone | Scope | Exit criterion | Target date |
|---|---|---|---|
| M1 | Authoring, validation, Git source of truth, server SDK for one language | Two pilot services read flags locally | YYYY-MM-DD |
| M2 | Host agents and proxies in one region; staged rollout | Propagation p99 below 30 s for 14 days | YYYY-MM-DD |
| M3 | Client evaluation service, web and mobile SDKs | Conformance vectors pass on all SDKs | YYYY-MM-DD |
| M4 | Migration of legacy mechanisms; lifecycle reports | Legacy key-value reads below 1% | YYYY-MM-DD |

### 11.2 Rollout and rollback

[*Example:* Migrate one namespace at a time. During migration the SDK
provider reads both the old mechanism and the new service and logs
mismatches; switch the provider to the new service once mismatches are zero
for 7 days.]

### 11.3 Risks

| ID | Risk | Likelihood | Impact | Mitigation | Owner |
|---|---|---|---|---|---|
| R-01 | *Example:* Teams bypass the service with environment variables | Medium | Medium | Golden path templates; lint rule | [@name] |
| R-02 | *Example:* Flag debt grows faster than cleanup | High | Medium | Mandatory expiry; weekly report | [@name] |
| R-03 | *Example:* SDK hash mismatch across languages | Low | High | Shared conformance vectors in CI | [@name] |
| R-04 | *Example:* Specification changes in experimental OpenFeature sections | Medium | Low | Depend only on stable or hardening sections; wrap experimental features | [@name] |
| [R-nn] | [Risk] | [L / M / H] | [L / M / H] | [Mitigation] | [@name] |

## 12. Appendices

<!-- Guidance: Supporting detail, glossary, open questions and references. -->

### 12.1 Open questions

| ID | Question | Owner | Due |
|---|---|---|---|
| OQ-01 | *Example:* Does the experimentation platform need per-user exposure events for all flags or only experiment flags? | [@name] | YYYY-MM-DD |
| OQ-02 | *Example:* Do mobile apps need offline evaluation of a subset of rules? | [@name] | YYYY-MM-DD |
| [OQ-nn] | [Question] | [@name] | YYYY-MM-DD |

### 12.2 Traceability

| Requirement | Design section | Verification |
|---|---|---|
| FR-03, NFR-04 | 7.2 | Bucketing conformance vectors on every SDK |
| FR-04 | 7.3 | Game day with a deliberately harmful value |
| FR-05 | 7.3 | Kill switch drill, measured end to end |
| NFR-02 | 7.1, 8 | Canary flag probe |
| NFR-03 | 9.1 | Chaos test with proxies and agents stopped |
| NFR-06 | 4.3, 9.2 | Payload review of client responses |

### 12.3 Glossary

| Term | Meaning |
|---|---|
| Flag | A named switch with typed variants and rules that select one |
| Variant | One possible value of a flag |
| Targeting key | The stable identifier used for bucketing, usually a user or device ID |
| Segment | A reusable named set of targeting rules |
| Kill switch | A pre-declared off variant applied everywhere through a fast path |
| Snapshot | A compiled, versioned copy of a namespace's definitions |
| Provider | In OpenFeature terms, the component that resolves flag values for the SDK |

### 12.4 References

| Reference | Use |
|---|---|
| OpenFeature specification, release v0.9.0 (July 2026), openfeature.dev/specification; OpenFeature is a CNCF incubating project | SDK interface, evaluation details, reason and error codes (4.2) |
| OpenFeature Remote Evaluation Protocol (OFREP), defined in the open-feature/protocol repository (described in specification Appendix C) | Option for the client evaluation endpoint (4.3) |
| C. Tang et al., "Holistic Configuration Management at Facebook", SOSP 2015, DOI 10.1145/2815400.2815401 | Inspiration for the box in 7.4 (paraphrased) |
| RFC 9457 Problem Details for HTTP APIs | Authoring API errors |
| Release Checklist Template (`dev/release-checklist`) | Release gates that reference flags |
| API Design Guidelines Template (`architecture/api-design-guidelines`) | Authoring API conventions |
| Notification Service System Design Template (`system-design/design-notification-service`) | Pattern for fan-out to many devices |

## Standard and licence note

Based on the OpenFeature specification (release v0.9.0, sections with their
own stability status) for the SDK interface and on RFC 9457 for errors. The
Facebook paper is used as inspiration only and is paraphrased, not copied.
All numbers are illustrative examples. Check the current version and
section status of the specification at openfeature.dev before you
implement. Original text written for this collection. Template licence:
MIT.