Markdown Editor

장애 관리 키트

심각도 매트릭스, 알림 정책, 장애 지휘관 핸드북, 온콜, 런북, 장애 커뮤니케이션, 포스트모템, 모의 훈련, 게임 데이, DR 테스트, 토일 검토.

1단계

정책

  1. Incident Severity Matrix Template (SEV1 to SEV4)

    전체

    Organisation SEV1 to SEV4 standard, impact criteria, response per level, security track, crisis flag, upgrade and downgrade rules, worked examples

    PagerDuty Incident Response docs; NIST SP 800-61 Rev. 3
  2. Alerting Policy Template (SLO Burn-Rate Alerts)

    전체

    Symptom-based paging, page vs ticket vs log, multi-window burn-rate rules with worked numbers, alert metadata, naming, routing, noise budget, review, alerts as code

    OpenTelemetry semantic conventions 1.44.0; SLO burn-rate alerting
2단계

프로덕션 준비

  1. Incident Commander Handbook Template

    전체

    IC and response roles, RACI, staffing by severity, declaring and taking command, battle rhythm, decision log, escalation, handover, closing, one-page IC checklist

    PagerDuty Incident Response docs; FEMA NIMS 3rd ed. (ICS)
  2. On-call Handbook Template

    표준

    On-call handbook: rotation and handover, severity examples linked to the matrix, response targets, escalation, first 15 minutes, comms templates, wellbeing

    Links ops/incident-severity-matrix; Google SRE books (cited)
  3. Runbook Template

    표준

    Service runbook: overview, dependencies, dashboards, per-alert diagnosis and mitigation with commands, common tasks, rollback, contacts

3단계

장애

  1. Incident Communication Template

    표준

    Incident comms plan: audiences, severity matrix, roles, cadence, status page and customer messages, executive brief

  2. Planned Maintenance Notice Template

    경량

    Planned maintenance notice with times in several time zones, impact, actions required, status link, plus a completion follow-up

  3. Incident Postmortem Template

    전체

    Blameless incident postmortem: severity per matrix, impact and error budget, UTC timeline, root cause, 5 Whys, security incident links, action items

    Google SRE postmortem practice (concepts, 1st and 2nd ed.); NIST SP 800-61 Rev. 3
4단계

리허설과 테스트

  1. Tabletop Exercise Template (CISA CTEP, HSEEP, NIST SP 800-84)

    전체

    Discussion-based exercise pack: objectives and criteria, roles, ground rules, ransomware scenario with injects, questions, facilitator notes, AAR and Improvement Plan

    CISA Tabletop Exercise Packages; FEMA HSEEP 2020; NIST SP 800-84
  2. Game Day Plan Template

    표준

    Game day exercise: objectives, scenarios, roles, schedule, safety controls, comms, detection and response scoring, debrief

  3. DR Test Report Template (NIST SP 800-84, RTO and RPO Evidence)

    전체

    Evidence of one DR test: SP 800-84 event type, scope, scenario, timeline, RTO and RPO target vs achieved, integrity checks, deviations, findings, actions, sign-off

    NIST SP 800-84; NIST SP 800-34 Rev. 1; NIST SP 800-53 Rev. 5 CP-4, CP-9, CP-10; ISO 22301:2019 (clauses only)
5단계

평가

  1. Toil Review Template (Quarterly Operational Load and Automation ROI)

    전체

    Quarterly toil review: toil definition tests, measurement method, toil share per engineer vs cap, pages per shift, on-call health, toil inventory, automation ROI ranking, decisions

    Google SRE book (cited by name only)