# Administrator Guide: [Product name] [version]

> Complete guide for the people who run, secure and maintain a software
> product after installation. The structure follows the information for
> administrators and operators described in ISO/IEC/IEEE 26514:2022.
> Replace every `[bracketed]` value and every row marked *Example*.

<!--
Guidance: How to use this template
- Write for an administrator who inherits the system at short notice. They
  need to know what exists, where it lives, what to check daily and what to
  do when something breaks.
- Keep procedures task-based (numbered steps with expected results) and
  keep reference facts in tables. Link to the Installation Guide for setup.
- Record real values for your deployment (hosts, schedules, retention) in
  your internal copy; the published copy uses placeholders.
- Review this guide after every major upgrade and every incident that
  revealed a gap.
-->

## Document control

<!-- Guidance: Identify the product version and deployment the guide covers. Status moves Draft -> In review -> Approved -> Superseded. -->

| Field | Value |
|---|---|
| Document ID | [AG-PRODUCT-001] |
| Product and version | [Product name] [3.2] |
| Guide version | [0.1] |
| Status | [Draft / In review / Approved / Superseded] |
| Owner | [@platform-owner] |
| Approvers | [@service-owner], [@security-officer] |
| Classification | [Internal / Confidential] |
| Last updated | YYYY-MM-DD |
| Related documents | [Installation Guide], [User Manual], [Runbooks], [DR plan] |

### Approvals

| Role | Name | Decision | Date |
|---|---|---|---|
| Service owner | [@name] | [Approved / Rejected / Approved with comments] | YYYY-MM-DD |
| Security officer | [@name] | [Decision] | YYYY-MM-DD |
| Operations lead | [@name] | [Decision] | YYYY-MM-DD |

## Revision history

<!-- Guidance: One row per published version. Mention the product release or incident that triggered the change. -->

| Version | Date | Author | Change summary |
|---|---|---|---|
| 0.1 | YYYY-MM-DD | [@name] | Initial draft |
| 1.0 | YYYY-MM-DD | [@name] | *Example:* Approved for 3.2 rollout |
| 1.1 | YYYY-MM-DD | [@name] | *Example:* Added restore drill after incident INC-2291 |

## Contents

1. [Introduction](#1-introduction)
2. [Roles and permissions](#2-roles-and-permissions)
3. [Architecture overview](#3-architecture-overview)
4. [User and access management](#4-user-and-access-management)
5. [Configuration management](#5-configuration-management)
6. [Backup and restore](#6-backup-and-restore)
7. [Monitoring and alerting](#7-monitoring-and-alerting)
8. [Logs](#8-logs)
9. [Maintenance schedule](#9-maintenance-schedule)
10. [Scaling](#10-scaling)
11. [Security operations](#11-security-operations)
12. [Upgrades and patching](#12-upgrades-and-patching)
13. [Disaster recovery](#13-disaster-recovery)
14. [Audit and compliance](#14-audit-and-compliance)
15. [Reference](#15-reference)

## 1. Introduction

<!-- Guidance: Say what the guide covers, who it is for, and what it assumes. Point to the other guides so readers do not look for installation steps here. -->

### 1.1 Purpose and scope

This guide describes how to administer [Product name] [3.2] in the
[production / staging] environment: managing users and access, changing
configuration, protecting data, monitoring health, maintaining and scaling
the service, and responding to security events. Installation is covered
in the [Installation Guide].

### 1.2 Audience and assumptions

This guide is for system administrators, site reliability engineers and
security operators. It assumes familiarity with Linux administration,
PostgreSQL, TLS certificates and the organisation's change management
process.

### 1.3 First week checklist for a new administrator

| # | Task | Section |
|---|---|---|
| 1 | Get the System administrator role and MFA enrolled | 2, 4.4 |
| 2 | Walk through the component map and find each host | 3 |
| 3 | Join the on-call rotation and read every runbook linked in 7.2 | 7 |
| 4 | Perform a supervised restore test | 6.3 |
| 5 | Read the last three incident reports and the open risk list | 11, 14 |

### 1.4 Conventions

Commands prefixed with `#` require root or `sudo`. **Caution** marks steps
that can cause downtime or data loss. Times are in UTC.

## 2. Roles and permissions

<!-- Guidance: List every administrative role, what it can do and who holds it. Apply least privilege: each person should hold the smallest role that lets them do their job. -->

| Role | Can do | Cannot do | Held by |
|---|---|---|---|
| *Example:* Super administrator | Everything, including role assignment and deletion of tenants | Nothing restricted | 2 named people, break-glass only |
| *Example:* System administrator | Configuration, backups, upgrades, monitoring | Read user content, assign roles | Platform team |
| *Example:* User administrator | Create, disable and reassign users; reset MFA | Change system configuration | Service desk |
| *Example:* Auditor | Read audit logs and reports | Change anything | Internal audit |
| [Role] | [Permissions] | [Restrictions] | [Team or person] |

Changes to role assignments require a ticket approved by [@service-owner]
and are reviewed quarterly (see 14.2).

## 3. Architecture overview

<!-- Guidance: Give administrators the component map they need to reason about failures: what runs where, what depends on what, and where data lives. Link to the architecture description for design rationale. -->

The service consists of stateless application nodes behind a load
balancer, a primary database with one replica, an object store for
files and a background worker pool.

```mermaid
flowchart LR
    U["Users"] --> LB["Load balancer"]
    LB --> A1["App node 1"]
    LB --> A2["App node 2"]
    A1 --> DB[("PostgreSQL primary")]
    A2 --> DB
    DB --> R[("Replica")]
    A1 --> OS["Object store"]
    W["Worker pool"] --> DB
    W --> OS
```

| Component | Host or service | Data held | Failure impact |
|---|---|---|---|
| *Example:* App nodes | app01, app02 (4 vCPU, 8 GB) | None (stateless) | Reduced capacity; no data loss |
| *Example:* PostgreSQL | db01 primary, db02 replica | All metadata | Full outage until failover |
| *Example:* Object store | [S3 bucket `prod-files`] | Uploaded files | Uploads and downloads fail |
| [Component] | [Host] | [Data] | [Impact] |

## 4. User and access management

<!-- Guidance: Cover the whole account life cycle: joiner, mover, leaver. State the identity source and what happens to a leaver's content. -->

### 4.1 Identity source

Users authenticate through [SSO with SAML 2.0 / OpenID Connect against
the corporate identity provider]. Local accounts are allowed only for the
two break-glass super administrators.

### 4.2 Add a user

1. In the admin console, go to **Users > Add user**.
2. Enter the email address and select a role.
3. Select **Send invitation**.
   **Result:** The user appears with status **Invited**; it changes to
   **Active** after the first sign-in.

With SSO provisioning (SCIM) enabled, users are created automatically when
they are added to the group `[app-users]` in the identity provider.

### 4.3 Disable a leaver

1. Go to **Users**, search for the person and select **Disable**.
2. Choose a new owner for their documents and select **Transfer**.
   **Result:** Active sessions end within one minute; content is kept.

### 4.4 Access policy

| Policy | Setting |
|---|---|
| Multi-factor authentication | Required for all administrator roles |
| Session timeout | 8 hours idle, 24 hours absolute |
| Password policy (local accounts) | Minimum 14 characters, checked against breached-password lists |
| Inactive account review | Accounts unused for 90 days are disabled automatically |

## 5. Configuration management

<!-- Guidance: Say where configuration lives, how changes are made and reviewed, and how to roll back. Configuration drift is a common root cause of incidents. -->

All configuration is stored in the repository `[infra/productname-config]`
and applied by [the deployment pipeline]. Manual edits on servers are
overwritten on the next deploy.

| Change type | Approval | How applied | Rollback |
|---|---|---|---|
| *Example:* Feature flag | Peer review | Pipeline, no restart | Revert commit |
| *Example:* Database pool size | Peer review + change ticket | Pipeline, rolling restart | Revert commit |
| *Example:* Authentication provider | Change advisory board | Maintenance window | Restore previous config and restart |

Validate before applying: `productname config validate --file config.yaml`.

## 6. Backup and restore

<!-- Guidance: State what is backed up, how often, where, how long it is kept and how restores are tested. A backup that has never been restored is not a backup. -->

### 6.1 Backup policy

| Data | Method | Frequency | Retention | Location |
|---|---|---|---|---|
| *Example:* Database | Continuous WAL archiving + nightly base backup | Continuous / 02:00 daily | 35 days | Separate region, encrypted |
| *Example:* Object store | Versioning + cross-region replication | Continuous | 90 days of versions | Secondary region |
| *Example:* Configuration | Git repository | Every change | Indefinite | Code host |
| [Data] | [Method] | [Frequency] | [Retention] | [Location] |

### 6.2 Restore the database to a point in time

> **Caution:** Restoring replaces current data. Confirm the target time
> with the service owner and announce downtime first.

1. Put the service in maintenance mode: `productname maintenance on`.
2. Restore: `productname db restore --target-time "YYYY-MM-DD HH:MM:SS UTC"`.
3. Run integrity checks: `productname db check`; expect `0 errors`.
4. Turn maintenance mode off and run the checks in 7.3.

### 6.3 Restore testing

A full restore to an isolated environment is performed [monthly] and
recorded in the table below.

| Date | Backup tested | Restore time | Result | Tester |
|---|---|---|---|---|
| YYYY-MM-DD | *Example:* Nightly base backup of YYYY-MM-DD | 42 min | Pass | [@name] |

## 7. Monitoring and alerting

<!-- Guidance: List the signals that tell you the service is healthy, the thresholds that page someone, and who gets paged. Every alert needs a runbook link. -->

### 7.1 Service level objectives

Availability is measured as the share of successful requests:

$$
\text{Availability} = \frac{\text{requests with status} < 500}{\text{all requests}} \geq 99.9\% \text{ per 30 days}
$$

which allows about 43 minutes of full outage per 30 days.

### 7.2 Alerts

| Alert | Condition | Severity | Notifies | Runbook |
|---|---|---|---|---|
| *Example:* HighErrorRate | 5xx above 2% for 5 min | Page | On-call SRE | RB-01 |
| *Example:* DatabaseReplicaLag | Lag above 60 s for 10 min | Ticket | Platform team | RB-04 |
| *Example:* DiskAlmostFull | Data volume above 85% | Ticket | Platform team | RB-07 |
| *Example:* CertificateExpiry | TLS certificate expires in under 14 days | Ticket | Platform team | RB-09 |
| [Alert] | [Condition] | [Severity] | [Recipient] | [Runbook] |

### 7.3 Health checks

- `GET /healthz` returns `200` when the node can serve traffic.
- `GET /readyz` returns `200` when database and object store are reachable.
- Admin console **System > Status** shows queue depth and worker count.

## 8. Logs

<!-- Guidance: Say what each log contains, where it goes, how long it is kept and how to search it. Note any personal data in logs and how it is protected. -->

| Log | Content | Location | Retention | Contains personal data |
|---|---|---|---|---|
| *Example:* Application | Requests, errors, latency | [Central log platform], index `app-prod` | 30 days | User ID only (pseudonymous) |
| *Example:* Audit | Admin actions, sign-ins, permission changes | Write-once store | 1 year | Yes, access restricted to Auditor role |
| *Example:* Database | Slow queries over 500 ms | db01 `/var/log/postgresql` | 14 days | No |

Example search for failed sign-ins in the last hour:
`service:productname event:auth.failed | stats count by user_id`.

## 9. Maintenance schedule

<!-- Guidance: List recurring tasks with frequency and owner so nothing depends on memory. Automate what you can and say so. -->

| Task | Frequency | Owner | Automated |
|---|---|---|---|
| Review alerts and error budget | Daily | On-call SRE | Dashboard |
| Apply OS security patches | Weekly, Tuesday 03:00 | Platform team | Yes |
| Rotate service credentials | Every 90 days | Platform team | Partly |
| Restore test (6.3) | Monthly | Platform team | No |
| Access review (14.2) | Quarterly | Service owner | No |
| Renew TLS certificates | Automatic, checked monthly | Platform team | Yes |
| Capacity review (10) | Quarterly | SRE lead | No |

## 10. Scaling

<!-- Guidance: Describe when and how to add capacity, and the limits of each component. Give the signal that triggers scaling, not only the procedure. -->

| Component | Scale trigger | How to scale | Known limit |
|---|---|---|---|
| *Example:* App nodes | CPU above 70% for 15 min at peak | Add a node to the pool; autoscaling group max 8 | Database connections |
| *Example:* Workers | Queue depth above 1,000 for 10 min | Increase `workers.count` | Object store request rate |
| *Example:* Database | CPU above 60% sustained or storage above 70% | Vertical resize in maintenance window | Largest instance class |

Connections needed: $\text{nodes} \times \text{pool\_size} \leq \text{max\_connections} - \text{reserve}$.

## 11. Security operations

<!-- Guidance: Cover vulnerability handling, secrets, certificates, and what an administrator does when they suspect a breach. Link to the incident response plan rather than duplicating it. -->

- **Vulnerabilities:** Critical findings are patched within [7] days, high within [30].
- **Secrets:** Stored in [secret manager]; never in configuration files or tickets.
- **Certificates:** Issued by [internal CA / ACME]; inventory in 15.3.
- **Suspected compromise:** Follow the [Incident Response Plan]. First steps:
  1. Preserve logs: export the audit log for the affected period.
  2. Revoke affected sessions: **Users > [user] > End all sessions**.
  3. Rotate exposed credentials and notify [@security-officer].

## 12. Upgrades and patching

<!-- Guidance: Define the release cadence you follow, the test path (staging first), and the rollback rule. -->

1. Read the release notes and check for breaking changes.
2. Upgrade staging and run the smoke tests for one business day.
3. Raise a change ticket with the rollback plan.
4. Upgrade production with a rolling restart (nodes one at a time).
5. Monitor error rate for 60 minutes. Roll back if 5xx exceeds 2%.

| Version | Staging date | Production date | Result |
|---|---|---|---|
| *Example:* 3.2.1 | YYYY-MM-DD | YYYY-MM-DD | Success |

## 13. Disaster recovery

<!-- Guidance: Summarise objectives and the recovery sequence, then link to the full DR plan. Administrators need the order of operations under pressure. -->

| Objective | Target |
|---|---|
| Recovery time objective (RTO) | [4 hours] |
| Recovery point objective (RPO) | [15 minutes] |

Recovery order: database, object store, app nodes, workers, DNS switch.
Full procedure: [Disaster Recovery Plan, section n]. Last DR test:
YYYY-MM-DD, result [Pass / Partial].

## 14. Audit and compliance

<!-- Guidance: Say which controls administrators operate and what evidence they keep. Auditors ask for evidence, not intent. This section records practice; it is a format, not legal advice. -->

### 14.1 Control evidence

| Control | Evidence | Where kept |
|---|---|---|
| *Example:* Access reviews quarterly | Signed review export | [Compliance folder] |
| *Example:* Backups restored monthly | Restore log (6.3) | This guide and ticket |
| *Example:* Changes approved | Change tickets linked to commits | [Ticket system] |

### 14.2 Access review

Each quarter, export **Users > Export**, ask each team lead to confirm
their users and roles, disable unconfirmed accounts and file the signed
export.

## 15. Reference

<!-- Guidance: Facts administrators look up during work: commands, paths, ports, contacts. -->

### 15.1 Admin commands

| Command | Purpose |
|---|---|
| `productname status` | Show node, database and queue status |
| `productname maintenance on` | Enable maintenance mode (users see a notice, writes are blocked) |
| `productname maintenance off` | Return to normal service |
| `productname backup create` | Take an on-demand backup |
| `productname support-bundle` | Collect diagnostics without secrets |

### 15.2 Important paths

| Path | Content |
|---|---|
| `/etc/productname/config.yaml` | Main configuration |
| `/var/lib/productname` | Local cache and temporary files |
| `/var/log/productname` | Local logs before shipping |

### 15.3 Contacts and escalation

| Situation | Contact |
|---|---|
| Production outage | On-call SRE via [paging tool] |
| Security event | [security@example.com], [@security-officer] |
| Vendor support | [support portal URL], contract [number] |

### 15.4 Glossary

| Term | Meaning in this guide |
|---|---|
| Break-glass account | Local super administrator account used only when SSO is unavailable; every use is reviewed |
| Maintenance mode | State in which the service shows a notice and rejects writes |
| Error budget | Allowed unreliability under the SLO, for example 0.1% of requests per 30 days |
| Rolling restart | Restarting nodes one at a time so the service stays available |
| [Term] | [Meaning] |

## Standard and licence note

Structure aligned with ISO/IEC/IEEE 26514:2022, *Systems and software
engineering: Design and development of information for users*, for
information given to administrators and operators (roles, tasks,
reference and troubleshooting information). This template is original
text written for this collection and reproduces no text from the
standard; obtain the standard from ISO, IEC or IEEE for its normative
requirements. Template licence: MIT.