Kit rekayasa AI

PRD for an AI-powered feature: why AI, failure and uncertainty UX, metrics, eval thresholds, cost per request, rollout

Problem framing, metrics, data, features, baselines, training, offline and A/B evaluation, serving, monitoring, rollout

Design a RAG system: ingestion, chunking, embeddings, hybrid search, citations, evaluation, ACLs, security and cost

Specify a tool-using AI agent: autonomy level, tools and permissions, guardrails, security, trajectory evals, cost

Specify an MCP server (spec revision 2026-07-28): transport, tools, resources, prompts, OAuth, security, testing

System prompt spec: purpose, model parameters, role, instructions, safety rules, output schema, examples, tools, tests

Batch or streaming pipeline: sources, lineage, schedule, mapping, quality checks, backfills, volumes, runbook

Labelling guidelines: task, label taxonomy, edge cases, decision rules, gold sets, Cohen's kappa, training, privacy

Dataset motivation, composition, collection, labelling, uses, distribution, maintenance, privacy and consent

One fine-tuning run: base model licence, data, LoRA or full method, hyperparameters, eval vs baseline, safety, decision

LLM app evaluation: test sets and red team, metrics, LLM-as-judge calibration, human review, CIs, release gates

Adversarial testing of an LLM app: threat model, OWASP LLM01-LLM10 and ATLAS mapping, test cases, findings, retest, sign-off

Model details, intended and out-of-scope uses, factors, metrics, data, disaggregated results, ethics, caveats

System-level card for a deployed AI product: components, uses, limitations, safety and bias evals, red-teaming

Production ML monitoring: performance, drift (PSI), data quality, fairness, latency, alerts, retraining triggers

Report an AI incident or near miss: harm, severity, timeline, root cause, containment, notifications incl. AI Act Art. 73