Markdown Editor
Open the editor

AI Engineering Kit

Build AI features end to end: PRD, ML and RAG design, agent and MCP specs, prompts, data and fine-tuning, LLM evaluation, red teaming and monitoring.

Step 1

Framing

  1. AI Feature PRD Template

    Full

    PRD for an AI-powered feature: why AI, failure and uncertainty UX, metrics, eval thresholds, cost per request, rollout

  2. ML System Design Document Template

    Full

    Problem framing, metrics, data, features, baselines, training, offline and A/B evaluation, serving, monitoring, rollout

Step 2

Build

  1. RAG System Design Template (Retrieval-Augmented Generation)

    Full

    Design a RAG system: ingestion, chunking, embeddings, hybrid search, citations, evaluation, ACLs, security and cost

  2. AI Agent Specification Template

    Full

    Specify a tool-using AI agent: autonomy level, tools and permissions, guardrails, security, trajectory evals, cost

  3. MCP Server Specification Template (Model Context Protocol)

    Standard

    Specify an MCP server (spec revision 2026-07-28): transport, tools, resources, prompts, OAuth, security, testing

    Model Context Protocol specification, revision 2026-07-28

  4. Prompt Specification Template (System Prompt)

    Standard

    System prompt spec: purpose, model parameters, role, instructions, safety rules, output schema, examples, tools, tests

Step 3

Data

  1. Data Pipeline Specification Template

    Full

    Batch or streaming pipeline: sources, lineage, schedule, mapping, quality checks, backfills, volumes, runbook

  2. Data Annotation Guidelines Template

    Standard

    Labelling guidelines: task, label taxonomy, edge cases, decision rules, gold sets, Cohen's kappa, training, privacy

  3. Datasheet for Datasets Template (Gebru et al. 2021)

    Full

    Dataset motivation, composition, collection, labelling, uses, distribution, maintenance, privacy and consent

    Datasheets for Datasets (Gebru et al., 2021)

  4. Fine-Tuning Run Report Template

    Standard

    One fine-tuning run: base model licence, data, LoRA or full method, hyperparameters, eval vs baseline, safety, decision

Step 4

Evaluation and safety

  1. LLM Evaluation Plan Template

    Full

    LLM app evaluation: test sets and red team, metrics, LLM-as-judge calibration, human review, CIs, release gates

  2. AI Red Team Plan and Report Template (OWASP LLM Top 10 2025)

    Full

    Adversarial testing of an LLM app: threat model, OWASP LLM01-LLM10 and ATLAS mapping, test cases, findings, retest, sign-off

    OWASP Top 10 for LLM Applications 2025; MITRE ATLAS (tactic names)

  3. Model Card Template (Mitchell et al. 2019)

    Full

    Model details, intended and out-of-scope uses, factors, metrics, data, disaggregated results, ethics, caveats

    Model Cards for Model Reporting (Mitchell et al., 2019)

  4. AI System Card Template

    Full

    System-level card for a deployed AI product: components, uses, limitations, safety and bias evals, red-teaming

Step 5

Operations

  1. Model Monitoring Plan Template (MLOps)

    Standard

    Production ML monitoring: performance, drift (PSI), data quality, fairness, latency, alerts, retraining triggers

  2. AI Incident Report Template

    Standard

    Report an AI incident or near miss: harm, severity, timeline, root cause, containment, notifications incl. AI Act Art. 73

    OECD AI incident concepts (2024); EU AI Act Article 73