Markdown Editor
Editor öffnen

Kit KI-Engineering

KI-Funktionen durchgängig bauen: PRD, ML- und RAG-Entwurf, Agenten- und MCP-Spezifikation, Prompts, Daten und Fine-Tuning, LLM-Evaluierung, Red Teaming.

Schritt 1

Problemdefinition

  1. AI Feature PRD Template

    Vollständig

    PRD for an AI-powered feature: why AI, failure and uncertainty UX, metrics, eval thresholds, cost per request, rollout

  2. ML System Design Document Template

    Vollständig

    Problem framing, metrics, data, features, baselines, training, offline and A/B evaluation, serving, monitoring, rollout

Schritt 2

Umsetzung

  1. RAG System Design Template (Retrieval-Augmented Generation)

    Vollständig

    Design a RAG system: ingestion, chunking, embeddings, hybrid search, citations, evaluation, ACLs, security and cost

  2. AI Agent Specification Template

    Vollständig

    Specify a tool-using AI agent: autonomy level, tools and permissions, guardrails, security, trajectory evals, cost

  3. MCP Server Specification Template (Model Context Protocol)

    Standard

    Specify an MCP server (spec revision 2026-07-28): transport, tools, resources, prompts, OAuth, security, testing

    Model Context Protocol specification, revision 2026-07-28

  4. Prompt Specification Template (System Prompt)

    Standard

    System prompt spec: purpose, model parameters, role, instructions, safety rules, output schema, examples, tools, tests

Schritt 3

Daten

  1. Data Pipeline Specification Template

    Vollständig

    Batch or streaming pipeline: sources, lineage, schedule, mapping, quality checks, backfills, volumes, runbook

  2. Data Annotation Guidelines Template

    Standard

    Labelling guidelines: task, label taxonomy, edge cases, decision rules, gold sets, Cohen's kappa, training, privacy

  3. Datasheet for Datasets Template (Gebru et al. 2021)

    Vollständig

    Dataset motivation, composition, collection, labelling, uses, distribution, maintenance, privacy and consent

    Datasheets for Datasets (Gebru et al., 2021)

  4. Fine-Tuning Run Report Template

    Standard

    One fine-tuning run: base model licence, data, LoRA or full method, hyperparameters, eval vs baseline, safety, decision

Schritt 4

Evaluierung und Sicherheit

  1. LLM Evaluation Plan Template

    Vollständig

    LLM app evaluation: test sets and red team, metrics, LLM-as-judge calibration, human review, CIs, release gates

  2. AI Red Team Plan and Report Template (OWASP LLM Top 10 2025)

    Vollständig

    Adversarial testing of an LLM app: threat model, OWASP LLM01-LLM10 and ATLAS mapping, test cases, findings, retest, sign-off

    OWASP Top 10 for LLM Applications 2025; MITRE ATLAS (tactic names)

  3. Model Card Template (Mitchell et al. 2019)

    Vollständig

    Model details, intended and out-of-scope uses, factors, metrics, data, disaggregated results, ethics, caveats

    Model Cards for Model Reporting (Mitchell et al., 2019)

  4. AI System Card Template

    Vollständig

    System-level card for a deployed AI product: components, uses, limitations, safety and bias evals, red-teaming

Schritt 5

Betrieb

  1. Model Monitoring Plan Template (MLOps)

    Standard

    Production ML monitoring: performance, drift (PSI), data quality, fairness, latency, alerts, retraining triggers

  2. AI Incident Report Template

    Standard

    Report an AI incident or near miss: harm, severity, timeline, root cause, containment, notifications incl. AI Act Art. 73

    OECD AI incident concepts (2024); EU AI Act Article 73