Markdown Editor
Open the editor

LLM Application Engineer Kit

Build LLM apps end to end: enterprise RAG, context engineering, prompts, agents, tool catalogue, agent skills, MCP, guardrails, evaluation, observability and cost.

Step 1

Design

  1. Enterprise RAG Search System Design Template

    Full

    Enterprise RAG search design: connectors, document and chunk ACL filtering, permission sync, hybrid retrieval, RRF, reranking, citation verification, tenancy

    OWASP LLM Top 10 2026 (LLM08, LLM09)

  2. RAG System Design Template (Retrieval-Augmented Generation)

    Full

    Design a RAG system: ingestion, chunking, embeddings, hybrid search, citations, evaluation, ACLs, security and cost

  3. Context Engineering Specification Template

    Full

    Context engineering spec: window budget, instructions, retrieval, memory, tool results, ordering, compaction, prompt caching, untrusted content, evals

    MCP 2026-07-28 (tool definitions); OWASP Top 10 for Agentic Applications 2026 (mapping)

  4. Prompt Specification Template (System Prompt)

    Standard

    System prompt spec: purpose, model parameters, role, instructions, safety rules, output schema, examples, tools, tests

Step 2

Build

  1. AI Agent Specification Template

    Full

    Specify a tool-using AI agent: autonomy level, tools and permissions, guardrails, security, trajectory evals, cost

  2. Agent Tool and Function Catalogue Template (MCP)

    Full

    Agent tool catalogue: model description, JSON Schema in/out, MCP annotations, side-effect class, idempotency, auth scope, rate limits, owner, tests

    MCP 2026-07-28 (Tool, ToolAnnotations); JSON Schema 2020-12

  3. Agent Skill Specification Template (SKILL.md)

    Full

    Agent Skill spec: activation description, SKILL.md frontmatter, layout, scripts, progressive disclosure budget, versioning, security review, evals

    Agent Skills specification (agentskills.io, checked 2026-10-22)

  4. MCP Server Specification Template (Model Context Protocol)

    Standard

    Specify an MCP server (spec revision 2026-07-28): transport, tools, resources, prompts, OAuth, security, testing

    Model Context Protocol specification, revision 2026-07-28

  5. Guardrails and Safety Filter Specification Template (OWASP LLM Top 10 2026)

    Full

    Policy taxonomy, input, output and tool-call guards, thresholds and actions, false-positive and latency budgets, guard evaluation, OWASP LLM 2026 mapping

    OWASP Top 10 for LLM Applications 2026; OWASP Top 10 for Agentic Applications 2026
Step 3

Verification

  1. LLM Evaluation Plan Template

    Full

    LLM app evaluation: test sets and red team, metrics, LLM-as-judge calibration, human review, CIs, release gates

  2. AI Agent Evaluation Plan Template (Task Success, Trajectory, Tool Calls)

    Full

    Agent task suite, success, trajectory, tool-call and argument metrics, sandboxed harness, online tests, calibrated graders, statistics, ASI safety cases, gates

    OWASP Top 10 for Agentic Applications 2026 (adversarial cases)

  3. Model Benchmark Report Template

    Full

    Benchmark choice and validity, contamination checks, protocol, prompts and decoding, repeated runs, confidence intervals, McNemar test, comparison table

    Practice from HELM (TMLR 2023), Miller 2024 and Sainz et al. 2023
Step 4

Operations

  1. LLM and Agent Observability Specification Template (OpenTelemetry GenAI)

    Full

    LLM/agent telemetry: objectives, span model, gen_ai.* attributes, token/latency/cost metrics, prompt capture controls, eval hooks, dashboards, SLOs, retention

    OpenTelemetry semantic conventions for generative AI (Development)

  2. LLM Inference Cost Model Template

    Full

    Cost per request and task, tokens in and out, caching and routing scenarios, self-hosted GPU vs API break-even, utilisation, sensitivity table, cost guardrails

  3. LLM and Agent Incident Postmortem Template

    Full

    LLM and agent postmortem: impact, harmful output, data exposure, timeline, trace replay, eval regression, AI failure taxonomy, actions, notification check

    NIST SP 800-61 Rev. 3; EU AI Act Art. 73; GDPR Art. 33-34; NIST AI 100-2 E2025