Markdown Editor
Ouvrir l'éditeur

Kit ingénieur applications LLM

Construire des applications LLM : RAG d'entreprise, ingénierie du contexte, prompts, agents, catalogue d'outils, skills, MCP, garde-fous, évaluation, observabilité, coûts.

Étape 1

Conception

  1. Enterprise RAG Search System Design Template

    Complet

    Enterprise RAG search design: connectors, document and chunk ACL filtering, permission sync, hybrid retrieval, RRF, reranking, citation verification, tenancy

    OWASP LLM Top 10 2026 (LLM08, LLM09)

  2. RAG System Design Template (Retrieval-Augmented Generation)

    Complet

    Design a RAG system: ingestion, chunking, embeddings, hybrid search, citations, evaluation, ACLs, security and cost

  3. Context Engineering Specification Template

    Complet

    Context engineering spec: window budget, instructions, retrieval, memory, tool results, ordering, compaction, prompt caching, untrusted content, evals

    MCP 2026-07-28 (tool definitions); OWASP Top 10 for Agentic Applications 2026 (mapping)

  4. Prompt Specification Template (System Prompt)

    Standard

    System prompt spec: purpose, model parameters, role, instructions, safety rules, output schema, examples, tools, tests

Étape 2

Construction

  1. AI Agent Specification Template

    Complet

    Specify a tool-using AI agent: autonomy level, tools and permissions, guardrails, security, trajectory evals, cost

  2. Agent Tool and Function Catalogue Template (MCP)

    Complet

    Agent tool catalogue: model description, JSON Schema in/out, MCP annotations, side-effect class, idempotency, auth scope, rate limits, owner, tests

    MCP 2026-07-28 (Tool, ToolAnnotations); JSON Schema 2020-12

  3. Agent Skill Specification Template (SKILL.md)

    Complet

    Agent Skill spec: activation description, SKILL.md frontmatter, layout, scripts, progressive disclosure budget, versioning, security review, evals

    Agent Skills specification (agentskills.io, checked 2026-10-22)

  4. MCP Server Specification Template (Model Context Protocol)

    Standard

    Specify an MCP server (spec revision 2026-07-28): transport, tools, resources, prompts, OAuth, security, testing

    Model Context Protocol specification, revision 2026-07-28

  5. Guardrails and Safety Filter Specification Template (OWASP LLM Top 10 2026)

    Complet

    Policy taxonomy, input, output and tool-call guards, thresholds and actions, false-positive and latency budgets, guard evaluation, OWASP LLM 2026 mapping

    OWASP Top 10 for LLM Applications 2026; OWASP Top 10 for Agentic Applications 2026
Étape 3

Vérification

  1. LLM Evaluation Plan Template

    Complet

    LLM app evaluation: test sets and red team, metrics, LLM-as-judge calibration, human review, CIs, release gates

  2. AI Agent Evaluation Plan Template (Task Success, Trajectory, Tool Calls)

    Complet

    Agent task suite, success, trajectory, tool-call and argument metrics, sandboxed harness, online tests, calibrated graders, statistics, ASI safety cases, gates

    OWASP Top 10 for Agentic Applications 2026 (adversarial cases)

  3. Model Benchmark Report Template

    Complet

    Benchmark choice and validity, contamination checks, protocol, prompts and decoding, repeated runs, confidence intervals, McNemar test, comparison table

    Practice from HELM (TMLR 2023), Miller 2024 and Sainz et al. 2023
Étape 4

Exploitation

  1. LLM and Agent Observability Specification Template (OpenTelemetry GenAI)

    Complet

    LLM/agent telemetry: objectives, span model, gen_ai.* attributes, token/latency/cost metrics, prompt capture controls, eval hooks, dashboards, SLOs, retention

    OpenTelemetry semantic conventions for generative AI (Development)

  2. LLM Inference Cost Model Template

    Complet

    Cost per request and task, tokens in and out, caching and routing scenarios, self-hosted GPU vs API break-even, utilisation, sensitivity table, cost guardrails

  3. LLM and Agent Incident Postmortem Template

    Complet

    LLM and agent postmortem: impact, harmful output, data exposure, timeline, trace replay, eval regression, AI failure taxonomy, actions, notification check

    NIST SP 800-61 Rev. 3; EU AI Act Art. 73; GDPR Art. 33-34; NIST AI 100-2 E2025