Markdown Editor
Відкрити редактор

Набір інженера LLM-застосунків

LLM-застосунки від початку до кінця: корпоративний RAG, context engineering, промпти, агенти, каталог інструментів, skills, MCP, guardrails, оцінювання, спостережуваність, вартість.

Крок 1

Проєктування

  1. Enterprise RAG Search System Design Template

    Повний

    Enterprise RAG search design: connectors, document and chunk ACL filtering, permission sync, hybrid retrieval, RRF, reranking, citation verification, tenancy

    OWASP LLM Top 10 2026 (LLM08, LLM09)

  2. RAG System Design Template (Retrieval-Augmented Generation)

    Повний

    Design a RAG system: ingestion, chunking, embeddings, hybrid search, citations, evaluation, ACLs, security and cost

  3. Context Engineering Specification Template

    Повний

    Context engineering spec: window budget, instructions, retrieval, memory, tool results, ordering, compaction, prompt caching, untrusted content, evals

    MCP 2026-07-28 (tool definitions); OWASP Top 10 for Agentic Applications 2026 (mapping)

  4. Prompt Specification Template (System Prompt)

    Стандарт

    System prompt spec: purpose, model parameters, role, instructions, safety rules, output schema, examples, tools, tests

Крок 2

Розробка

  1. AI Agent Specification Template

    Повний

    Specify a tool-using AI agent: autonomy level, tools and permissions, guardrails, security, trajectory evals, cost

  2. Agent Tool and Function Catalogue Template (MCP)

    Повний

    Agent tool catalogue: model description, JSON Schema in/out, MCP annotations, side-effect class, idempotency, auth scope, rate limits, owner, tests

    MCP 2026-07-28 (Tool, ToolAnnotations); JSON Schema 2020-12

  3. Agent Skill Specification Template (SKILL.md)

    Повний

    Agent Skill spec: activation description, SKILL.md frontmatter, layout, scripts, progressive disclosure budget, versioning, security review, evals

    Agent Skills specification (agentskills.io, checked 2026-10-22)

  4. MCP Server Specification Template (Model Context Protocol)

    Стандарт

    Specify an MCP server (spec revision 2026-07-28): transport, tools, resources, prompts, OAuth, security, testing

    Model Context Protocol specification, revision 2026-07-28

  5. Guardrails and Safety Filter Specification Template (OWASP LLM Top 10 2026)

    Повний

    Policy taxonomy, input, output and tool-call guards, thresholds and actions, false-positive and latency budgets, guard evaluation, OWASP LLM 2026 mapping

    OWASP Top 10 for LLM Applications 2026; OWASP Top 10 for Agentic Applications 2026
Крок 3

Верифікація

  1. LLM Evaluation Plan Template

    Повний

    LLM app evaluation: test sets and red team, metrics, LLM-as-judge calibration, human review, CIs, release gates

  2. AI Agent Evaluation Plan Template (Task Success, Trajectory, Tool Calls)

    Повний

    Agent task suite, success, trajectory, tool-call and argument metrics, sandboxed harness, online tests, calibrated graders, statistics, ASI safety cases, gates

    OWASP Top 10 for Agentic Applications 2026 (adversarial cases)

  3. Model Benchmark Report Template

    Повний

    Benchmark choice and validity, contamination checks, protocol, prompts and decoding, repeated runs, confidence intervals, McNemar test, comparison table

    Practice from HELM (TMLR 2023), Miller 2024 and Sainz et al. 2023
Крок 4

Експлуатація

  1. LLM and Agent Observability Specification Template (OpenTelemetry GenAI)

    Повний

    LLM/agent telemetry: objectives, span model, gen_ai.* attributes, token/latency/cost metrics, prompt capture controls, eval hooks, dashboards, SLOs, retention

    OpenTelemetry semantic conventions for generative AI (Development)

  2. LLM Inference Cost Model Template

    Повний

    Cost per request and task, tokens in and out, caching and routing scenarios, self-hosted GPU vs API break-even, utilisation, sensitivity table, cost guardrails

  3. LLM and Agent Incident Postmortem Template

    Повний

    LLM and agent postmortem: impact, harmful output, data exposure, timeline, trace replay, eval regression, AI failure taxonomy, actions, notification check

    NIST SP 800-61 Rev. 3; EU AI Act Art. 73; GDPR Art. 33-34; NIST AI 100-2 E2025