Markdown Editor
エディターを開く

LLMアプリケーションエンジニアキット

LLMアプリを一貫して構築：企業RAG、コンテキストエンジニアリング、プロンプト、エージェント、ツールカタログ、エージェントスキル、MCP、ガードレール、評価、可観測性、コスト。

ステップ 1

設計

  1. Enterprise RAG Search System Design Template

    フル

    Enterprise RAG search design: connectors, document and chunk ACL filtering, permission sync, hybrid retrieval, RRF, reranking, citation verification, tenancy

    OWASP LLM Top 10 2026 (LLM08, LLM09)

  2. RAG System Design Template (Retrieval-Augmented Generation)

    フル

    Design a RAG system: ingestion, chunking, embeddings, hybrid search, citations, evaluation, ACLs, security and cost

  3. Context Engineering Specification Template

    フル

    Context engineering spec: window budget, instructions, retrieval, memory, tool results, ordering, compaction, prompt caching, untrusted content, evals

    MCP 2026-07-28 (tool definitions); OWASP Top 10 for Agentic Applications 2026 (mapping)

  4. Prompt Specification Template (System Prompt)

    標準

    System prompt spec: purpose, model parameters, role, instructions, safety rules, output schema, examples, tools, tests

ステップ 2

構築

  1. AI Agent Specification Template

    フル

    Specify a tool-using AI agent: autonomy level, tools and permissions, guardrails, security, trajectory evals, cost

  2. Agent Tool and Function Catalogue Template (MCP)

    フル

    Agent tool catalogue: model description, JSON Schema in/out, MCP annotations, side-effect class, idempotency, auth scope, rate limits, owner, tests

    MCP 2026-07-28 (Tool, ToolAnnotations); JSON Schema 2020-12

  3. Agent Skill Specification Template (SKILL.md)

    フル

    Agent Skill spec: activation description, SKILL.md frontmatter, layout, scripts, progressive disclosure budget, versioning, security review, evals

    Agent Skills specification (agentskills.io, checked 2026-10-22)

  4. MCP Server Specification Template (Model Context Protocol)

    標準

    Specify an MCP server (spec revision 2026-07-28): transport, tools, resources, prompts, OAuth, security, testing

    Model Context Protocol specification, revision 2026-07-28

  5. Guardrails and Safety Filter Specification Template (OWASP LLM Top 10 2026)

    フル

    Policy taxonomy, input, output and tool-call guards, thresholds and actions, false-positive and latency budgets, guard evaluation, OWASP LLM 2026 mapping

    OWASP Top 10 for LLM Applications 2026; OWASP Top 10 for Agentic Applications 2026
ステップ 3

検証

  1. LLM Evaluation Plan Template

    フル

    LLM app evaluation: test sets and red team, metrics, LLM-as-judge calibration, human review, CIs, release gates

  2. AI Agent Evaluation Plan Template (Task Success, Trajectory, Tool Calls)

    フル

    Agent task suite, success, trajectory, tool-call and argument metrics, sandboxed harness, online tests, calibrated graders, statistics, ASI safety cases, gates

    OWASP Top 10 for Agentic Applications 2026 (adversarial cases)

  3. Model Benchmark Report Template

    フル

    Benchmark choice and validity, contamination checks, protocol, prompts and decoding, repeated runs, confidence intervals, McNemar test, comparison table

    Practice from HELM (TMLR 2023), Miller 2024 and Sainz et al. 2023
ステップ 4

運用

  1. LLM and Agent Observability Specification Template (OpenTelemetry GenAI)

    フル

    LLM/agent telemetry: objectives, span model, gen_ai.* attributes, token/latency/cost metrics, prompt capture controls, eval hooks, dashboards, SLOs, retention

    OpenTelemetry semantic conventions for generative AI (Development)

  2. LLM Inference Cost Model Template

    フル

    Cost per request and task, tokens in and out, caching and routing scenarios, self-hosted GPU vs API break-even, utilisation, sensitivity table, cost guardrails

  3. LLM and Agent Incident Postmortem Template

    フル

    LLM and agent postmortem: impact, harmful output, data exposure, timeline, trace replay, eval regression, AI failure taxonomy, actions, notification check

    NIST SP 800-61 Rev. 3; EU AI Act Art. 73; GDPR Art. 33-34; NIST AI 100-2 E2025