Markdown Editor
Mở trình soạn thảo

Bộ mẫu kỹ sư ứng dụng LLM

Xây ứng dụng LLM trọn vẹn: RAG doanh nghiệp, context engineering, prompt, agent, danh mục công cụ, agent skill, MCP, guardrail, đánh giá, quan sát và chi phí.

Bước 1

Thiết kế

  1. Enterprise RAG Search System Design Template

    Đầy đủ

    Enterprise RAG search design: connectors, document and chunk ACL filtering, permission sync, hybrid retrieval, RRF, reranking, citation verification, tenancy

    OWASP LLM Top 10 2026 (LLM08, LLM09)

  2. RAG System Design Template (Retrieval-Augmented Generation)

    Đầy đủ

    Design a RAG system: ingestion, chunking, embeddings, hybrid search, citations, evaluation, ACLs, security and cost

  3. Context Engineering Specification Template

    Đầy đủ

    Context engineering spec: window budget, instructions, retrieval, memory, tool results, ordering, compaction, prompt caching, untrusted content, evals

    MCP 2026-07-28 (tool definitions); OWASP Top 10 for Agentic Applications 2026 (mapping)

  4. Prompt Specification Template (System Prompt)

    Chuẩn

    System prompt spec: purpose, model parameters, role, instructions, safety rules, output schema, examples, tools, tests

Bước 2

Xây dựng

  1. AI Agent Specification Template

    Đầy đủ

    Specify a tool-using AI agent: autonomy level, tools and permissions, guardrails, security, trajectory evals, cost

  2. Agent Tool and Function Catalogue Template (MCP)

    Đầy đủ

    Agent tool catalogue: model description, JSON Schema in/out, MCP annotations, side-effect class, idempotency, auth scope, rate limits, owner, tests

    MCP 2026-07-28 (Tool, ToolAnnotations); JSON Schema 2020-12

  3. Agent Skill Specification Template (SKILL.md)

    Đầy đủ

    Agent Skill spec: activation description, SKILL.md frontmatter, layout, scripts, progressive disclosure budget, versioning, security review, evals

    Agent Skills specification (agentskills.io, checked 2026-10-22)

  4. MCP Server Specification Template (Model Context Protocol)

    Chuẩn

    Specify an MCP server (spec revision 2026-07-28): transport, tools, resources, prompts, OAuth, security, testing

    Model Context Protocol specification, revision 2026-07-28

  5. Guardrails and Safety Filter Specification Template (OWASP LLM Top 10 2026)

    Đầy đủ

    Policy taxonomy, input, output and tool-call guards, thresholds and actions, false-positive and latency budgets, guard evaluation, OWASP LLM 2026 mapping

    OWASP Top 10 for LLM Applications 2026; OWASP Top 10 for Agentic Applications 2026
Bước 3

Kiểm chứng

  1. LLM Evaluation Plan Template

    Đầy đủ

    LLM app evaluation: test sets and red team, metrics, LLM-as-judge calibration, human review, CIs, release gates

  2. AI Agent Evaluation Plan Template (Task Success, Trajectory, Tool Calls)

    Đầy đủ

    Agent task suite, success, trajectory, tool-call and argument metrics, sandboxed harness, online tests, calibrated graders, statistics, ASI safety cases, gates

    OWASP Top 10 for Agentic Applications 2026 (adversarial cases)

  3. Model Benchmark Report Template

    Đầy đủ

    Benchmark choice and validity, contamination checks, protocol, prompts and decoding, repeated runs, confidence intervals, McNemar test, comparison table

    Practice from HELM (TMLR 2023), Miller 2024 and Sainz et al. 2023
Bước 4

Vận hành

  1. LLM and Agent Observability Specification Template (OpenTelemetry GenAI)

    Đầy đủ

    LLM/agent telemetry: objectives, span model, gen_ai.* attributes, token/latency/cost metrics, prompt capture controls, eval hooks, dashboards, SLOs, retention

    OpenTelemetry semantic conventions for generative AI (Development)

  2. LLM Inference Cost Model Template

    Đầy đủ

    Cost per request and task, tokens in and out, caching and routing scenarios, self-hosted GPU vs API break-even, utilisation, sensitivity table, cost guardrails

  3. LLM and Agent Incident Postmortem Template

    Đầy đủ

    LLM and agent postmortem: impact, harmful output, data exposure, timeline, trace replay, eval regression, AI failure taxonomy, actions, notification check

    NIST SP 800-61 Rev. 3; EU AI Act Art. 73; GDPR Art. 33-34; NIST AI 100-2 E2025