LLM 애플리케이션 엔지니어 키트

Enterprise RAG search design: connectors, document and chunk ACL filtering, permission sync, hybrid retrieval, RRF, reranking, citation verification, tenancy

Design a RAG system: ingestion, chunking, embeddings, hybrid search, citations, evaluation, ACLs, security and cost

Context engineering spec: window budget, instructions, retrieval, memory, tool results, ordering, compaction, prompt caching, untrusted content, evals

System prompt spec: purpose, model parameters, role, instructions, safety rules, output schema, examples, tools, tests

Specify a tool-using AI agent: autonomy level, tools and permissions, guardrails, security, trajectory evals, cost

Agent tool catalogue: model description, JSON Schema in/out, MCP annotations, side-effect class, idempotency, auth scope, rate limits, owner, tests

Agent Skill spec: activation description, SKILL.md frontmatter, layout, scripts, progressive disclosure budget, versioning, security review, evals

Specify an MCP server (spec revision 2026-07-28): transport, tools, resources, prompts, OAuth, security, testing

Policy taxonomy, input, output and tool-call guards, thresholds and actions, false-positive and latency budgets, guard evaluation, OWASP LLM 2026 mapping

LLM app evaluation: test sets and red team, metrics, LLM-as-judge calibration, human review, CIs, release gates

Agent task suite, success, trajectory, tool-call and argument metrics, sandboxed harness, online tests, calibrated graders, statistics, ASI safety cases, gates

Benchmark choice and validity, contamination checks, protocol, prompts and decoding, repeated runs, confidence intervals, McNemar test, comparison table

LLM/agent telemetry: objectives, span model, gen_ai.* attributes, token/latency/cost metrics, prompt capture controls, eval hooks, dashboards, SLOs, retention

Cost per request and task, tokens in and out, caching and routing scenarios, self-hosted GPU vs API break-even, utilisation, sensitivity table, cost guardrails

LLM and agent postmortem: impact, harmful output, data exposure, timeline, trace replay, eval regression, AI failure taxonomy, actions, notification check