Markdown Editor
에디터 열기

AI 엔지니어링 키트

AI 기능을 처음부터 끝까지: PRD, ML 및 RAG 설계, 에이전트와 MCP 명세, 프롬프트, 데이터와 파인튜닝, LLM 평가, 레드팀, 모니터링.

1단계

문제 정의

  1. AI Feature PRD Template

    전체

    PRD for an AI-powered feature: why AI, failure and uncertainty UX, metrics, eval thresholds, cost per request, rollout

  2. ML System Design Document Template

    전체

    Problem framing, metrics, data, features, baselines, training, offline and A/B evaluation, serving, monitoring, rollout

2단계

구축

  1. RAG System Design Template (Retrieval-Augmented Generation)

    전체

    Design a RAG system: ingestion, chunking, embeddings, hybrid search, citations, evaluation, ACLs, security and cost

  2. AI Agent Specification Template

    전체

    Specify a tool-using AI agent: autonomy level, tools and permissions, guardrails, security, trajectory evals, cost

  3. MCP Server Specification Template (Model Context Protocol)

    표준

    Specify an MCP server (spec revision 2026-07-28): transport, tools, resources, prompts, OAuth, security, testing

    Model Context Protocol specification, revision 2026-07-28

  4. Prompt Specification Template (System Prompt)

    표준

    System prompt spec: purpose, model parameters, role, instructions, safety rules, output schema, examples, tools, tests

3단계

데이터

  1. Data Pipeline Specification Template

    전체

    Batch or streaming pipeline: sources, lineage, schedule, mapping, quality checks, backfills, volumes, runbook

  2. Data Annotation Guidelines Template

    표준

    Labelling guidelines: task, label taxonomy, edge cases, decision rules, gold sets, Cohen's kappa, training, privacy

  3. Datasheet for Datasets Template (Gebru et al. 2021)

    전체

    Dataset motivation, composition, collection, labelling, uses, distribution, maintenance, privacy and consent

    Datasheets for Datasets (Gebru et al., 2021)

  4. Fine-Tuning Run Report Template

    표준

    One fine-tuning run: base model licence, data, LoRA or full method, hyperparameters, eval vs baseline, safety, decision

4단계

평가와 안전

  1. LLM Evaluation Plan Template

    전체

    LLM app evaluation: test sets and red team, metrics, LLM-as-judge calibration, human review, CIs, release gates

  2. AI Red Team Plan and Report Template (OWASP LLM Top 10 2025)

    전체

    Adversarial testing of an LLM app: threat model, OWASP LLM01-LLM10 and ATLAS mapping, test cases, findings, retest, sign-off

    OWASP Top 10 for LLM Applications 2025; MITRE ATLAS (tactic names)

  3. Model Card Template (Mitchell et al. 2019)

    전체

    Model details, intended and out-of-scope uses, factors, metrics, data, disaggregated results, ethics, caveats

    Model Cards for Model Reporting (Mitchell et al., 2019)

  4. AI System Card Template

    전체

    System-level card for a deployed AI product: components, uses, limitations, safety and bias evals, red-teaming

5단계

운영

  1. Model Monitoring Plan Template (MLOps)

    표준

    Production ML monitoring: performance, drift (PSI), data quality, fairness, latency, alerts, retraining triggers

  2. AI Incident Report Template

    표준

    Report an AI incident or near miss: harm, severity, timeline, root cause, containment, notifications incl. AI Act Art. 73

    OECD AI incident concepts (2024); EU AI Act Article 73