# RESOURCES

Каталог источников для изучения. Сгруппировано по ролям. Приоритет — первоисточники.

## Видео (ядерные)

- [Hamel Husain — AI Evaluations Clearly Explained, 52 мин](https://www.youtube.com/watch?v=uiza7wp1KrE) — 100 трейсов, spreadsheet, binary, TPR/TNR. Реклама курса, но карта тем рабочая.
- [Chip Huyen × Pragmatic Engineer, 75 мин](https://www.youtube.com/watch?v=98o_L3jlixw) — AI Eng vs ML Eng, RAG vs FT, почему eval самая тяжёлая часть.
- [Gaurav Sen — Mastering RAG, 0:00–34:44](https://www.youtube.com/watch?v=r37RoPQBXNU) — indexing, rerank, query rewrite.
- [Aishwarya Srinivasan — Agent AI System Design, 27 мин](https://www.youtube.com/watch?v=mwN75EiGfCE) — production blocks, trace evals, approval gates.

## Видео (второй ряд)

- [DL.AI — Building and Evaluating Advanced RAG, ~2 ч](https://www.deeplearning.ai/short-courses/building-evaluating-advanced-rag/) — RAG triad руками.
- [DL.AI — Evaluating AI Agents, ~2.5 ч](https://www.deeplearning.ai/short-courses/evaluating-ai-agents/) — router/skill/trajectory evals.
- [How to Evaluate LLM Apps, ~1 ч](https://www.youtube.com/watch?v=-sL7QzDFW-4) — golden set, retrieval metrics.
- [Think Software — What is RAG, 20 мин](https://www.youtube.com/watch?v=8Qxvv3yHC24) — быстрый каркас.
- [30 RAG Interview Questions, только 1:09:58–1:22:55](https://www.youtube.com/watch?v=ID85S5ZjgTs) — RAGAS + production секции.

## Статьи (обязательные)

- [Hamel — Your AI Product Needs Evals](https://hamel.dev/blog/posts/evals/) — три уровня eval.
- [Hamel — Field Guide to Rapidly Improving AI Products](https://hamel.dev/blog/posts/field-guide/) — error analysis, trust.
- [Hamel — LLM Evals FAQ](https://hamel.dev/blog/posts/evals-faq/) — конспект курса, 30+ вопросов.
- [Hamel — Creating a LLM-as-a-Judge](https://hamel.dev/blog/posts/llm-judge/) — пошагово судья.
- [Anthropic — Building Effective Agents](https://www.anthropic.com/engineering/building-effective-agents) — workflow ≠ agent, паттерны, ACI.
- [Anthropic — Contextual Retrieval](https://www.anthropic.com/news/contextual-retrieval) — −49%/−67%, промпт контекстуализации.
- [Anthropic — Demystifying evals for AI agents](https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents) — task/trial/grader/transcript/outcome, pass@k vs pass^k.
- [Anthropic — How we built multi-agent research system](https://www.anthropic.com/engineering/built-multi-agent-research-system) — +90.2%, 15× токенов, game of telephone.
- [Cognition — Don't Build Multi-Agents](https://cognition.ai/blog/dont-build-multi-agents) — контекст-инжиниринг, принципы.
- [Anthropic — Effective harnesses for long-running agents](https://www.anthropic.com/engineering/effective-harnesses-for-long-running-agents) — initializer + coding agent.
- [Anthropic — Writing tools for agents](https://www.anthropic.com/engineering/writing-tools-for-agents) — ACI, tool evals.
- [OpenAI — A Practical Guide to Building Agents (PDF)](https://cdn.openai.com/business-guides-and-resources/a-practical-guide-to-building-agents.pdf) — деплой, HITL, guardrails.

## Статьи (глубина)

- [Eugene Yan — Evaluating LLM-Evaluators](https://eugeneyan.com/writing/llm-evaluators/) — direct/pairwise/reference, баги судей.
- [Jason Liu — Levels of Complexity: RAG](https://jxnl.github.io/blog/writing/2024/02/28/levels-of-complexity-rag-applications/) — что логировать.
- [Evidently — полный гайд по RAG evaluation](https://www.evidentlyai.com/llm-guide/rag-evaluation) — retrieval vs generation.
- [Pinecone — Chunking Strategies](https://www.pinecone.io/learn/chunking-strategies/) — trade-off размера.
- [Cookbook: Contextual Retrieval](https://platform.claude.com/cookbook/capabilities-contextual-embeddings-guide) — рабочий код, Pass@10 87→95%.
- [Anthropic — Prompt Caching](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching) — cache_control.

## Papers

- [MT-Bench / Judging LLM-as-a-Judge (arXiv 2306.05685)](https://arxiv.org/abs/2306.05685) — баги судей, >80% согласия.
- [Who Validates the Validators? (arXiv 2404.12272)](https://arxiv.org/abs/2404.12272) — criteria drift.
- [MemGPT (arXiv 2310.08560)](https://arxiv.org/abs/2310.08560) — контекст как виртуальная память.
- [HyDE (arXiv 2212.10496)](https://arxiv.org/abs/2212.10496) — гипотетические документы.
- [RAG, Lewis et al. (arXiv 2005.11401)](https://arxiv.org/abs/2005.11401) — термин RAG.
- [τ-bench (arXiv 2406.12045)](https://arxiv.org/abs/2406.12045) — pass^k, таксономия ошибок тулов.

## Доки инструментов

- [Ragas metrics](https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/) — faithfulness, context recall, relevancy.
- [Promptfoo](https://promptfoo.dev/docs/) — open-source eval CLI.
- [Braintrust](https://www.braintrust.dev/docs) — eval-first платформа.
- [LangSmith evaluation](https://docs.langchain.com/langsmith/evaluation) — трейсинг + eval.
- [Langfuse](https://langfuse.com/) — self-hosted observability.
- [Arize Phoenix](https://arize.com/docs/phoenix) — open-source трейсы.
- [MCP intro](https://modelcontextprotocol.io/docs/getting-started/intro) — «USB-C для AI».
- [Apache Tika](https://github.com/apache/tika/blob/main/README.md) — текст из 1000+ форматов.
- [Marimo](https://docs.marimo.io/) — реактивный Python-ноутбук.
- [PAIR Guidebook](https://pair.withgoogle.com/guidebook) — UX AI-продуктов.
- [OpenAI Agents SDK](https://openai.github.io/openai-agents-python/) — primitives, sessions, guardrails.

## Бесплатные серии

- [Hamel/Shreya — 17 писем по evals](https://ai.hamel.dev/eval-course) — по письму утром.

## Платное (не на эту неделю)

- [Maven — AI Evals for Engineers & PMs](https://maven.com/parlance-labs/evals) — лучший курс рынка, 4 недели.
- [Hamel — Mastering LLMs (open course)](https://hamel.dev/blog/posts/course/) — 40+ часов, справочник.