AI Engineer Learning Map

Build, route, secure, evaluate, package, and verify Agent Skills in real hosts.

Prerequisites

• Files, Python, and the command line — You can edit a text file, inspect a directory, and run a Python command. No agent framework experience is required.
• Node.js, npx, and Python 3 for real labs — The install and bundled-script checkpoints require working node, npx, and python3 commands. The website and docs remain a conceptual fallback when these commands are unavailable.
• One selected skill-capable host and install scope — Choose a supported agent host plus a writable project or user skill scope before installation. Record the exact host, scope, and installed path; do not infer discovery from installer success.
• The Tool Interface — Skim this first if tool schemas and side-effect boundaries are new to you.
• Tool Schema Design — Skim this first if you have not designed typed tool contracts.
• Tool poisoning and untrusted instructions — Before Lesson 26, explain why skill and tool metadata is untrusted input. Lesson 15 is an optional refresher outside this five-lesson route when this preflight is not yet confirmed.

Ordered lessons

1. 22Agent Skills: Portable Contract and Runtime Boundary90 min
2. 24Skill Discovery and Progressive Disclosure105 min
3. 25Skill Invocation and Routing105 min
4. 26Skill Permissions, Sandboxes, and Trust120 min
5. 27Skill Evals, Packaging, and Portability150 min

Engineer tool-using agent loops with explicit context, memory, orchestration, safety, evaluation, and production control.

What the role does

• Design tool contracts and close the thought, action, observation loop.
• Control context, memory, state transitions, and orchestration.
• Model failures and defend tool and instruction boundaries.
• Evaluate behavior and operate the runtime from traces and controls.

Good fit if

• You enjoy reasoning about stateful systems, control flow, and failure recovery.
• You want autonomy to be bounded by explicit contracts and evidence.
• You prefer testable orchestration over opaque agent demos.

Common Core

Build a typed tool surface and a bounded agent loop.

1. 01The Tool Interface — Why Agents Need Structured I/O
2. 05Tool Schema Design — Naming, Descriptions, Parameter Constraints
3. 06MCP Fundamentals: Stateless Requests and JSON-RPC
4. 01The Agent Loop: Observe, Think, Act
5. 06Tool Use and Function Calling

A deterministic agent loop with typed tools, validation, stop conditions, and a transcript.

Role Practice

Make context, memory, state, and orchestration explicit and testable.

1. 05Context Engineering: Windows, Budgets, Memory, and Retrieval
2. 07Agent Memory — Virtual Context and Memory Paging
3. 13Stateful Graph Orchestration — Durable Execution and Checkpoints
4. 28Orchestration Patterns: Supervisor, Swarm, Hierarchical

A stateful workflow with a context budget, memory policy, checkpoints, and recovery paths.

Proof Project

Stress the system against realistic failures and unsafe inputs before release.

1. 26Failure Modes: Why Agents Break
2. 27Prompt Injection and the PVE Defense
3. 30Eval-Driven Agent Development

An observable agent system with adversarial tests, trajectory evaluations, and explicit release gates.

Interview and Readiness Evidence

Show that the runtime can be operated from controls and trace evidence.

1. 29Production Runtimes: Queue, Event, Cron
2. 24Agent Observability: Langfuse, Phoenix, Opik

A case study with architecture, failure evidence, runtime controls, traces, and justified tradeoffs.

Build reliable data, feature, embedding, retrieval, evaluation, and observability pipelines for AI systems.

What the role does

• Design reproducible ingestion, transformation, and feature pipelines.
• Build embedding and retrieval paths with explicit quality measures.
• Detect leakage, drift, missing data, and retrieval regressions.
• Instrument data-dependent behavior from source through production.

Good fit if

• You care about lineage, repeatability, and data quality more than demos.
• You enjoy tracing model behavior back to data and retrieval decisions.
• You want to connect data systems with measurable AI outcomes.

Common Core

Build reproducible data, feature, and language-model preparation pipelines.

1. 09Data Management
2. 08Feature Engineering & Selection
3. 13ML Pipelines
4. 03Data Pipelines for Pre-Training

A versioned pipeline with deterministic transforms, split checks, and a reproducible run record.

Role Practice

Build context and retrieval paths whose quality can be measured independently.

1. 04Embeddings & Vector Representations
2. 05Context Engineering: Windows, Budgets, Memory, and Retrieval
3. 06RAG (Retrieval-Augmented Generation)
4. 07Advanced RAG (Chunking, Reranking, Hybrid Search)

A retrieval system with a documented corpus, index configuration, and retrieval metrics.

Proof Project

Connect the data path to an evaluated production-style application.

1. 10Evaluation & Testing LLM Applications
2. 13Building a Production LLM Application

A traceable AI data pipeline with end-to-end quality checks and application-level evaluation.

Interview and Readiness Evidence

Show how telemetry separates data, retrieval, and generation failures.

1. 13LLM Observability Stack Selection

A failure analysis that links a production symptom to the responsible data or retrieval stage.

Build credible integrations, examples, and reusable agent packages, then turn developer friction into clearer tools and teaching.

What the role does

• Design clear API, tool, and protocol contracts for developer use.
• Build runnable examples and reusable agent packages.
• Teach runtime behavior, tradeoffs, and failure modes accurately.
• Turn developer friction and feedback into product and documentation improvements.

Good fit if

• You enjoy building software and explaining why it works.
• You care about first-run success, examples, documentation, and developer trust.
• You want developer feedback to shape the product.

Common Core

Build a sound capability interface from credentials through tool and protocol contracts.

1. 04APIs & Keys
2. 01The Tool Interface — Why Agents Need Structured I/O
3. 05Tool Schema Design — Naming, Descriptions, Parameter Constraints
4. 06MCP Fundamentals: Stateless Requests and JSON-RPC
5. 07Building an MCP Server: Stateless Python and TypeScript

A runnable integration with a clear contract, setup path, and observable error behavior.

Role Practice

Make reusable agent knowledge discoverable, invocable, and progressively disclosed.

1. 22Agent Skills: Portable Contract and Runtime Boundary
2. 24Skill Discovery and Progressive Disclosure
3. 25Skill Invocation and Routing

A skill package with concise metadata, staged references, and explicit invocation examples.

Proof Project

Prove that the package and its surrounding tool ecosystem work across supported contexts.

1. 27Skill Evals, Packaging, and Portability
2. 23Capstone: Stateless Tool Ecosystem

A developer adoption kit with validation results, portability notes, and a working end-to-end example.

Interview and Readiness Evidence

Show how developer feedback changed the integration or its teaching path.

1. 54Build a Feedback Ratchet with Ownership and Retirement

A short adoption case study with friction evidence, the change made, and the measured result.

Measure model and agent behavior, expose failure modes, instrument the runtime, and build release and incident controls around evidence.

What the role does

• Design model, LLM, and agent evaluations around real failure risks.
• Instrument inference behavior and system health with useful telemetry.
• Test load, experiments, canaries, and rollback conditions before release.
• Use incidents and chaos tests to improve reliability controls.

Good fit if

• You prefer measurable evidence to confident completion claims.
• You enjoy finding edge cases, regressions, and system failure paths.
• You want evaluation and operations to share the same release evidence.

Common Core

Define behavior, evaluation layers, failure modes, and trace semantics.

1. 09Model Evaluation
2. 10Evaluation & Testing LLM Applications
3. 30Eval-Driven Agent Development
4. 26Failure Modes: Why Agents Break
5. 23OpenTelemetry GenAI Semantic Conventions

A layered evaluation plan with a failure taxonomy, trace schema, and regression thresholds.

Role Practice

Measure inference health, production behavior, experiments, and capacity limits.

1. 08Inference Metrics — TTFT, TPOT, ITL, Goodput, P99
2. 13LLM Observability Stack Selection
3. 21A/B Testing LLM Features — GrowthBook, Statsig, and the Vibes Problem
4. 22Load Testing LLM APIs — Why k6 and Locust Lie

An observability and load report that connects service metrics with behavior and user outcomes.

Proof Project

Prove release and recovery behavior under controlled failure.

1. 20Shadow Traffic, Canary Rollout, and Progressive Deployment for LLMs
2. 24Chaos Engineering for LLM Production

A regression and reliability gate with canary thresholds, rollback evidence, and a chaos result.

Interview and Readiness Evidence

Show how objectives, incidents, and corrective actions form an operating system for reliability.

1. 23SRE for AI — Multi-Agent Incident Response, Runbooks, Predictive Detection

A reliability case study with SLOs, failure evidence, response decisions, and verified corrective actions.

Turn language-model capability into evaluated, safe, cost-aware product behavior that can survive production traffic.

What the role does

• Shape model behavior with prompts, schemas, context, retrieval, and tools.
• Define evaluation sets and guardrails around product requirements.
• Balance quality, latency, cost, and operational risk.
• Ship through production interfaces and controlled rollout stages.

Good fit if

• You want to connect model behavior directly to a user-facing product.
• You enjoy balancing quality, safety, latency, and cost.
• You prefer measured product behavior over isolated model demos.

Common Core

Shape language-model behavior through prompts, schemas, context, retrieval, and tools.

1. 01Prompt Engineering: Techniques & Patterns
2. 03Structured Outputs: JSON, Schema Validation, Constrained Decoding
3. 04Embeddings & Vector Representations
4. 05Context Engineering: Windows, Budgets, Memory, and Retrieval
5. 06RAG (Retrieval-Augmented Generation)
6. 09Function Calling & Tool Use

A grounded feature prototype with a typed user contract and observable tool or retrieval behavior.

Role Practice

Measure quality and control cost and safety before production exposure.

1. 10Evaluation & Testing LLM Applications
2. 11Caching, Rate Limiting & Cost Optimization
3. 12Guardrails, Safety & Content Filtering

An evaluation suite with guardrail tests and a quality, latency, and cost scorecard.

Proof Project

Package the feature as a production-style application behind operational controls.

1. 13Building a Production LLM Application
2. 19AI Gateways — LiteLLM, Portkey, Kong AI Gateway, Bifrost

A deployed LLM feature with routing, limits, observability hooks, and documented failure behavior.

Interview and Readiness Evidence

Show how the feature advances through controlled exposure with rollback evidence.

1. 20Shadow Traffic, Canary Rollout, and Progressive Deployment for LLMs

A product case study with eval results, cost tradeoffs, release gates, and a rollback decision.

Build an AI feature from prompt and structured output through retrieval, evaluation, production serving, observability, and release.

Ordered lessons

1. 01Prompt Engineering: Techniques & Patterns60 min
2. 03Structured Outputs: JSON, Schema Validation, Constrained Decoding60 min
3. 04Embeddings & Vector Representations60 min
4. 05Context Engineering: Windows, Budgets, Memory, and Retrieval60 min
5. 06RAG (Retrieval-Augmented Generation)75 min
6. 10Evaluation & Testing LLM Applications75 min
7. 11Caching, Rate Limiting & Cost Optimization60 min
8. 12Guardrails, Safety & Content Filtering60 min
9. 13Building a Production LLM Application90 min
10. 13LLM Observability Stack Selection60 min
11. 20Shadow Traffic, Canary Rollout, and Progressive Deployment for LLMs60 min
12. 23SRE for AI — Multi-Agent Incident Response, Runbooks, Predictive Detection60 min

Discover a customer workflow, reduce its riskiest assumptions, and carry a useful AI system through measurement and rollout.

What the role does

• Observe the real workflow and define the outcome with users.
• Map assumptions, constraints, dependencies, and failure risks.
• Build the smallest vertical slice that can test the core claim.
• Define success metrics, rollout controls, and feedback loops.

Good fit if

• You want to work close to users and ambiguous operating environments.
• You enjoy moving between discovery, implementation, evaluation, and rollout.
• You can explain tradeoffs to both technical and operational stakeholders.

Common Core

Frame the customer outcome, workflow, assumptions, and smallest useful test.

1. 47Define the Outcome Before You Choose the Output
2. 48Discover the Workflow People Actually Perform
3. 49Map Assumptions and Resolve the Riskiest One First
4. 50Choose the Smallest Slice That Can Change the Decision
5. 51Write Specifications That Preserve Judgment

A workflow brief with outcome, evidence, assumptions, risks, and an executable slice specification.

Role Practice

Build and evaluate the narrow AI system against explicit success criteria.

1. 06RAG (Retrieval-Augmented Generation)
2. 10Evaluation & Testing LLM Applications
3. 13Building a Production LLM Application
4. 52Design Success Metrics Before the Result Exists

A working vertical slice with an evaluation set, baseline, and success scorecard.

Proof Project

Choose the right release stage and move the pilot through controlled exposure.

1. 53Choose Prototype, Pilot, or Production Deliberately
2. 20Shadow Traffic, Canary Rollout, and Progressive Deployment for LLMs

A measured pilot with release gates, rollback conditions, and recorded results.

Interview and Readiness Evidence

Show how evidence and user feedback determine the next iteration.

1. 54Build a Feedback Ratchet with Ownership and Retirement

A concise case study that connects user evidence, system results, tradeoffs, and the next decision.

Build, secure, verify, and operate stateless MCP systems from JSON-RPC envelopes through conformance release gates.

Prerequisites

• Python and command-line basics — You can inspect JSON, edit a file, and run a Python command. Prior MCP or agent-framework experience is not required.
• Python 3 for executable wire evidence — The deterministic implementations and tests use Python 3. A TypeScript runner is optional for the second Lesson 07 implementation.
• The Tool Interface — Skim this lesson if typed tool calls and side-effect boundaries are new to you.
• Tool Schema Design — Skim this lesson if you have not designed or validated JSON Schema tool inputs.

Ordered lessons

1. 06MCP Fundamentals: Stateless Requests and JSON-RPC55 min
2. 07Building an MCP Server: Stateless Python and TypeScript85 min
3. 08Building an MCP Client: Discovery, Routing, and Dual-Era Fallback85 min
4. 09MCP Transports: stdio and Stateless Streamable HTTP65 min
5. 10MCP Resources and Prompts: Addressable Context for Stateless Servers60 min
6. 11MCP Model Input: Sampling Migration and Stateless MRTR75 min
7. 12Explicit Scope and Stateless Elicitation60 min
8. 13MCP Tasks Extension: Durable Work on a Stateless Core90 min
9. 14MCP Apps on the Stateless Protocol75 min
10. 15MCP Security: Poisoned Metadata, Routing, and MRTR State60 min
11. 16MCP Authorization: CIMD, Issuer Binding, PKCE, and Step-Up90 min
12. 18MCP Auth in Production: Issuer-Bound Enrollment and Tokens90 min
13. 17Stateless MCP Gateways and Registry Admission75 min
14. 28MCP Tool Contracts and Content120 min
15. 29MCP Reliability, Cancellation, and Flow Control120 min
16. 30MCP Registry Supply Chain: Admission, Drift, and Rollback90 min
17. 31MCP Conformance Engineering: Versioning, Evidence, and Operations100 min

Read a compiled program you built yourself: parse the file, decode the instructions, lift them to pseudocode, record evidence, and prove a rebuild equal, then learn the field around that pipeline with its main tools.

Prerequisites

• Python, hexadecimal, and the command line — You can read Python, run a command, and convert between hexadecimal and decimal. No prior assembly or binary analysis experience is required.
• Python 3.10 or later — Every reader, decoder, lifter, and ledger in the path uses the Python standard library only. The compiled fixtures are in the repository.
• clang for rebuilding fixtures and Lesson 36 — clang rebuilds the fixtures from source and compiles the Lesson 36 differential driver. Without it, Lesson 36 reads the committed objects and reports its behaviour claims as unknown.
• Analyze only programs you built or are authorized to analyze — Before Lesson 32's lab, confirm that every target is a program you wrote and compiled, or one with written authorization. Lesson 32 opens with the law and the authorization rules.

Ordered lessons

1. 32Program Anatomy: An ELF Reader From Scratch120 min
2. 33Disassembly and Control Flow: From Bytes to a Graph You Can Check150 min
3. 34Lifting to Pseudocode: IR, SSA, and Structuring From Scratch150 min
4. 35Evidence, Confidence, and Unknowns in Binary Analysis105 min
5. 36Rebuild and Prove Equal: Differential Tests, Byte Comparison, and the Evidence Loop110 min
6. 37The Reverse Engineering Landscape: History, Tool Families, and How a Decompiler Is Built150 min
7. 38Calling Conventions and Stack Frames: System V, Windows x64, and AArch64140 min
8. 39Compiler Idioms: What Optimization Does to Your Code and How to Read It Back150 min
9. 40Linking, Loading, and Stripped Binaries: PLT, GOT, Imports, Fixups, and Function Identification160 min
10. 41Recovering Data and Types: Structs, Arrays, C++ Objects, and DWARF140 min
11. 42Dynamic Analysis of Your Own Program: Debuggers, Traces, and Confirming Static Claims130 min
12. 43Beyond Native Code: Python, JVM, .NET, and WebAssembly Bytecode150 min
13. 44Binary Diffing and Symbolic Execution: Matching Functions and Solving for Inputs150 min
14. 45AI-Assisted Reverse Engineering: LLM Decompilers, Analysis Agents, and Ground Truth140 min

Turn evidence into outcomes, assumptions, testable slices, executable specifications, measurement plans, and owned feedback loops.

Ordered lessons

1. 47Define the Outcome Before You Choose the Output60 min
2. 48Discover the Workflow People Actually Perform70 min
3. 49Map Assumptions and Resolve the Riskiest One First65 min
4. 50Choose the Smallest Slice That Can Change the Decision65 min
5. 51Write Specifications That Preserve Judgment75 min
6. 52Design Success Metrics Before the Result Exists70 min
7. 53Choose Prototype, Pilot, or Production Deliberately70 min
8. 54Build a Feedback Ratchet with Ownership and Retirement75 min

Build the repository, debugging, testing, interface, security, release, and operational foundations AI systems depend on.

Ordered lessons

1. 01Dev Environment45 min
2. 02Git & Collaboration60 min
3. 06Python Environments45 min
4. 07Docker for AI60 min
5. 09Data Management60 min
6. 10Terminal & Shell45 min
7. 12Debugging and Profiling60 min
8. 01The Tool Interface — Why Agents Need Structured I/O60 min
9. 05Tool Schema Design — Naming, Descriptions, Parameter Constraints60 min
10. 38Verification Gates55 min
11. 20Shadow Traffic, Canary Rollout, and Progressive Deployment for LLMs60 min
12. 25Security — Secrets, API Key Rotation, Audit Logs, Guardrails60 min
13. 23SRE for AI — Multi-Agent Incident Response, Runbooks, Predictive Detection60 min

Frame, plan, execute, delegate, verify, review, and improve coding-agent work inside real repositories.

Ordered lessons

1. 31Agent Workbench Engineering: Why Capable Models Still Fail45 min
2. 32The Minimal Agent Workbench45 min
3. 33Agent Instructions as Executable Constraints50 min
4. 34Repo Memory and Durable State60 min
5. 35Initialization Scripts for Agents45 min
6. 36Scope Contracts and Task Boundaries50 min
7. 43Frame the Task Before the Agent Writes Code60 min
8. 44Build an Evidence-Backed Execution Plan65 min
9. 37Runtime Feedback Loops50 min
10. 38Verification Gates55 min
11. 39Reviewer Agent: Separate Builder from Marker55 min
12. 45Delegate Agent Work with Isolation and Merge Contracts70 min
13. 40Multi-Session Handoff50 min
14. 41The Workbench on a Real Repo60 min
15. 46Turn Every Agent Correction into a System Improvement65 min
16. 42Capstone: Ship a Reusable Agent Workbench Pack75 min