The AI-native stack atlas.
Follow a shopping cart through 16 layers of an AI application, from the first model call to a verified payment.
The syllabus
Open a lesson to read the explanation, worked example and practice question.
Layer 01Models and inference
Match each request to a model and an execution environment.
Why this layer exists
Inference runs a trained model on a new input. Choose an endpoint by the inputs it understands, the tools it can call, the outputs it can produce, and how it performs on your own examples. A hosted text model and an image editor solve distinct parts of a room-design task.
Tools to know
DeepSeek, GPT, Claude, Gemini, vLLM, SGLang, NVIDIA Dynamo, llm-d
Worked example
A customer asks for matching curtains. A vision-language model identifies the window, a text route extracts the budget, and an image editor renders the approved swatch. Each route has a separate test.
Terms in this layer
- Prefill/decode disaggregation
- Prefill processes the prompt, and decode produces output tokens. Separate hardware pools can scale those phases independently when the workload and interconnect justify the extra transfers.
- KV-cache-aware routing
- A router considers reusable attention state and queue load when selecting a model replica. This is distinct from choosing a different model or provider.
- Model snapshot
- A named checkpoint or version for a model. Record the version actually used and validate the endpoint whenever an alias or provider changes.
- Foundation models, LLMs and small language models
- A large language model (LLM) interprets requests and generates text or tool calls. A smaller, specialized model can handle frequent, narrow tasks once it passes your quality tests.
- Multimodal models and vision-language models
- A vision-language model (VLM) can interpret images alongside instructions. Input and output modalities are separate capabilities, so check whether the endpoint can read an image, generate pixels or do both.
- Model APIs, SDKs and capability matching
- An API defines the request format and a software development kit wraps it in your language. Check tool calling, schema support, file inputs, streaming and model-specific limits for the exact endpoint.
- Prompts, tokens and inference controls
- System instructions set the role and constraints. Few-shot examples demonstrate the output you want. Tokens consume context and cost. Supported controls such as output limits, reasoning effort and temperature change the response budget or variation. A token is one unit of encoded input or output, often a word fragment.
- Inference, hosted endpoints and open weights
- Inference means running a trained model on a new input. Hosted endpoints simplify launch. Open weights give you control over the runtime subject to the checkpoint’s license.
- GPU runtimes and serving engines
- The engine loads weights and schedules work. Language generation and image diffusion have different operations, tuning knobs and useful performance measures.
- Inference benchmarks and GPU economics
- Time to first token measures the initial text wait. Inter-token latency describes streaming pace. For images, measure queue time, full render time, tail latency and accepted results. Track time to first token (TTFT), inter-token latency (ITL), tokens per second (TPS) and GPU-seconds per image.
- Continuous batching, KV cache and prefix caching
- Continuous batching schedules active requests together. A key-value cache retains attention state during decoding. Prefix caching reuses compatible shared prompt prefixes across requests.
- Quantization and distributed inference
- Quantization represents weights or activations at lower precision to reduce memory or computation. Replicas spread requests across GPUs. Tensor parallelism splits a model across GPUs.
- Autoscaling and capacity planning
- A cold worker must initialize and load weights. A warm pool lowers the first customer’s wait while increasing idle cost. Concurrency limits keep the queue manageable.
Put it into practice
Run five ambiguous requests through two endpoints. Record extraction accuracy, first-token delay, complete-response time and cost. If self-hosting, repeat with cold and warm caches.
When would cache-aware replica selection become useful?
Reveal an answer
When several replicas could handle a request and some already hold a compatible prompt prefix. Weigh the saved prefill work against queue delay. A single replica has no routing choice.
Primary sources
Layer 02Gateways, caching and privacy
Control where model calls go and what can be reused.
Why this layer exists
A model gateway chooses among approved endpoints and can enforce budgets. Every route that receives customer data needs a known retention policy. Caching saves repeated work when permissions, source versions and freshness rules match.
Tools to know
OpenRouter, LiteLLM, Portkey, Cloudflare AI Gateway, Redis, RedisVL
Worked example
Many customers ask how to measure a curtain drop. Reuse a published answer for the same policy version and locale. Fetch each customer’s quote and stock status from the order service.
Terms in this layer
- Zero data retention (ZDR)
- A provider-specific data handling commitment. Verify endpoint, feature and account eligibility and apply compatible rules to fallback and shadow routes.
- PII tokenization
- Replace a sensitive identifier with a scoped opaque value before sending data to another system. Protect the mapping if reidentification is needed.
- Prompt-prefix caching
- Reuse model computation for an identical eligible input prefix while generating a fresh response.
- Model gateways, routing and fallbacks
- A gateway provides a shared model interface. Provider failover can switch hosts for one model. A model fallback can change quality or supported features, so qualify each route with the same tests. Per-user quotas, call limits and spend budgets help contain the cost of a looping request.
- Semantic response caching
- A semantic cache can return a prior answer to a differently worded request judged sufficiently similar. Similarity threshold, tenant scope, content version and expiration govern reuse. Prompt-prefix caching reuses model computation to generate a new answer.
- Application caching and invalidation
- A cache holds reusable results. A time to live (TTL) limits how long a value stays eligible. Invalidation evicts a result when its source changes. The cache key must include the scope of the result.
Put it into practice
Write cache-key and expiry rules for one public FAQ and one private cart. Test a request from another household and a change in the price list.
If the primary provider fails, may the gateway send a room photo to any available model?
Reveal an answer
Use only a fallback approved for this asset type, region, retention policy and required capability. If no route qualifies, offer the catalog or a later retry.
Primary sources
Layer 03Agent harnesses and durable work
Give a model a bounded loop and persist slow work.
Why this layer exists
The agent harness assembles context, validates proposed tool calls, executes authorized actions, supplies results, and decides when to stop. Long-running workflows record progress so another worker can continue after a crash. External effects still need idempotency.
Tools to know
Python, Pydantic AI, OpenAI Agents SDK, LangGraph, Deep Agents, Temporal, Inngest
Worked example
The assistant looks up eligible fabrics and calls the quote service. If the width is missing, it pauses. An image job can finish after the customer closes the browser.
Terms in this layer
- Parallel fan-out
- Run independent reads concurrently. Reads that depend on a fresh cart, stock or permissions must use the correct version and await their prerequisites.
- Latency budget
- Assign a deadline to the overall task and its slower steps. Track time to first useful feedback and time to a validated proposal separately.
- Deterministic fallback
- A known application path, such as filtered catalog search, that remains useful when a model or tool exceeds its time or spend limit.
- Agent harness and bounded agent loop
- A harness runs each turn, supplies context and tools, executes permitted calls, feeds back results and stops or escalates when the job is complete or reaches its limits.
- Agent orchestration, graphs and checkpoints
- A graph defines allowed transitions between steps. A checkpoint saves state. In an agent graph the model can choose from permitted routes. A business workflow can instead prescribe a fixed sequence. LangGraph and Temporal can serve different parts of the same application.
- Managed agent runtimes and sandboxes
- A hosted runtime can operate agent sessions, tools and recovery. A sandbox isolates code and file work. Choose runtime ownership based on which permissions, data boundaries and operational controls you need. LangChain Deep Agents is a harness SDK. Managed Deep Agents is its hosted offering.
- Durable execution, workflows and activities
- Durable execution records the progress of a process so it can resume. A workflow coordinates the steps. Activities perform outside work, such as calling an editor. A signal can wake a workflow waiting for a person or a provider.
- Job queues, workers and backpressure
- A queue buffers jobs while workers are busy. Backpressure limits intake to a sustainable rate. A dead-letter queue (DLQ) or failed-job table preserves work that exhausts its attempts.
- Timeouts, retries, circuit breakers and fallbacks
- A deadline bounds the wait. Exponential backoff spaces out retries, jitter spreads them across time, and a circuit breaker temporarily pauses calls to an unhealthy dependency.
Put it into practice
Define three tools, a missing-measurement branch, a step limit and a deadline. Restart the worker immediately after it submits an edit and verify a single paid render job.
How does the harness handle an editor that times out after accepting a job?
Reveal an answer
Persist the provider job ID or idempotency key. Look up the existing job when possible, then resume or retry within the deadline.
Primary sources
Layer 04Tool protocols and connectors
Connect a model to application capabilities and partner data.
Why this layer exists
Typed application functions are sufficient for an initial product. Protocols become valuable when independent clients or partners need a shared contract. The host enforces authorization on every request and the application keeps its business state.
Tools to know
MCP, Agent2Agent (A2A), Agent Skills, Nango, Composio, OAuth
Worked example
Expose public swatch search and authorized cart reads to a website and an installer assistant. Each call identifies the project and runs the same ownership checks.
Terms in this layer
- Progressive disclosure
- Expose a short skill description or tool catalog first, and load the instructions or data needed for the active task on demand.
- Agent Card
- An A2A description of an agent’s capabilities and interaction endpoints, used by another system to discover suitable delegation targets.
- MCP, tool servers and connectors
- The Model Context Protocol standardizes how hosts discover and access capabilities such as tools, resources and prompt templates. It is useful when several clients or partners need the same adapter. The host is the application, the client connects from that host, and the server exposes capabilities.
- A2A, AG-UI and generative UI
- Agent2Agent (A2A) supports delegation across independent agent systems using Agent Cards, tasks and artifacts. AG-UI streams agent events to a frontend. A2UI describes widgets a trusted client can render.
- Managed connectors and delegated OAuth
- A connected account gives an app scoped permission to use somebody else’s service. The connector can handle consent, refreshing tokens, revocation and status. The application still decides who may invoke each operation.
- Visual workflow automation and partner operations
- A low-code builder connects triggers, API steps and conditional branches. It can save engineering time on integrations where the business operation already has a stable API.
Put it into practice
Define a swatch tool with a narrow input and compact result. Decide which parts belong in the internal API and which parts an MCP server should expose.
Does a stateless MCP deployment make a shopping cart stateless?
Reveal an answer
The protocol can route independent requests without session affinity. The business service still stores the cart, revision, quote, customer identity and approval.
Primary sources
Layer 05Commerce and payment protocols
Connect an approved proposal to a real order and payment.
Why this layer exists
The merchant’s systems own the SKU, stock, quote, approval and order. An external commerce protocol can help another assistant discover a merchant and submit a checkout. A payment provider verifies the money movement.
Tools to know
Razorpay, UCP, Poppy, ACP, AP2, x402, MPP
Worked example
A customer approves a quote for a specified fabric, length and installation. The checkout uses that exact revision. A signed callback confirming capture can release the order to the installation workflow.
Terms in this layer
- Poppy
- Personal Agent Protocol lets a company publish how personal agents can interact with it. Discovery is followed by the required authentication and authorization.
- Checkout Mandate and Payment Mandate
- In AP2, a closed Checkout Mandate binds the selected merchant checkout. A Payment Mandate conveys payment authorization. The merchant still verifies the resulting payment state.
- Stock-keeping unit (SKU)
- The stable identifier for a sellable variant, such as a particular fabric, width and color. Use it to join the preview, quote and fulfillment record.
- Catalog, stock-keeping unit (SKU) and versioned quote
- A SKU identifies the exact sellable variant. A pricing function calculates a quote from catalog prices, measurements, labour and fees. A quote version records what the customer saw.
- Cart, reservations and order state
- A cart holds editable choices. A reservation with a time to live (TTL) holds stock or a visit slot temporarily. An order state machine defines which changes are valid.
- Checkout, capture and settlement
- Checkout collects payment. Authorization approves the charge, capture completes the payment for fulfillment, and settlement transfers funds to the merchant on the processor’s schedule.
- Webhook, idempotency and reconciliation
- A webhook delivers a provider event to your server. An idempotent handler makes a repeated event harmless. Reconciliation compares local state with the payment provider after delays or failures.
- External agent commerce protocols
- Universal Commerce Protocol (UCP) describes discovery through checkout and post-purchase. Agentic Commerce Protocol (ACP) enables agent-facing checkout. Agent Payments Protocol (AP2) represents verifiable authority for an agent-assisted payment.
- Machine payments and external agent identity
- HTTP 402 can request payment for an HTTP resource. x402 and the Machine Payments Protocol (MPP) specify ways for machines to exchange payment information. Network programs can help a merchant recognize a third-party agent and its intent.
Put it into practice
Draw the order state changes from draft to paid. Replay a payment event twice and a late authorization after capture. Confirm that only one job becomes eligible for fulfillment.
Does a public discovery manifest give an external assistant permission to pay?
Reveal an answer
The manifest advertises supported capabilities. The caller needs identity and delegated permissions, the customer must authorize the relevant transaction, and the merchant verifies payment separately.
Primary sources
Layer 06Context engineering and retrieval
Give the model the evidence and constraints it needs for this turn.
Why this layer exists
Context includes instructions, tool shapes, the active project, retrieved facts and selected history. Retrieval-augmented generation (RAG) supplies relevant evidence at answer time. Keep hard constraints in validated fields and restrict retrieval to sources the customer can access.
Tools to know
PostgreSQL, pgvector, Qdrant, OpenSearch, Docling, LlamaIndex, Cohere Rerank
Worked example
A request for “warm beige blackout” benefits from meaning-based search. The exact fabric code and an installed budget need precise filters. Provide the SKU, width, opacity and quoted cost to the model.
Terms in this layer
- Tool-output projection
- Select and validate the small set of response fields the model needs. Preserve the full authoritative data in its owning service.
- Constraint pinning
- Store binding requirements outside a compacted transcript and reload them for each relevant decision. Enforce them again on the resulting action.
- Compaction
- Reduce the amount of conversational history in the active context, often by summarizing older turns. Retain durable source records for later use.
- Context windows, compaction and prompt caching
- The context window limits how much a model sees on a turn. Compaction summarizes long histories. Prompt caching can reuse repeated input prefixes to reduce repeated processing.
- Context engineering, sessions, memory and skills
- Context engineering decides what this turn sees. A session tracks one conversation. Durable memory retains selected facts across sessions. A skill packages reusable instructions and supporting resources that can load on demand. Context can include instructions, tool descriptions, a photo, retrieved records and tool results. Progressive disclosure makes a short description available first, then loads detail on demand.
- Ingestion, parsing and provenance
- Ingestion turns source material into searchable records. OCR reads scanned pages, layout parsing reconstructs tables, and chunking splits text into passages. Provenance records each passage’s origin.
- Embeddings and vector indexes
- An embedding represents meaning as numbers. A vector index retrieves nearby representations. An approximate nearest-neighbour index such as HNSW can speed search at the cost of some missed matches. Text and image embeddings from a compatible multimodal model can also retrieve a swatch using a photo.
- Hybrid search and reranking
- Lexical search scores words and codes, while dense retrieval scores meaning. Reciprocal rank fusion combines ranked lists. A reranker then scores the strongest candidates against the request. BM25 is a common lexical scoring method. A cross-encoder can rerank a small candidate list.
- RAG, grounding and access control
- Retrieval-augmented generation supplies selected evidence to the model at answer time. Grounding ties claims to that evidence. Document-level access rules limit retrieval to records the caller may see.
- Agentic retrieval, GraphRAG and knowledge graphs
- Query rewriting and multi-step retrieval help answer questions that need several lookups. A knowledge graph records entities and their relationships. GraphRAG retrieves evidence using graph structure.
- Hard constraints, preferences and substitutions
- A hard constraint decides eligibility. A soft preference ranks eligible choices. A substitution policy records which lines the customer will let the assistant change. An infeasible result means the current options cannot meet every hard constraint.
Put it into practice
Compare exact keyword, vector and hybrid search on ten requests. Include one SKU code, a color paraphrase and a request with a missing required attribute. Trace which stage lost a good result.
Why keep “keep the existing track” outside the conversation summary?
Reveal an answer
A summary can omit or soften a restriction. Save it as a project constraint, reload it into planning and validate the final proposal against it in code.
Primary sources
Layer 07Memory and application state
Keep track of facts, preferences, events and permissions over time.
Why this layer exists
The active room and proposed change belong to the session. A paid order, approved quote and project restriction require durable structured records. Long-term memory can store consented preferences or earlier design decisions with clear ownership and a way to correct them.
Tools to know
PostgreSQL, Redis, LangGraph checkpoints, Mem0, Zep, Graphiti
Worked example
The customer prefers washable fabric in general but wants blackout curtains in one bedroom. The design can reuse the first preference only within its intended scope and must re-read current availability.
Terms in this layer
- Episodic memory
- A record of an event, such as the customer rejecting a swatch because the texture looked coarse.
- Semantic memory
- Durable facts or preferences such as a preference for washable fabrics, stored with scope and provenance.
- Procedural memory
- Instructions or known procedures, such as how to measure a curtain drop and when to request confirmation.
- Working memory
- Information used for the current task, such as the selected window and the most recent edit.
- Transactional and object storage
- A relational database preserves relationships and commits related changes together. Object storage holds large files under stable keys.
- Session state and long-term memory
- Session state tracks the active task. Long-term memory stores selected facts across visits. Compaction summarizes older conversation to fit the model’s context window. Scope room decisions to the project and save cross-project preferences only when appropriate. Working memory holds the active task. Episodic memory records earlier events, semantic memory stores facts, and procedural memory provides instructions.
Put it into practice
Write four records for an active room, a saved preference, a rejected swatch and a payment. Give each record a scope, owner, retention policy and source of truth.
Which memory item can authorize a future charge?
Reveal an answer
A recalled preference or previous payment is insufficient. Use a valid grant or fresh approval bound to the amount and purchase the customer authorized.
Primary sources
Layer 08Tools and action execution
Let verified application code perform exact operations.
Why this layer exists
A tool has typed inputs, predictable effects and a permission boundary. Schema validation checks shape. Domain rules check ownership, units, availability and affordability. An isolated execution environment is useful for narrowly scoped code or browser jobs.
Tools to know
Pydantic, Zod, JSON Schema, FastAPI, Google OR-Tools, Playwright, E2B
Worked example
A model proposes a new fabric SKU. The quote service calculates material use, stitching and installation with verified dimensions and returns a versioned total in paise.
Terms in this layer
- Constrained decoding
- An inference technique that restricts token choices to an allowed format. Handle refusals, interruptions and schema limitations, and check the business meaning in application code.
- Dry run
- Compute a proposed change and its effects before committing any mutation. A cart proposal can be revalidated against its original version when approved.
- Backend APIs and typed contracts
- An API is the interface the app calls to do work. REST exposes resources over HTTP, OpenAPI describes the operations and JSON Schema describes valid data. Validation catches missing fields or incorrect units before work starts.
- Structured outputs, schemas and validation
- A structured output constrains model data to a supported schema. Application validation checks that the values make sense and the customer can act on the referenced object.
- Function calling, tool contracts and permissions
- The model proposes a named operation with typed arguments. The host validates the request, applies authorization and invokes the real service. Read tools and write tools can carry different permissions.
- Browser agents, computer use and WebMCP
- Computer-use agents interact with a browser through screenshots, actions or a browser driver. WebMCP is a proposed web standard for a page to expose structured actions to a browser agent. A server-side API offers a stable contract when a partner supplies one.
- Promotions, loyalty and basket optimization
- Basket optimization compares combinations. A promotion engine prices each eligible basket using actual rules, which may depend on quantities, bundles and coupon exclusions. A loyalty service supplies available rewards and redemption rules.
- Sandbox and least-privilege credentials
- A sandbox isolates code or browser sessions an agent runs. Least privilege gives each worker only the credentials, destinations and resource budget required for its task. Apply a tool allowlist and an egress policy to limit permitted actions and outbound connections.
Put it into practice
Build a pure function that prices one window from width, drop, fabric width and costs. Test invalid units and a cheaper swatch that needs more fabric.
Why can valid structured output still produce the wrong cart?
Reveal an answer
The payload may reference a real-looking but unknown SKU, another customer’s room or out-of-date stock. Validate domain data and permissions immediately before the write.
Primary sources
Layer 09Multi-agent patterns
Choose where to delegate work and where to keep a fixed workflow.
Why this layer exists
A planner, executor and evaluator can have different information or permissions. Each handoff consumes time and introduces another interface to test. Split a task when isolated permissions, independent inputs or measurable quality gains justify the coordination.
Tools to know
LangGraph subgraphs, OpenAI Agents SDK handoffs, Google ADK, Claude Agent SDK, CrewAI
Worked example
A design assistant proposes three fabrics while an independent visual check scores target preservation. A deterministic checkout service applies the customer’s accepted quote.
Terms in this layer
- Handoff
- Transfer a scoped task and relevant state to a specialist or person. Record which actor is responsible for the next action.
- Planner–executor
- A planner proposes steps while a separate execution layer checks and carries them out with its own tools and permissions.
- Evaluator–optimizer
- One component produces a candidate and another evaluates it against explicit criteria. Bound refinement attempts and escalate persistent failures.
- Agent patterns, handoffs and multi-agent systems
- Routing selects a path, parallel workers handle independent tasks, and an evaluator checks a result. A handoff transfers control to a specialist or person. A planner and executor can handle open-ended work when a fixed flow is insufficient. ReAct interleaves model reasoning with actions and observations. Parallel agents can handle independent subtasks.
Put it into practice
Compare one bounded loop with a planner and verifier on ten ambiguous shopping requests. Record disagreements, cycle time and whether a human accepted each proposal.
Which tasks could run in parallel, and which must wait?
Reveal an answer
Independent swatch and design-guideline searches can run together. Confirmed stock for selected SKUs depends on the selected items. Applying a quote must wait for approval and revalidation.
Primary sources
Layer 10Identity, guardrails and governance
Decide who can act, on whose behalf, with which data.
Why this layer exists
Authorization belongs at the service that owns the action. A model can interpret an untrusted supplier description, but the description cannot grant access or spending authority. Keep a record of the person or workload, its scope and the exact approved revision.
Tools to know
OpenID Connect, OAuth, PostgreSQL RLS, Open Policy Agent, Vault, Cloudflare R2, OpenTelemetry redaction
Worked example
An installer can read measurements for an assigned job. A background worker may notify a shopper about a price change only within its authorization. Buying the item requires separate authority.
Terms in this layer
- Non-human identity (NHI)
- A service account or workload identity assigned to software. Give it scoped credentials and record the human delegation or business rule behind sensitive actions.
- Human-in-the-loop (HITL)
- A human reviews or authorizes a specific action. Store the approved object, version, parameters, amount and expiry where relevant.
- Indirect prompt injection
- Instructions hidden in content an agent reads. The application should treat that content as data and enforce permissions on tool calls.
- Authentication, authorization and tenant isolation
- Authentication identifies a user. Authorization checks permission for a particular action. Role-based access control (RBAC) assigns permissions by role, and row-level security (RLS) can scope database rows to each household. OpenID Connect (OIDC) enables federated sign-in. OAuth scopes limit delegated access.
- Private uploads and expiring access
- A presigned URL grants temporary access to one object. File size limits, allowed types and file-content checks reduce risk from uploaded media.
- Prompt injection, guardrails and tool permissions
- An indirect prompt injection is an instruction hidden in content an agent reads. Guardrails can flag inputs or outputs. A backend permission check constrains each tool action even if the model is misled. Content moderation checks suitability. Treat uploaded text and tool responses as untrusted input.
- Human approval and audit trail
- Human-in-the-loop approval binds a person’s decision to an exact action and its parameters. An audit trail records who approved it, when and which tool ran.
- Privacy, redaction and retention
- Minimization limits what you collect. Redaction removes sensitive fields from telemetry. A retention policy specifies how long originals, previews and traces are kept and how deletions propagate.
Put it into practice
Write a permission matrix for shopper, installer, support staff and background worker. Try cross-project reads, an expired signed URL and a tool call whose price changed after approval.
A supplier note asks the assistant to send a room photo to a new service. What should stop it?
Reveal an answer
An allowlist and resource-level authorization in the host should prevent the transfer. Treat the supplier note as untrusted data and verify approved model routes for private media.
Primary sources
Layer 11Evaluation and regression testing
Test model choices, tool trajectories and final application state.
Why this layer exists
An eval harness replays realistic scenarios and records scores. Use exact assertions for arithmetic and permission rules, human-calibrated rubrics for visual quality, and repeated trials for variable behavior. Keep an independent real-world holdout.
Tools to know
pytest, Promptfoo, Braintrust, LangSmith, Inspect AI, Ragas, VCR.py
Worked example
A model may describe the right total after attempting checkout too early. Check both the order of actions and the saved quote. For an image edit, check that the lamp and coffee table remain consistent.
Terms in this layer
- Tool cassette
- A sanitized recording of an external response used as a test fixture. Run the candidate model live against that input to evaluate its current decisions.
- Contract test
- A test against a real or contract-faithful service that checks the integration assumptions, such as supported fields and payment state transitions.
- Trajectory
- The observable sequence of tool calls, arguments, decisions and results during a task. Score required ordering while allowing equivalent safe paths.
- End-to-end in-cart acceptance test
- Start with an existing cart fixture and define the conditions for a valid change. Exercise happy paths, stale inventory, a changed budget, concurrent edits and the case where no feasible basket exists.
- Golden datasets and regression gates
- A golden set contains representative examples with expected behavior. Offline evaluation replays these before release. A regression gate blocks changes that fail agreed checks. An evaluation harness is the test runner that executes cases and collects scores.
- Task-specific graders and human review
- A code grader checks exact rules. A rubric gives human reviewers or a model judge consistent criteria for subjective outcomes. Calibrate a model judge against human-scored examples. LLM-as-judge is the term for automated model grading.
- Retrieval and grounded-answer quality
- Recall@k measures the share of relevant records found in the top k results. Faithfulness asks whether the answer’s claims follow from the evidence supplied to the model. A hallucination is a plausible claim the available evidence fails to support.
- Trajectory evaluation and repeated trials
- A trajectory records the observable sequence of actions, tool arguments and results. Outcome checks inspect the final state. Repeated runs reveal reliability when the same model produces different behavior. Allow harmless alternatives in tool order.
Put it into practice
Record sanitized tool responses for a stale-price scenario. Run the new model against those fixtures and verify that it requests fresh approval. Then run a separate live contract test.
What can a recorded tool fixture prove, and what still needs a live check?
Reveal an answer
The fixture makes a past tool response repeatable so you can compare model behavior. A live check is needed to detect schema, stock or provider behavior changes since recording.
Primary sources
Layer 12Observability and unit economics
Trace failures and measure the cost of useful outcomes.
Why this layer exists
A trace links the user’s request to application spans for search, model inference, rendering and checkout. Logs and metrics can explain latency, retries and drop-off without retaining private room photos or full chat transcripts. Compare costs over the same completed outcome.
Tools to know
OpenTelemetry, Langfuse, Sentry, Prometheus, Grafana, Datadog, PostHog
Worked example
A customer waits 14 seconds for a preview. Trace spans reveal nine seconds of queue wait and five of rendering. Include failed and rejected renders when calculating cost per accepted preview.
Terms in this layer
- Time to first feedback
- Elapsed time from the user action to a useful acknowledgement, progress signal or partial result. Measure the full proposal completion time separately.
- Trace correlation
- Carry an identifier across request, job, model call and order so individual records can be investigated together.
- Contribution margin
- Revenue less variable costs for the order. Compare model and operational spend against the margin that remains after material, payment and fulfillment costs.
- Product events and conversion funnels
- Product analytics measures which customer actions lead to a booking. A funnel counts users or projects that reach successive steps. Give each event a precise definition so the team can compare releases.
- Traces, spans, logs and metrics
- A trace follows one request across services. A span times one operation, a log records an event, and a metric tracks a number over time.
- Service latency, reliability and business cost
- A p95 latency is the time within which 95% of measured requests complete. An SLO sets a target for that measure or for success rate. Cost per accepted preview includes the attempts customers rejected.
- Product analytics and online evaluation
- Product events show where customers drop out. Online evaluations grade sampled live runs. A/B testing randomly assigns variants so the team can measure a change’s effect.
Put it into practice
Attach one correlation ID to a mock edit, its queue work and its quote. Compare p95 time to feedback, time to finished preview, acceptance rate and spend per accepted preview.
If a faster editor is cheaper per render, how could it still cost more?
Reveal an answer
Lower acceptance can trigger repeat requests and human review. Compare total spend, infrastructure and review cost for the same number of accepted previews at comparable quality.
Primary sources
Layer 13Interfaces, visual media and cart sync
Show exactly what changed and keep review screens current.
Why this layer exists
Use stable IDs, original media and typed UI components to make a proposal inspectable. Streaming can show progress while workers run. A cart revision prevents a delayed edit from replacing a newer choice when the server enforces the version at commit.
Tools to know
React, Vercel AI SDK, A2UI, AG-UI, WebSockets, SSE, SAM, Diffusers, LiveKit, Three.js
Worked example
A customer changes the window width in another tab while the assistant proposes a fabric swap. The UI displays one diff, and the server checks the expected version when approval arrives.
Terms in this layer
- Cart diff
- A reviewable list of added, removed, substituted and resized lines with reasons, amounts, and a reference to the original cart version.
- ETag and If-Match
- An ETag identifies a representation version. An If-Match condition can reject a stale write, provided the backend checks it atomically at commit.
- Multimodal fidelity
- Whether an edit follows the requested target and preserves protected regions, geometry, texture and lighting. Use source images and customer-selected swatches to judge it.
- Web and mobile chat
- A multimodal interface lets customers combine text, photos and taps in one conversation. A responsive web app works on a phone and is easy to share. Native apps can reuse the same backend.
- Streaming and progress events
- Streaming delivers pieces of a reply as they arrive. Server-sent events (SSE) carry updates from server to client over HTTP. WebSockets support messages in both directions on a persistent connection.
- Interactive cards and generative UI
- Typed cards give the customer controls for choosing products or approving a quote. For richer interfaces, AG-UI standardizes interaction events and A2UI lets the agent describe a screen that your app renders with approved components. MCP Apps lets a tool supply an interactive interface inside a compatible MCP host.
- Job identity, status and reconnection
- A conversation ID groups messages. A job ID identifies one long-running edit, while an event ID identifies one update. A saved status lets a reconnecting client find work already in progress.
- Visual understanding and grounding
- A vision-language model interprets the picture together with the request. Grounding connects words to a location or object.
- Segmentation, masks and compositing
- Segmentation marks the pixels belonging to a surface. A mask defines the editable region and compositing blends the result back into the original.
- Image-to-image editing and visual conditioning
- An editor changes an existing image. Inpainting edits a selected region. A reference image supplies the swatch, texture or product to match. Diffusion and flow-matching models produce an image through a series of refinement steps. The chosen editor determines which controls it exposes.
- Image quality and revision history
- A convincing picture can still change the wrong object. Evaluate target accuracy, preservation outside the mask, geometry and customer acceptance.
- Voice sessions and turn-taking
- A voice app must carry audio, detect when a customer has finished and handle interruption. A cascaded pipeline joins speech recognition, a language model and speech synthesis. ASR or STT means speech to text. TTS means text to speech. Native speech-to-speech is another architecture.
- Persistent 3D scenes and measured geometry
- A scene graph gives walls and objects stable identities across camera views. Parametric geometry stores dimensions so an edit can follow the same wall throughout the room. A mesh describes surfaces and a material describes how they look.
- Dry run, cart diff and version-bound approval
- A dry run computes a proposed change without mutating the cart. A diff shows additions, removals, substitutions, quantity changes and their reasons. The approval references both the base version and the exact proposal.
- Database transactions, locks and version checks
- An ACID transaction commits related changes together. A uniqueness constraint protects an invariant. Optimistic concurrency compares an expected version before accepting a write.
Put it into practice
Prototype a side-by-side card showing old SKU, new SKU, price difference, reason and quote expiry. In another tab edit the cart. Verify that the old button produces a fresh quote.
Does receiving the latest cart through a WebSocket eliminate concurrent writes?
Reveal an answer
The client may still act on an older version. The server must apply an atomic version check, such as a strong ETag with If-Match backed by a database condition.
Primary sources
Layer 14LLMOps and application infrastructure
Release code, model choices and prompts with a reliable path back.
Why this layer exists
Behavior can change when a model alias, tool description, retrieval index or prompt changes even if the application code stays fixed. Record versions, validate migrations, run quality gates, and plan rollback. Put durable business state near the services that enforce its invariants.
Tools to know
Docker, PostgreSQL, Cloudflare Workers, GitHub Actions, OpenFeature, Terraform, OpenTofu, Temporal, LaunchDarkly
Worked example
A shadow pipeline runs a new planner on approved samples using simulated write tools. Its outputs are scored alongside the current planner. Only the production path owns live payment or booking credentials.
Terms in this layer
- Release fingerprint
- A recorded combination of model snapshot, prompt version, tool schema, retrieval data and application release that produced observed behavior.
- Shadow traffic
- Send a copy of eligible inputs to a candidate and compare outputs without using its result for the customer. Isolate credentials, side effects and sensitive data.
- Canary release
- Expose a candidate to a small, identifiable fraction of real traffic and compare quality, latency and failure rates before expanding.
- Transactional outbox and compensation
- An outbox saves an event in the same database transaction as the order update. A saga coordinates several services. Compensation is a business action that repairs a partially completed workflow.
- Hosting, containers, workers and the network edge
- A container packages code and dependencies. A reverse proxy handles incoming HTTPS and routing. A content delivery network (CDN) can serve public assets close to the user. Separate worker processes can run long jobs.
- CI/CD, release gates and rollback
- Continuous integration runs checks on each change. Continuous delivery produces a deployable version. Release code, prompts, schemas and model configuration together when their behavior depends on one another. MLOps and LLMOps extend release discipline to models, datasets, prompts, evaluations and monitoring.
- Secrets, backups and recovery objectives
- A secrets manager scopes credentials. A restore test verifies recoverability. RTO is the target time to resume service. RPO is the maximum acceptable data loss measured in time.
- SLOs, capacity tests and incident response
- A service-level indicator (SLI) measures user-visible behavior. A service-level objective (SLO) sets a target. An error budget describes the allowable misses. A runbook explains how to investigate and recover.
- Infrastructure as code, autoscaling and progressive delivery
- Infrastructure as code (IaC) puts deployment settings under version control. A canary sends a small share of traffic to a candidate version. Shadow traffic can test a second system with its side effects disabled.
- Data pipelines, warehouses and freshness
- ETL and ELT extract data and transform it before or after loading. Change data capture (CDC) propagates changes. A warehouse supports analysis across catalog, orders and product events. Lineage shows where a value came from.
Put it into practice
Create a release record containing the model snapshot, prompt revision, tool schema version, index revision and eval run. Roll back a failed candidate, then restore a test quote from backup.
Why disable payment tools in a shadow run if customers cannot see its output?
Reveal an answer
A mirrored request still executes on the candidate. Discarding the response does not undo tool calls. Use isolated credentials and simulated external effects.
Primary sources
Layer 15Fine-tuning and the data flywheel
Turn reviewed failures into better datasets and targeted model improvements.
Why this layer exists
Start with labels that describe the cause of failure and keep a test set independent of training data. Fine-tuning, preference optimization or distillation can help a repeated problem after simpler prompt, retrieval and product changes have been measured.
Tools to know
Hugging Face PEFT, TRL, LoRA, QLoRA, Label Studio, MLflow, DVC, Feast
Worked example
A rejected sofa recolor also changed the coffee table. Label unintended change outside the mask, add a regression case and keep photos from distinct homes for final evaluation. Record consent before using customer photos in training.
Terms in this layer
- Distillation
- Train a smaller student using outputs or other signals from a teacher model, then measure how much task quality, speed and cost change.
- RLVR
- Reinforcement learning with verifiable rewards uses automatic checks for properties such as exact totals. A measurable reward covers only the behavior it actually checks.
- Holdout by household
- Keep all related views and edits of one home on the same side of a train/test split, reducing leakage from near-duplicate photographs.
- Feedback labels, synthetic cases and holdouts
- A data flywheel turns reviewed failures into regression cases. Synthetic cases expand coverage of rare inputs. A holdout of independent real examples measures generalization. Feedback that may be reused for training needs a separate permission and data-use decision.
- Recommendations, personalization and ranking
- Candidate generation finds a shortlist. Ranking orders eligible items for a goal. A content-based recommender uses item attributes. Collaborative filtering uses patterns across many users or items.
- Model registry, data versions and feature stores
- A registry tracks candidate model versions and their evaluation records. Data versioning records which training examples produced them. A feature store shares consistent ML inputs across training and serving.
- Fine-tuning, adapters and distillation
- Supervised fine-tuning (SFT) learns from demonstrations. Low-rank adaptation (LoRA) trains small adapters. QLoRA trains adapters over a quantized base. Distillation teaches a smaller model from a teacher. Direct preference optimization (DPO) learns from preferred and rejected responses.
Put it into practice
Build a small failure taxonomy covering wrong target, changed surroundings and poor texture match. Compare a baseline with one candidate using the same held-out rooms.
Why is an abandoned cart an unreliable training label for visual quality?
Reveal an answer
The shopper may have disliked the price, timing or checkout experience. Ask for a reason or review independent evidence before assigning an image-quality label.
Primary sources
Layer 16Developer tools and release practice
Make every change easy to review, reproduce and teach.
Why this layer exists
Keep application contracts, examples, prompts and eval gates close to the code. A coding assistant can prepare changes and run checks in a developer environment. The human reviewer needs the specification, the diff and the evidence needed to judge customer impact.
Tools to know
Git, OpenCode, Codex, Claude Code, GitHub Copilot, Cursor, AGENTS.md, Promptfoo, Harness.io
Worked example
A pull request changes the fabric-search tool description. CI runs deterministic pricing tests and ambiguous shopping conversations, then posts which proposals or tool choices changed.
Terms in this layer
- Prompt registry
- A versioned record of prompts, ownership and release state. Link each change to its evaluation dataset and deployment.
- Spec-driven development
- Describe the expected behavior, interfaces and acceptance examples before implementation so a reviewer can verify the result.
- Pull-request evaluation diff
- A summary of changed results, costs and traces for two application or model revisions evaluated on the same cases.
- Coding agents and developer harnesses
- A repository-aware coding assistant can edit code, run tests and prepare changes for review. Repository instructions explain the architecture and the tests that a change must pass.
Put it into practice
Write a ticket for one constraint-preserving cart substitution. Include three acceptance cases, an expected trace and a rollback condition. Ask a coding assistant to implement the smallest reviewable change.
What are the three different uses of “harness” in this syllabus?
Reveal an answer
The agent harness runs customer tasks. The evaluation harness executes and scores tests. Harness.io provides CI/CD products for building and deploying software.
Primary sources
Edition updates
GPT-6.2 model routes and tool calling
OpenAI introduced GPT-6.2 Astra and Sol. Tool calling for these models uses the Responses API. Test the exact endpoint, schema and tool loop before promoting a candidate.
OpenAI API changelogRazorpay payments in a WhatsApp business conversation
Razorpay announced payment links through Meta Business Agent in the WhatsApp Business app for eligible small businesses in India. Check product eligibility and keep payment confirmation in the merchant workflow.
RazorpayPoppy 1.0 for external agent discovery
Sierra released Personal Agent Protocol 1.0 and reported that Shopify enabled it for merchants. A discovery profile advertises capabilities. Authentication and customer authority still matter on each sensitive action.
SierraDeepSeek expands its application APIs
DeepSeek added application API access for capabilities including file input, URL reading and web search. Check your model gateway separately because its feature surface can differ from the direct provider.
DeepSeek API release notesA2UI 1.0 for interactive agent interfaces
Google announced A2UI 1.0. A compatible client can render structured component descriptions. A cart comparison card remains subject to server-side version checks and approval rules.
Google DevelopersImage editing with markup guidance
OpenAI announced Images 2.6. Put a candidate editor through target-localization and surrounding-pixel preservation tests on real, consented rooms before changing the production route.
OpenAIGoogle begins its Canadian UCP checkout rollout
Google announced direct checkout for selected eligible Canadian retailers. Validate capabilities and onboarding requirements for the channel and market where you intend to sell.
Google Canadallm-d 1.0 for distributed inference
The llm-d project released version 1.0. Compare cache-aware replica placement and queue time in your own workload before choosing a serving topology.
llm-dMCP publishes a stateless protocol core
The July specification formalized stateless protocol requests, header-based routing and cacheable list responses. Keep project state in the application and use explicit resource handles.
Model Context ProtocolReview templates
Weekly change brief
Record the lesson, dated primary source, change, decision and one experiment.
Monthly syllabus review
Revisit the lesson, document the evidence behind each revised decision and write one new practice question.
Ask the tutor
Open this HTML in a browser with JavaScript enabled to ask questions and get links back to the lessons.