Production & Operations
Shipping an AI system is the start, not the finish. Agentic software running against real users brings cost, governance, reliability, and safety concerns that ordinary apps don’t. Here’s the operating model.
Once your helper is doing real work for real people, you watch it like a lifeguard — is it fast? is it making mistakes? is it costing too much? — and keep improving it from what you learn.
Environments, config & secrets
☺ Like you’re 10: It’s like having a messy practice kitchen and a spotless restaurant kitchen — you try new recipes in the practice one so mistakes don’t reach real diners, and you keep the secret spice jars (your keys) locked away.
The fundamentals still apply, with an AI twist. Keep separate environments — development, staging, production — and use a cheaper/faster model in dev and the production-grade model in prod (most SDKs and assistant tools let you set the model per environment — Anthropic’s API, OpenAI’s, GitHub Copilot, and others). Keep configuration — model choice, temperature, prompts, feature flags — external to code so you can tune without redeploying. And treat API keys as first-class secrets: model-provider keys, MCP server credentials, and tokens live in a secret manager, never in source. A leaked model key is a direct, metered bill.
Cost control (the one that surprises teams)
☺ Like you’re 10: Every time the AI thinks, it’s like feeding a coin into an arcade machine — a few coins is fine, but if it keeps playing by itself all night you come back to an empty piggy bank, so you set limits and an alarm.
AI features have a per-request marginal cost that traditional software doesn’t — every call burns tokens or premium requests. Controls that matter:
- Right-size the model. Don’t use a frontier model where a fast one suffices. Route by difficulty.
- Cache. Identical or near-identical requests shouldn’t re-run the model.
- Cap iterations. Agent loops should have a maximum step count so a confused agent can’t loop forever on your dime.
- Set budgets and alerts. Know your per-day and per-user spend, and get paged before a runaway hits the invoice.
- Watch agentic multipliers. Multi-step and multi-agent runs multiply cost — monitor them specifically.
Governance — control, policy, audit
☺ Like you’re 10: Think of a school with a headteacher: there are rules about what everyone’s allowed to do, a sign-in book that records who did what, and rooms that are off-limits — so nobody gets to do whatever they want with no one watching.
Organizations need to manage AI use centrally, and the major enterprise assistant platforms are built for it — GitHub Copilot, Anthropic’s Claude (Team and Enterprise), OpenAI’s enterprise offerings, and others all provide the same building blocks:
- A control plane. An admin console — Copilot’s Agent HQ and enterprise surfaces, Claude’s admin/organization settings, and comparable dashboards elsewhere — lets orgs manage agents and policies from one place.
- Policy management. Control which features, models, and extensions are enabled for whom (typically on Business/Enterprise tiers across vendors).
- Audit logs. Keep a record of what agents did — for example, a cloud agent’s commit-by-commit trail, or logged tool calls and transcripts — for debugging and compliance.
- Content exclusions. Specify repositories or paths the assistant must not read (Copilot calls these content exclusions; other tools expose similar data-access and workspace scoping controls).
- IP indemnity. Many enterprise plans (Copilot, Anthropic, OpenAI, and others) include indemnification for model output, which matters to legal teams.
Reliability — for non-deterministic systems
☺ Like you’re 10: It’s like riding a bike with training wheels and a spare tire: if one part wobbles or pops, backups and safety rules keep you from crashing — and you always double-check the map before you follow it.
AI systems fail in new ways: a model provider has an outage, a response is malformed, an agent gets stuck, latency spikes. Build for it:
- Fallbacks. A secondary model (or a graceful degraded mode) when the primary is unavailable.
- Retries with backoff for transient errors — but bounded, so you don’t amplify an outage.
- Timeouts. Don’t let a hung model call hang your whole request.
- Circuit breakers. Stop hammering a failing dependency; fail fast and recover.
- Output validation. Never trust output blindly — validate structure (e.g. against a schema) before acting on it.
Responsible AI
☺ Like you’re 10: It’s like being a fair referee in a board game: you tell everyone a robot is playing, you let a grown-up step in for big decisions, and you make sure the rules treat every player the same.
Operating AI responsibly is an ongoing duty, not a launch checkbox:
- Transparency. Tell users when they’re interacting with AI.
- Human oversight. Keep humans in the loop for consequential decisions — the recurring theme of this whole course.
- Bias & fairness. Monitor outputs for biased or harmful behavior across user groups.
- Privacy. Be deliberate about what data goes to models and how it’s retained; respect content exclusions and data-handling rules.
- Accountability. Keep audit trails so decisions can be explained after the fact.
The operating loop
☺ Like you’re 10: It’s like practising a video game level over and over: you play, watch where you keep messing up, change your strategy, and try again — a little better each time, forever.
Production AI runs on a continuous loop, not a one-time deploy:
The loop in action. A cost alert fires — spend tripled overnight:
MONITOR cost up 3x since 02:00, p95 latency up LEARN traces show one agent retrying a flaky tool ~8x per request IMPROVE cap retries at 2 + cache the tool's results + add an eval for the case DEPLOY ship the fix MONITOR cost back to baseline; the new eval guards against a repeat
Watch the system, turn what you learn (especially user feedback and failures) into new eval cases, improve prompts/models/tools, and redeploy. We make this concrete in the capstone’s day-2 section.
Foxy: The helper messed up once in the real world. Do we just fix it and forget it?
Timmy the Turtle: Never forget it. I write down exactly what went wrong and turn that failure into a permanent eval — a test the helper must pass forever after.
Professor Owl: That’s the flywheel: every real mistake becomes a guardrail, so the same slip can’t sneak back in.
Foxy: And if the fix does something risky, like spending money or deleting things?
Timmy the Turtle: Then I stop and ask a human first. Owl’s rule is simple — big actions wait for a person to say yes.
The applied engineering craft behind these operations gets dedicated lessons in the AI Engineering (Applied) track — especially Evaluating AI Systems and Cost, Latency & Observability.
(1) Name three cost-control levers for an AI feature. (2) Which governance tools let an org control and audit an assistant’s use — across Copilot, Claude, and comparable platforms? (3) List four reliability patterns for AI systems. (4) Why does “validate the output” matter more for AI than for deterministic services?
Check your answers
- Cost-control levers: Any three of — right-size the model (route by difficulty instead of always using a frontier model), cache identical or near-identical requests, cap agent iterations so a confused loop can’t run forever, set per-day and per-user budgets with alerts, and watch agentic multipliers where multi-step or multi-agent runs multiply cost.
- Governance tools: The major enterprise assistant platforms provide the same building blocks — a control plane (admin console like Copilot’s Agent HQ or Claude’s admin/organization settings), policy management over which features/models/extensions are enabled, audit logs recording what agents did, content exclusions scoping which repos or paths the assistant can read, and IP indemnity for model output.
- Reliability patterns: Fallbacks (a secondary model or graceful degraded mode when the primary is down), retries with bounded backoff for transient errors, timeouts so a hung call can’t hang the whole request, circuit breakers that fail fast on a failing dependency, and output validation against a schema before acting.
- Validating output: Because an AI “service” is probabilistic, not deterministic — the same input can yield a malformed or wrong response — so you must validate its structure (e.g. against a schema) before acting, as rigorously as you guard its uptime; a deterministic service returns predictable output that doesn’t need the same per-response scrutiny.