Cost, Latency & Observability

A feature that works in a demo isn’t shipped — it’s demoed. Shipping means it stays fast enough that people don’t leave, cheap enough that the finance team doesn’t panic, and transparent enough that when something goes wrong you can see inside and fix it. This lesson is about the three dials every foundation-model app lives and dies by — money, speed, and observability — and how watching them turns a one-off launch into a system that keeps getting better.

The token economy — what you pay for, and the clocks that matter

☺ Like you’re 10: The helper charges you by the word — both the words you say to it and the words it says back. And it doesn’t answer all at once: there’s a pause before the first word appears, then words come out one at a time like a printer. Two different clocks to watch.

Every foundation-model API bills the same basic way: you pay per token, where a token is a chunk of text (roughly ¾ of a word). There are two prices, and they’re usually different:

• Input tokens — everything you send in: the system prompt, the conversation history, any retrieved documents, tool definitions, the user’s message. Big prompts cost real money on every call.
• Output tokens — everything the model generates back. Output is typically priced higher than input, so a chatty model that rambles for 800 tokens when 80 would do is quietly doubling your bill.

Your cost per request is simply (input_tokens × input_price) + (output_tokens × output_price). That means the two biggest levers on cost are how much you stuff into the prompt (the whole point of context engineering) and how long the answer runs. Neither is free, and both are under your control.

Speed has two clocks too, and confusing them is a classic mistake:

• Time-to-first-token (TTFT) — how long the user waits before anything appears. This is the "loading spinner" feeling, and it’s what dominates perceived speed. Big inputs push TTFT up because the model has to read everything before it starts writing.
• Tokens per second (throughput) — how fast text streams once it’s started. A long answer at slow tokens/sec still feels sluggish even if TTFT was quick.

Caching — stop paying twice for the same thing

☺ Like you’re 10: If someone asks you the exact same question they asked five minutes ago, you don’t work it out again from scratch — you just repeat your saved answer. Caching is the app keeping a little notebook of "I’ve seen this before" so it can skip the expensive part.

Caching means reusing work you already did instead of paying for it again. In model apps there are three flavors, from strictest to loosest match:

• Exact-response cache. If the identical request comes in again, return the stored answer — zero tokens, near-zero latency. Great for popular, deterministic queries (an FAQ, a fixed lookup). Fragile: one different character and it misses.
• Prompt (prefix) caching. Many providers let you mark a long, unchanging prefix — your big system prompt, tool definitions, a fixed document — so the model doesn’t re-process it on every call. You still generate a fresh answer, but the expensive "reading the prefix" step is discounted and faster. This is the single easiest win when you have a large, stable system prompt.
• Semantic cache. Match on meaning, not exact text. Embed the incoming question (the same embedding trick used in retrieval), and if it’s close enough to a question you’ve already answered, reuse that answer. "What’s your refund window?" and "how long do I have to return something?" hit the same cached reply.

The catch — and it’s a big one — is invalidation. A cache is a snapshot, and snapshots go stale. If your refund policy changes but the cache still holds the old answer, you’ll confidently serve last month’s policy. Every cache needs an expiry (a time-to-live) and a way to clear entries when the underlying facts change. A semantic cache adds a second risk: set the "close enough" threshold too loose and you’ll serve a saved answer to a question that only looked similar.

Model routing & cascades — right-sizing every request

☺ Like you’re 10: You don’t call the world’s top surgeon to put on a bandage. Most questions are easy, so send them to a small, cheap, fast helper — and only escalate to the big expensive genius when the little one is out of its depth.

Not every request needs your most powerful (and most expensive, slowest) model. Model routing sends each request to the right-sized model. The most common pattern is a cascade: try a small model first, and only escalate to a larger one when needed.

• Cheap-model-first. A small model handles the easy majority — classification, short factual answers, formatting — at a fraction of the cost and latency.
• Escalate on need. If the small model’s answer fails a check (low confidence, doesn’t match a required schema, an eval gate, or the task is flagged hard), retry the same request on the larger model.
• Route by task, not just size. Sometimes the "right" model is the one specialized for the job — a cheaper model for extraction, a stronger one for multi-step reasoning.

A related lever is batching: when the work isn’t interactive (nightly summaries, bulk classification, back-filling labels), send many requests together. Providers often offer a cheaper, higher-latency batch tier precisely because you’ve told them "I don’t need this in the next second." Trading latency you don’t need for money you’d rather keep is one of the best deals in the stack.

Latency as UX — perceived speed and hiding the slow parts

☺ Like you’re 10: Waiting for a whole story feels forever; watching the words appear as they’re written feels quick, even if it takes the same total time. And if part of a job is slow and boring, do it in the back room so the customer up front never has to wait for it.

Here’s the trick that makes model apps bearable: latency you can see is latency you can hide. The single most effective UX move is streaming — showing tokens the instant they’re generated instead of waiting for the whole answer. Total time is unchanged, but perceived latency plummets because the user gets feedback immediately (this is why TTFT matters so much more than total duration for how fast something feels).

Beyond streaming, you shift slow work out of the user’s way:

• Background / async work. If a task takes 30 seconds — a long report, a batch of documents — don’t make the user stare at a spinner. Kick it off, return immediately, and notify them when it’s done. Interactive-fast and thorough-slow are different lanes; keep them separate.
• Optimistic and progressive UI. Show a skeleton, a "thinking…" state, or partial results early. Perceived progress buys patience.
• Do less on the hot path. Pre-compute retrieval, cache the prefix, trim the prompt. Every millisecond you remove from the interactive request is one the user never waits for.

Observability — tracing so you can see inside

☺ Like you’re 10: When a toy breaks, you can’t fix it if it’s a sealed black box. Observability is putting a little camera on every step inside, so when the answer comes out wrong you can rewind and see exactly which step went sideways — and how much each step cost and how long it took.

A real feature is rarely one model call. It’s a chain: build the prompt, retrieve documents, call the model, maybe call tools, maybe loop, then format the output. When something goes wrong — a bad answer, a huge bill, a slow response — you need to see which step was responsible. That’s what tracing gives you.

A trace is the full record of one request, broken into spans — one span per step. Each span carries telemetry: how long it took, how many tokens it used, what it cost, and its inputs and outputs. Add them up and the trace tells you the total cost and latency of the request and exactly where each went.

Notice what the trace reveals at a glance: the model call dominates both the time and the cost, retrieval is cheap, and the tool call is a minor tax. Without tracing you’d be guessing which knob to turn; with it, you know exactly where to spend your optimization effort. Tracing is typically wired in with an SDK that wraps each step, or via an open standard so traces flow into whatever dashboard you already use.

# Pseudocode: wrap each step in a span; spans nest into one trace
with trace("answer_request") as t:
    with t.span("prompt_build") as s:
        prompt = build_prompt(user_msg, history)
    with t.span("retrieval") as s:
        docs = retriever.search(user_msg, k=4)
        s.record(cost=embed_cost, latency=s.elapsed)
    with t.span("model_call") as s:
        reply = model.generate(prompt + docs)      # streamed
        s.record(in_tokens=reply.in_tok,
                 out_tokens=reply.out_tok,
                 cost=token_cost(reply),
                 latency=s.elapsed)
    t.finish()   # trace now knows TOTAL cost, latency, tokens per request

The feedback loop — production traces feed the evals

☺ Like you’re 10: Every real question your app gets is a free practice question for next time. Save the tricky ones into your quiz pile, and each new version has to pass the quiz before it ships. That’s how it keeps getting better instead of secretly getting worse.

This is where the whole track closes into a circle. The traces you collect in production aren’t just for firefighting — they’re the richest source of real test cases you’ll ever have. The loop:

1. Capture. Every request leaves a trace with its inputs, outputs, cost, and latency.
2. Curate. The interesting ones — failures, low-confidence answers, thumbs-down feedback, surprising costs — get pulled out and added to your eval set.
3. Guard. Now every prompt tweak, model swap, or routing change must pass those real cases before it ships. Regressions get caught before users feel them.
4. Repeat. New production traffic surfaces new edge cases, which become new evals. The system compounds.

Alongside the eval loop you run ongoing monitoring and alerting: dashboards on cost-per-request, TTFT, error rates, and cache hit-rate, with alerts when any of them breaches a threshold (a sudden cost spike usually means a prompt got bloated or the cache stopped hitting). And you watch for drift — the slow, silent kind where nothing errors but quality quietly slides: users start asking different questions, a provider updates a model, your retrieved docs go stale. Drift doesn’t throw an exception; it just makes your feature worse a little at a time, which is exactly why the traces-to-evals loop is what catches it.

Pitfalls

☺ Like you’re 10: Three ways this goes wrong: running with your eyes shut (no cameras inside), keeping an answer past its expiry date (stale notebook), and being so clever about picking helpers that all the checking makes it slower than just answering.

The failures here are predictable, and all three come from optimizing without measuring:

• No tracing = flying blind. The biggest one. Without spans carrying cost and latency, you can’t tell whether the bill spike is retrieval or output length, or whether "slow" is TTFT or throughput. You’ll optimize the wrong thing, or worse, change something and never know if it helped. Tracing is not optional infrastructure — it’s the instrument panel.
• Cache-invalidation bugs. Caching is easy to add and easy to get wrong. Serve a stale answer after the underlying facts changed and you’ve got a confident, authoritative-looking mistake — worse than no cache at all. Set sensible expiries, clear entries when sources change, and don’t set a semantic-match threshold so loose that near-misses get the wrong saved reply.
• Over-routing that adds latency. A cascade is a win only if the routing/checking overhead is cheaper than what it saves. If every request pays for an extra classifier call, a confidence check, and frequently escalates anyway, you’ve added latency and cost to reach the same big model. Route only when the savings clearly beat the overhead — and measure it in your traces.

The common thread: every one of these is invisible until you measure it, and obvious once you do. Cost, latency, and observability aren’t a final polish step — they’re how you keep a shipped feature both alive and improving. And that improvement runs through the loop we just built: production traces become evals, evals gate every change, and the feature that shipped back where this tier began, with evals, keeps getting better all the way through this one. The tier ends where it began — measuring, so you can improve. For the broader operational picture beyond a single feature, keep going in Production & Ops.