Structured Outputs & Tool Use

A chatbot writes prose for a human to read. But when a model sits inside your program, prose is a liability — your code can’t reliably act on a paragraph. The applied craft here is to make the model behave like an API: constrain its output to a schema you defined, so what comes back is data your code can parse, validate, and act on — and let it request tools in that same structured form. This is what turns “the model said something” into “my program did something.”

Why free text breaks programs

☺ Like you’re 10: If everyone answers a question in their own words, a robot trying to sort the answers gets confused. But if everyone fills in the exact same little form, the robot knows precisely where to look every time.

A model’s default output is free text — fluent, human-friendly, and shaped however it feels like that day. That’s perfect for a human reading a chat window. It’s a nightmare for a program. The moment you want your code to do something with the answer — save it to a database, branch on a value, call another service — you have to turn that paragraph back into structured data. That step is called parsing, and doing it on free text is where things quietly fall apart.

Say you ask a model to extract an order from an email and it replies:

Sure! It looks like the customer wants 3 blue mugs shipped to
Berlin, and they'd like it by Friday. Let me know if you need anything else!

Your code needs quantity=3, item="mug", city="Berlin". To get them out of that sentence you’d write brittle string-matching or a regex — and it breaks the instant the model says “three” instead of “3,” adds a friendly preamble, wraps the JSON in ```` ```json ```` fences, or decides today’s reply needs an emoji. Every one of these is a real, everyday failure:

• Chatty wrappers. “Sure! Here’s the JSON:” before the actual data — now JSON.parse throws.
• Format drift. Same request, different shape each call: sometimes a list, sometimes a paragraph, sometimes keys renamed.
• Ambiguity. “by Friday” — which Friday? What date? Free text hides the precision your program needs.
• Silent wrongness. The parse succeeds but grabs the wrong number, and a bad value flows downstream where it’s much harder to catch.

The jump we’re making in this lesson is from “the model wrote something” (a string a human interprets) to “my code can act on it” (typed fields your program reads without guessing). You don’t get there by writing cleverer parsers. You get there by making the model produce the right shape in the first place.

Getting structure: JSON mode, constrained decoding, and tool schemas

☺ Like you’re 10: Instead of hoping the model fills the form neatly, you can put a frame around its answer so it literally can’t color outside the lines — it’s only allowed to write things that fit your form.

There are three levels of “make it structured,” from a gentle nudge to a hard guarantee. Modern model APIs offer some mix of these under names like JSON mode, structured outputs, or tool/function calling. The mechanics differ by provider, but the ideas are universal:

1. JSON mode (a promise, not a guarantee). You ask the model to reply with valid JSON, often via a flag plus a clear instruction. It usually returns parseable JSON — but “valid JSON” only means it parses; it does not mean the JSON has your fields with your types. You still have to validate.
2. Schema-constrained / grammar-guided decoding (a guarantee of shape). You hand the model a schema — typically a JSON Schema — and the decoding process is constrained so that, token by token, only outputs that conform to the schema are even possible. The model literally cannot emit a missing required field or a string where you asked for a number. This is the strongest structural guarantee: the shape is enforced by construction, not by hoping.
3. Tool / function schemas (structure + intent to act). You describe the tools your program exposes — each as a name, a description, and a parameter schema. When the model wants to use one, it doesn’t run it; it emits a structured request — the tool name plus arguments matching that schema — for your code to execute. It’s the same schema-constrained idea, aimed at doing rather than just returning.

All three lean on the same primitive: a schema that says exactly what a valid answer looks like. Here’s the neutral pattern for “return typed data” — pseudocode, since the exact flag and field names differ by provider:

# NEUTRAL PSEUDOCODE — ask for output matching a schema
schema = {
  "type": "object",
  "properties": {
    "quantity": { "type": "integer" },
    "item":     { "type": "string" },
    "city":     { "type": "string" },
    "due_date": { "type": "string", "format": "date" }
  },
  "required": ["quantity", "item", "city"]
}

result = model.respond(
  prompt = "Extract the order from this email:\n" + email,
  output_schema = schema        # provider enforces the shape
)

# result is now DATA, not prose — your code can act on it directly
save_order(result.quantity, result.item, result.city, result.due_date)

Getting the format to actually stick is a measurable property, not a vibe — “does the output conform to the schema every time?” is exactly the kind of thing you track with evals for format adherence. And the words in your prompt still matter even with hard enforcement: clear instructions make the content better, which is why prompting and schemas work together, not instead of each other.

Designing schemas the model fills correctly

☺ Like you’re 10: A good form has clear labels and checkboxes with only the right options — so it’s almost impossible to fill in wrong. A bad form has a giant blank box labeled “stuff,” and you get back a mess.

A schema isn’t just a validator your code runs afterward — the model reads it and uses your field names, descriptions, and types as instructions. A well-designed schema is half the prompt. The craft is making the schema so clear that the only easy thing to do is fill it in correctly.

• Name fields like you mean it. refund_amount_usd tells the model far more than amt. Descriptive, unambiguous names reduce the guessing.
• Add descriptions. Most schema formats let each field carry a description. Use it to disambiguate: “ISO-8601 date, e.g. 2026-07-02” beats a bare due_date every time.
• Constrain choices with enums. If a value can only be one of a few things, say so: "status": {"enum": ["pending","shipped","cancelled"]}. Now the model can’t invent “in-transit-ish,” and your downstream code has a closed set to switch on.
• Be deliberate about required vs optional. Mark a field required only if you truly always need it. Over-requiring forces the model to fabricate a value it doesn’t have; under-requiring lets it skip something you depend on. Where “no value” is legitimate, allow null explicitly rather than hoping.
• Keep it flat. Deeply nested objects-within-arrays-within-objects are harder for the model to fill reliably and harder for you to validate. Prefer a shallow, wide shape; flatten where you reasonably can.

# A schema that guides, not just validates
{
  "type": "object",
  "properties": {
    "sentiment":  { "enum": ["positive","neutral","negative"],
                    "description": "Overall tone of the review" },
    "issue_area": { "enum": ["shipping","billing","product","other"] },
    "summary":    { "type": "string",
                    "description": "One sentence, under 20 words" },
    "needs_human": { "type": "boolean",
                     "description": "True only if the customer is angry or asks to escalate" }
  },
  "required": ["sentiment", "issue_area", "summary", "needs_human"]
}

Notice how much of the “prompt” now lives in the schema itself: the enums pin the vocabulary, the descriptions set the rules, and needs_human is a clean boolean your routing code can branch on with zero parsing. Good schema design is the applied skill this whole lesson turns on. (For the sibling craft of engineering the surrounding context so the model has what it needs to fill the schema, see context engineering.)

The tool-calling loop

☺ Like you’re 10: The model can’t press buttons itself. So when it needs something done — look up the weather, charge a card — it hands you a filled-in request slip. You do the real action, hand back the result, and it keeps going. Back and forth, like passing notes.

Tool use is structured output pointed at action. The model never runs your code and never touches your database. Instead there’s a loop: the model proposes a tool call as structured data, your code executes it, you return the result, and the model continues with that result in hand. It may propose another call, or produce a final answer. You are always the one holding the keys.

Concretely, the round-trip looks like this — note that you run get_weather, never the model:

# NEUTRAL PSEUDOCODE — the propose → execute → return loop
tools = [{
  "name": "get_weather",
  "description": "Current weather for a city",
  "parameters": { "type": "object",
                  "properties": { "city": { "type": "string" } },
                  "required": ["city"] }
}]

messages = [ user("What should I pack for Berlin today?") ]

while True:
    reply = model.respond(messages, tools=tools)

    if reply.tool_call:                        # 1. model PROPOSES
        args = validate(reply.tool_call.args)  #    check before trusting
        output = run(reply.tool_call.name, args)   # 2. YOUR code EXECUTES
        messages.append(reply.tool_call)
        messages.append(tool_result(output))   # 3. RETURN result to model
        continue                               # 4. loop → model CONTINUES
    else:
        return reply.text                       # final answer, no more calls

This propose→execute→return loop is the engine under agents. It’s worth being precise about how it differs from two neighbours in this course:

• MCP is the protocol for exposing tools to a model over a standard interface — it standardizes how a tool describes itself and gets called, so any host can discover it. The loop above is what consumes those tools once they’re available. MCP answers “how does the tool advertise itself?”; this lesson answers “how does the model use it, turn by turn?”
• Building agents catalogs the build paths — frameworks, SDKs, and no-code routes for assembling an agent. This lesson is the mechanism those paths all run internally: strip any agent framework down and you find this exact loop.

Validation & repair

☺ Like you’re 10: Even with a form, someone might scribble outside a box or leave a blank. So before you act, you check the form. If it’s messed up, you hand it back and say “this part’s wrong, please fix it” — and usually the second try is clean.

Schema enforcement makes malformed output rare, but robust systems don’t assume it’s impossible — especially with plain JSON mode, streaming, or older models. The discipline is validate, then repair: check every tool call and every structured output against your schema before you act on it, and have a recovery path when the check fails.

• Validate on receipt. Parse the output and run it through a real validator (a JSON-Schema validator, or a typed model like a dataclass / Pydantic-style / Zod-style schema in your language). Reject unknown fields, wrong types, out-of-range enums, and missing required keys.
• Repair by re-prompting with the error. The single most effective fix: hand the exact validation error back to the model and ask it to correct just that. “Field quantity must be an integer; you sent ‘three’” gets a clean retry far more often than a blind “try again.”
• Bound your retries. Cap attempts (say 2–3) and fall back gracefully — a safe default, a human handoff, or a clear error — instead of looping forever. Every retry costs a call, which is why this ties into cost & latency ops.
• Make execution idempotent. If a retry might replay a call that partially ran, design tools so running them twice with the same inputs is safe (e.g. an idempotency key on “charge card” so the customer isn’t billed twice). Retries are only safe if repeats don’t compound.
• Handle partial & parallel calls. Streamed output can arrive incomplete — wait for the full call before validating. And a model may propose several tool calls at once; validate and execute each independently, and return each result tagged to its call so the model can match them up.

# NEUTRAL PSEUDOCODE — validate, then repair
for attempt in range(3):
    reply = model.respond(messages, output_schema=schema)
    ok, value, error = validate(reply, schema)
    if ok:
        return value                        # good → act on it
    messages.append(reply)                   # show the model what it sent
    messages.append(user("That was invalid: " + error +
                         ". Return corrected data matching the schema."))
# out of attempts → safe fallback, don't guess
raise NeedsHumanReview(last_error=error)

Validation is also your security boundary. The arguments the model proposes are untrusted input — the model may have been steered by a poisoned document or a crafted user message. Checking types isn’t enough; you also enforce policy: is this city on the allow-list, is this amount within limits, is this user permitted to call this tool at all? That authorization layer is the subject of guardrails, and it belongs on the arguments before execution, every time.

Pitfalls

☺ Like you’re 10: The form only helps if you make a good form and actually check it. A form with a hundred nested boxes confuses everyone, and a form nobody reads before acting on it is worse than no form at all.

Structured output removes a whole category of bugs — and quietly introduces a few new ways to shoot yourself in the foot. The common ones:

• Over-nested schemas. Objects inside arrays inside objects, four levels deep. The model fills these less reliably and your validation gets gnarly. Flatten aggressively; if a shape is genuinely complex, break the task into smaller calls rather than one giant schema.
• Hallucinated fields & arguments. A model can invent a field you didn’t define, or pass an argument value that doesn’t exist (a made-up order_id, a tool name you never registered). Plain JSON mode won’t stop this — schema enforcement and strict validation will.
• Skipping validation because “it worked in testing.” The one call in a thousand that comes back malformed is the one that corrupts your database or crashes production at 3am. Validation is not optional polish; it’s the contract. Test that your format holds up with evals, not just a few happy-path runs.
• Trusting arguments blindly. The deadliest one. The model proposes delete_account{ user_id: "…" } and your code just runs it. Model-proposed arguments are untrusted — enforce authorization and policy on every call before executing, per guardrails. Structure guarantees the shape of a request, never its legitimacy.
• Requiring fields the model can’t know. Mark something required that isn’t in the input and you force fabrication. If a value may be absent, allow null and handle it — don’t make “I don’t know” impossible to express.