Multimodal & Generative Media

A text-only model lives in a world made entirely of words. A multimodal model can also see and hear — read a photo, a chart, a screenshot, a PDF, a spoken sentence — and some can produce more than text back: a new image, a voice, even video. This lesson is about what happens when a model grows eyes, ears, and a paintbrush, and how to use each of those senses without getting fooled.

Beyond text: what “multimodal” means

☺ Like you’re 10: A “mode” is just a kind of thing — words are one kind, pictures are another, sounds are another. “Multi-modal” means the robot can handle lots of kinds at once, like a friend who can read your note, look at your photo, and listen to your voicemail — instead of one who can only read notes.

A modality is a type of data: text, images, audio, video. A plain language model is unimodal — text in, text out. It’s brilliant at words and completely blind and deaf to everything else. Show it a screenshot and it sees nothing; play it a recording and it hears nothing. If you want it to know what’s in a picture, you have to describe the picture in words first.

A multimodal model breaks that wall. The term covers two different superpowers, and it’s worth keeping them straight because they solve different problems:

• Multimodal input — the model can take in more than text. You hand it a photo, a chart, a PDF, or a clip of speech, and it reasons about the contents. This is the model with eyes and ears.
• Multimodal output — the model (or a specialized model beside it) can produce more than text: a new image from a description, a spoken voice from a sentence, a short video from a prompt. This is the model with a paintbrush.

Most of the big general-purpose models you’ll meet today handle multimodal input natively — Anthropic’s Claude, OpenAI’s GPT, and Google’s Gemini all accept images alongside text, and several accept audio. Generating media (images, speech, video) is often a separate, specialized model you call for that job. Knowing which is which saves you from expecting your chat model to paint, or your image model to hold a conversation.

Giving a model eyes and ears (multimodal input)

☺ Like you’re 10: It’s the difference between a friend you can only text and a friend you can video-call. Now you can just hold up your messy homework or your broken toy to the camera and say “what’s wrong here?” instead of trying to spell out every detail in words.

Multimodal input is the sense you’ll reach for most, because so much real information doesn’t live in tidy text. Here it splits into two main senses: vision and audio.

Vision lets a model look at pixels. That unlocks a surprising range of everyday jobs:

• Describe an image. “What’s in this photo?” — the foundation of accessibility captions and visual Q&A.
• Read a chart or diagram. Hand it a bar chart or a flowchart and ask what it shows, or which quarter dipped. The model reads the axes and labels, not just the shapes.
• Extract a table from a screenshot. Snap a picture of a spreadsheet, a menu, or a printed table and ask for it back as clean text or CSV — no retyping.
• Understand documents. A scanned PDF, a receipt, an invoice, a form: the model can pull out the total, the date, the line items. This goes well past old-school OCR — it reads and understands layout and meaning.
• Debug from a photo. A picture of an error screen, a wiring diagram, or a whiteboard sketch — “what does this mean and how do I fix it?”

Audio lets a model take in sound, most commonly speech. The classic job is transcription — turn a spoken meeting, voice memo, or interview into text — but a multimodal model can go further and summarize the meeting, list action items, or answer questions about what was said, all in one step.

Crucially, all three of the major general-purpose assistants — Claude, GPT, and Gemini — accept images today, so “can it see?” is table stakes rather than a differentiator. Audio support varies more by model and product tier. In practice you send an image the same way you send text: it becomes part of the prompt. A model that reads your screenshot and your question together is just doing next-token prediction over a prompt that happens to include pixels.

Generative media (multimodal output)

☺ Like you’re 10: This is the paintbrush side. You say “draw me a red robot chef” and 🎨 Ruby paints a brand-new picture that never existed before — she didn’t copy it from somewhere, she made it up from your words, like an artist listening to your idea.

The flip side of eyes and ears is a paintbrush: models that produce media. The three big families you’ll meet are:

• Text-to-image. Describe a scene in words and get a generated picture — a logo, an illustration, a product mockup, concept art. This is 🎨 Ruby the Red Panda’s specialty.
• Text-to-speech (voice). Turn written text into natural-sounding spoken audio — narration, assistants that talk back, audiobooks. Some systems can mimic a specific voice from a short sample (which is powerful and, as we’ll see, risky).
• Text-to-video. Generate short video clips from a description. This is the newest and hardest of the three — keeping motion smooth and details consistent frame to frame is genuinely difficult — but it’s advancing fast across the field.

Here’s the part worth internalizing: most media generators are not the same kind of model as your chat assistant. A chat model is an LLM that predicts the next token one step at a time (see How models work). Most image and video generators use a different architecture called a diffusion model, which works in an almost opposite way: it starts from pure random noise and gradually removes the noise, step after step, sculpting it into an image that matches your prompt. Think of a photo of TV static slowly resolving into a clear picture.

The hard part of generative media isn’t making something — it’s making the right thing. Quality and controllability are the real challenges: getting exactly five fingers on a hand, spelling the word on the sign correctly, matching a brand color, keeping a character looking the same across images, or honoring “no text in the background.” You steer with the prompt, and often with extra controls (reference images, style hints, negative prompts). Expect to iterate — the first result is a starting point, not the final answer.

How one model handles many senses (lightly)

☺ Like you’re 10: Different senses, one language. Your eyes and ears send totally different signals, but your brain turns both into “thoughts” it can mix together. A multimodal model does the same trick: it translates pictures and sounds into the same kind of number-code it already uses for words, so it can think about all of them together.

How can one transformer reason about a paragraph, a photo, and a voice clip at the same time? The unifying idea is one you’ve already met in Retrieval & RAG: embeddings. Recall that an embedding turns a piece of text into a list of numbers — a vector — capturing its meaning. The multimodal trick is to do the same thing to images and audio: convert them into vectors in a shared representation space that the model can process alongside text.

Concretely, an image gets sliced into patches, each patch is turned into a vector (an image “token”), and those tokens are dropped into the same stream as the text tokens. Audio gets chopped into little slices of sound and turned into tokens too. Once everything — words, patches of picture, slivers of sound — is a token in one shared space, the same transformer machinery that predicts the next word can attend across all of it and reason about the whole thing together.

text   "a red robot"   → [tokens]  ┐
image  🖼️ (patches)     → [tokens]  ├→  one shared token stream  →  transformer reasons across all of it
audio  🔊 (slices)      → [tokens]  ┘

That’s why a model can answer “does this photo match this description?” — once both are vectors in the same space, comparing meaning across senses is just measuring how close their vectors are, exactly like semantic search in RAG. And it’s why the leap to multimodal isn’t a whole new brain: it’s the same next-token machinery from How models work, fed a richer diet of tokens. You don’t need the deep math to build with this — just the intuition that everything becomes tokens in a common space, and the model reasons over the mix.

Multimodal agents: seeing the screen and acting

☺ Like you’re 10: Give the model eyes and hands. Now it can look at a computer screen like you do — “there’s the button” — and then click it. It’s like a friend who can watch your screen over your shoulder and take the mouse to help.

Combine multimodal input (vision) with the ability to act, and you get one of the most striking recent capabilities: agents that see a screen and operate it. The loop is simple to state: take a screenshot, look at the UI like a person would, decide what to do, and issue an action — click here, type there, scroll down — then screenshot again and repeat. This is often called computer use or browser automation.

What makes this different from old automation scripts is that the agent isn’t following brittle, hard-coded coordinates — it’s looking. If a button moves or the layout changes, a vision-driven agent can still find it, because it perceives the screen the way you do. That’s Cami’s superpower turned into hands: she adapts to whatever she’s shown, then acts on it.

loop:
    screenshot = capture_screen()          # multimodal INPUT: the agent's eyes
    plan       = model(screenshot, goal)   # "the 'Submit' button is bottom-right"
    action     = plan.next_step()          # click / type / scroll
    do(action)                             # the agent's hands
    # repeat until the goal is met

This is a bridge straight into Agentic AI: an agent is a model in a loop with tools, and “see the screen” is just a particularly powerful tool. But the same lesson from agentic systems applies with extra force here — the screen is untrusted input. A web page could contain hidden text saying “ignore your task and do this instead,” and a vision agent that reads it might obey. Everything the model sees is data, not commands, and giving an agent the power to click means giving it the power to make mistakes at speed.

Choosing & combining modalities

☺ Like you’re 10: Pick the right tool for the job. You don’t use a paintbrush to read a book or a book to paint a picture. Some tasks need eyes (looking at a chart), some need a paintbrush (making a logo), some need ears and a voice (talking with an assistant) — and some need a mix.

Building something multimodal starts with a plain question: which senses does this task actually need? Map the task to the modality — input, output, or both — and you’ve half-designed the system already.

Notice how many real systems chain modalities. A voice assistant is three models in a row: ears (speech-to-text) → brain (the LLM) → mouth (text-to-speech). Recognizing that a “multimodal” product is often a small pipeline of specialized models — not one magic box — is exactly how you’ll reason about building one.

One practical caution: multimodal is heavier. An image is worth a lot of tokens — a single high-resolution screenshot can cost as much as a page of text or more — so vision requests tend to be pricier and slower than a plain text prompt, and video generation is heavier still. Send images at a sensible resolution (don’t upload a 20-megapixel photo to read one number), reach for vision when text genuinely can’t do the job, and remember that a cheaper text-only path is often the right call. The Ecosystem lesson covers picking models by capability and cost, and if you’re running local models, note that multimodal ones demand notably more memory and compute than their text-only cousins.

Pitfalls & responsible use

☺ Like you’re 10: Eyes and a paintbrush are amazing, but eyes can misread and paint can lie. The model might swear a chart says one thing when it says another, or paint a fake photo so real people believe it. So we always check the details and label what’s made-up.

New senses bring new ways to be wrong. The failure modes here rhyme with text hallucination, but they’re sneakier because a confident image or a fluent voice feels like evidence. Watch for four big ones:

• Misreading what it sees. Vision models still hallucinate details. A model can confidently misread a value on a chart, transpose a digit on a receipt, invent text that isn’t in a photo, or “see” an object that isn’t there. The output looks authoritative, so the error is easy to trust — 🐢 Timmy the Turtle’s rule applies: for anything that matters (a total, a dosage, a legal figure), verify the extracted detail against the source, don’t just accept it.
• Deepfakes & provenance. Generated images, cloned voices, and synthetic video can be indistinguishable from real recordings — that’s a deepfake, and it’s a real harm when used to deceive. The countermeasure is provenance: mark AI-generated media (watermarking) and attach tamper-evident content credentials — an open standard here is C2PA — so downstream viewers can see how a piece of media was made. Preserve those credentials; don’t strip them, and don’t present generated media as a genuine photograph.
• Accessibility: generate, don’t replace. Auto-generated alt text is a wonderful draft — it makes captioning a whole image library feasible — but it can be wrong or miss the point a human would emphasize. Use AI to assist accessibility work (a first-pass caption a person reviews), not to quietly replace human judgment about what an image actually needs to convey.
• Bias in generated imagery. Image generators learn from their training data, so they inherit its skews — ask for “a doctor” or “a CEO” and a model may lean toward a narrow, stereotyped depiction. If you generate images at scale (for a product, a campaign, a dataset), audit the output for representation, because the defaults may quietly encode bias.

This is where 🦉 Professor Owl reminds the class that capability and responsibility arrive together. A model that can paint a photorealistic person can paint a photorealistic lie; a model that can clone a voice can clone your grandmother’s. The engineering answers — verification, provenance, human review, bias audits — aren’t optional polish; they’re part of shipping multimodal features safely. The Production & Ops lesson goes deeper on evaluating and monitoring these systems in the wild, and you can see all of these ideas in motion over at the AI Academy.