Vision and Multimodal Input

Claude reads pixels the same way it reads text — send it a screenshot, a photo, a chart, or a scanned form alongside your prompt, and it reasons about what's in the image directly.

How to send an image to Claude

☺ Like you're 10: Talking to Claude with a picture is like handing someone a photo instead of describing it out loud — you just need to say where the picture is (the raw bytes, or a link) and what you want done with it.

Images travel in the same messages array as everything else, as a content block of "type": "image" sitting next to "type": "text" blocks inside a user turn. The block's source object tells Claude where the bytes come from — there are two ways to populate it: inline base64-encoded data, or a URL Claude fetches itself.

{
  "role": "user",
  "content": [
    {
      "type": "image",
      "source": {
        "type": "base64",
        "media_type": "image/png",
        "data": "<base64-encoded bytes>"
      }
    },
    {
      "type": "image",
      "source": {
        "type": "url",
        "url": "https://example.com/dashboard.png"
      }
    },
    {
      "type": "text",
      "text": "Compare these two dashboards."
    }
  ]
}

Claude's vision input accepts the common web image formats — JPEG, PNG, GIF, and WebP — so you rarely need to convert anything before sending it. Whichever source type you use, the media_type (for base64) should match the actual file, and image blocks can be freely interleaved with text blocks in any order within the content array.

Here's a complete example that reads a local file, base64-encodes it, and asks Claude to summarize a chart:

import anthropic
import base64

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from env

with open("chart.png", "rb") as f:
    image_data = base64.standard_b64encode(f.read()).decode("utf-8")

message = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=1024,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": "image/png",
                        "data": image_data,
                    },
                },
                {
                    "type": "text",
                    "text": "What trend does this chart show? Summarize in two sentences.",
                },
            ],
        }
    ],
)

for block in message.content:
    if block.type == "text":
        print(block.text)

import Anthropic from "@anthropic-ai/sdk";
import { readFileSync } from "fs";

const client = new Anthropic(); // reads ANTHROPIC_API_KEY from env

const imageData = readFileSync("chart.png").toString("base64");

const message = await client.messages.create({
  model: "claude-sonnet-5",
  max_tokens: 1024,
  messages: [
    {
      role: "user",
      content: [
        {
          type: "image",
          source: {
            type: "base64",
            media_type: "image/png",
            data: imageData
          }
        },
        {
          type: "text",
          text: "What trend does this chart show? Summarize in two sentences."
        }
      ]
    }
  ]
});

for (const block of message.content) {
  if (block.type === "text") {
    console.log(block.text);
  }
}

Practical tips for working with images

A few habits make image prompts noticeably more reliable.

Keep image size reasonable. Very large, high-resolution scans add to your request size and input-token cost without necessarily improving Claude's ability to read them — Claude doesn't need a 20-megapixel TIFF to read a receipt. If you control the capture step (a screenshot tool, a phone camera app), resize or compress before sending; check the current API documentation for exact size and dimension limits, since these are the kind of details that get revised as the platform evolves.

You can send multiple images in one prompt. This is the basis for comparison tasks — before/after screenshots, two product photos, a chart from last quarter next to one from this quarter. Refer to images by their position ("the first image," "the second image") or, more reliably, precede each image with a short text block naming it, so there's no ambiguity about which image an instruction or answer applies to.

Labeled multi-image prompts look like this — two dashboards, one sent as base64, one as a URL, both introduced by name:

{
  "role": "user",
  "content": [
    { "type": "text", "text": "Here is the first image, last month's dashboard:" },
    {
      "type": "image",
      "source": { "type": "base64", "media_type": "image/png", "data": "<base64 data>" }
    },
    { "type": "text", "text": "Here is the second image, this month's dashboard:" },
    {
      "type": "image",
      "source": { "type": "url", "url": "https://example.com/dashboard-this-month.png" }
    },
    { "type": "text", "text": "What changed between the first image and the second image?" }
  ]
}

Realistic use cases

☺ Like you're 10: It's the same as asking a helpful friend "read this receipt for me," "what changed between these two screenshots," or "what's this chart telling us" — except Claude answers by looking, not by you typing everything out first.

Reading a chart or screenshot

Point Claude at a screenshot of a dashboard, a plotted chart, or a UI mockup and ask it to describe trends, spot anomalies, or explain what a control does. This is a fast way to get a second pair of eyes on visual data without manually transcribing it into text first, and it works well combined with a specific question rather than an open-ended "what do you see?"

Extracting data from a scanned document or receipt

A photographed or scanned receipt, invoice, or form is a classic vision use case: ask Claude to pull out line items, totals, dates, and vendor names, ideally into a structured format like JSON so downstream code can consume it directly.

Comparing two images

Send before/after screenshots, two design variants, or two versions of a scanned document in one prompt and ask Claude to describe the differences. Because it can hold both images in context at once, this is a natural fit for visual regression review, design critique, or spotting what changed between two states of the same screen.

Limitations to keep in mind

☺ Like you're 10: Even a sharp-eyed friend can misread a blurry receipt or misjudge exactly where something sits on a page. Claude's the same — great at the big picture, not a laser measuring tool for tiny text or exact coordinates.

Vision is genuinely useful, but it's worth being precise about what it isn't.