Streaming Responses and Managing Context

Stream Claude's output token by token instead of waiting for the full response to land all at once, then pick up the strategies that keep a long-running conversation from silently outgrowing the context window.

Why streaming changes how your app feels

☺ Like you're 10: It's the difference between watching subtitles appear word by word and staring at a black screen until the whole movie is done rendering.

A non-streaming call to /v1/messages blocks until Claude has generated the entire response, then hands it all back in one JSON payload. Ask for a 2,000-word explanation and your user stares at a spinner for however long that takes — even though the first sentence was ready seconds earlier. Streaming sends each piece of the response back the moment it's generated, so you can start rendering text (or updating a progress indicator) immediately. Total generation time doesn't change; perceived latency drops dramatically, because the user sees continuous progress instead of a blank wait.

Past a certain output size, streaming stops being a UX nicety and becomes an operational requirement. Current models support up to 128K output tokens per request, and the official SDKs will refuse a non-streaming request they estimate will run long enough to risk an idle-connection timeout. Anything with a large max_tokens — long-form generation, big refactors, multi-step agent turns — should default to streaming for that reason alone, independent of the UX win.

How streaming works: Server-Sent Events

You opt in by setting "stream": true on the same POST /v1/messages request you'd otherwise send. Instead of one response body, the API keeps the HTTP connection open and pushes a sequence of typed events over it using Server-Sent Events (SSE) — a simple, one-directional streaming format built on plain HTTP. Your client reads events off that connection as they arrive rather than waiting for it to close.

Each event on the stream has a type. The ones you'll see on a typical text response, in order:

You could parse this raw event stream yourself, but both official SDKs give you a helper that does it for you.

A minimal streaming call

☺ Like you're 10: The .stream() helper is a hose already hooked up to the faucet — you just turn it on and words flow into your print statement, instead of you plumbing it yourself.

The Python and TypeScript SDKs both expose a messages.stream() helper: it opens the connection, accumulates the parsed events into a running message for you, and exposes a simple text_stream (Python) / textStream (TypeScript) iterable you can loop over to print tokens as they arrive. The same call made over raw HTTP is just the usual JSON body with "stream": true added — but the helper below does that plumbing for you, so there's rarely a reason to hand-roll it.

import anthropic

client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from env

with client.messages.stream(
    model="claude-sonnet-5",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Write a short poem about the ocean."}],
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

    # After the loop, the fully-assembled message is available with no extra call
    final_message = stream.get_final_message()

print(f"\n\nTokens used: {final_message.usage.output_tokens}")

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic(); // reads ANTHROPIC_API_KEY from env

async function main() {
  const stream = client.messages.stream({
    model: "claude-sonnet-5",
    max_tokens: 1024,
    messages: [{ role: "user", content: "Write a short poem about the ocean." }],
  });

  for await (const text of stream.textStream) {
    process.stdout.write(text);
  }

  const finalMessage = await stream.finalMessage();
  console.log(`\n\nTokens used: ${finalMessage.usage.output_tokens}`);
}

main();

If you need lower-level access — say, to render thinking and text differently, or to build your own accumulator — pass stream=True directly to messages.create() instead of using the .stream() helper, and iterate the raw events yourself, branching on event.type. The helper is the right default for the common case of printing tokens as they arrive and getting the final message when done.

Context windows in practice

☺ Like you're 10: The context window is the whiteboard in a meeting room — everything said and typed gets written on it, and once it's full, nothing new fits until something gets erased.

Every model has a hard ceiling on how many tokens — input plus output combined — it can hold in a single request. The Messages API is stateless: it has no memory between calls, so on every request you resend the entire conversation history as the messages array. That history, plus your system prompt and any tool definitions, all counts against the context window. A long-running conversation, an agent loop with many tool calls, or a chat app that just keeps appending messages will eventually hit that ceiling.

When a response would overflow the window, the API returns stop_reason: "model_context_window_exceeded" — distinct from max_tokens, which means you hit your own requested output cap. Treat model_context_window_exceeded as a signal that the conversation itself needs to be pruned or summarized before the next request, not just retried with a bigger max_tokens.

Strategies for long conversations

☺ Like you're 10: It's like cleaning out an overstuffed backpack: drop everything before a certain point, boil old stuff down to a sticky note, or only pack what today's hike actually needs.

A 1M-token window is generous, but agentic loops, long research sessions, and customer-support threads that run for hours can still fill it — and even well inside the limit, stuffing the model with everything it's ever seen costs money and can dilute its attention on what actually matters for the current turn. Three complementary strategies keep a long conversation healthy.

Sliding window truncation

The simplest approach: keep only the most recent N turns and drop the rest. It's cheap — no extra API calls — but it's a blunt instrument, since anything outside the window is gone for good, including facts or decisions the user made early on that later turns might depend on.

MAX_TURNS = 20  # keep the most recent 20 user/assistant messages

def append_and_trim(conversation, role, content):
    conversation.append({"role": role, "content": content})
    if len(conversation) > MAX_TURNS:
        conversation[:] = conversation[-MAX_TURNS:]
    return conversation

Periodic summarization

A better-preserving approach: once the conversation crosses a token threshold, ask Claude to summarize the older turns into a few dense sentences, fold that summary into the system prompt, and keep only the most recent handful of turns verbatim. You lose fine detail on old turns but keep the gist — names, decisions, open questions — at a fraction of the token cost.

import anthropic

client = anthropic.Anthropic()
MODEL = "claude-sonnet-5"
SUMMARIZE_AFTER_TOKENS = 6000

conversation = []
running_summary = ""

def total_tokens(messages, system):
    count = client.messages.count_tokens(model=MODEL, messages=messages, system=system)
    return count.input_tokens

def send(user_text):
    global running_summary, conversation
    conversation.append({"role": "user", "content": user_text})
    system_prompt = f"Summary of earlier conversation:\n{running_summary}" if running_summary else None

    if total_tokens(conversation, system_prompt) > SUMMARIZE_AFTER_TOKENS:
        older, conversation[:] = conversation[:-4], conversation[-4:]  # keep last 2 exchanges verbatim
        summary_response = client.messages.create(
            model=MODEL,
            max_tokens=512,
            messages=older + [{
                "role": "user",
                "content": "Summarize the conversation above in a few dense sentences, "
                            "preserving names, decisions, and open questions.",
            }],
        )
        running_summary += "\n" + next(b.text for b in summary_response.content if b.type == "text")
        system_prompt = f"Summary of earlier conversation:\n{running_summary}"

    response = client.messages.create(
        model=MODEL, max_tokens=1024, system=system_prompt, messages=conversation,
    )
    reply = next(b.text for b in response.content if b.type == "text")
    conversation.append({"role": "assistant", "content": reply})
    return reply

Only summarize periodically — once the token count crosses your threshold — rather than on every turn. Summarizing every message adds an extra API call to every request for no benefit, since nothing has grown enough yet to need it.

Only include what's relevant to this turn

The third strategy applies less to chat history and more to reference material: don't resend an entire knowledge base, codebase, or document archive on every request just because the user might need any part of it. Retrieve and include only the documents actually relevant to the current question — the same underlying idea as retrieval-augmented generation, covered in depth on the RAG page. Even for plain conversation history, this generalizes: if turn 3 was about an unrelated topic the user has since moved on from, it doesn't need to ride along in every subsequent request just because it's technically part of the history.

Proactive vs. reactive context management

☺ Like you're 10: It's paying down a credit card before it maxes out, instead of waiting for it to get declined at checkout.