Building on the Gemini API

This is where you build your own product on Google’s Gemini models rather than living inside the Gemini app or the Gemini CLI. It’s the Google-track counterpart to Building with Claude — the generateContent call, native multimodal input, function calling, the agentic loop, and the levers (streaming, structured output, a very large context window) that make a real app reliable.

Two front doors: AI Studio vs Vertex AI

☺ Like you’re 10: There are two doors into the same kitchen. One is a side door you can walk through in a minute to start cooking (AI Studio). The other is the big loading-dock door with badges, cameras, and inventory logs — slower to set up, but built for a whole restaurant (Vertex AI). Same chefs behind both.

Google gives you two ways to reach the exact same Gemini models, and picking the right one early saves pain later. The Gemini Developer API — the one you get from Google AI Studio — is the fast on-ramp: grab an API key, install the SDK, and you’re calling the model in minutes. Vertex AI is the same models wrapped in Google Cloud: IAM permissions, regional data residency, VPC controls, logging, and MLOps tooling — the enterprise front door.

The good news: the modern Google Gen AI SDK is designed so the same code targets either backend — you flip a flag (and swap key-auth for Cloud auth) to move a prototype from AI Studio to Vertex AI. So prototype through the side door, then promote to the loading dock without a rewrite. (This mirrors how Claude offers its API directly and via Bedrock/Vertex, and how OpenAI offers its API directly and via Azure — same models, different governance wrappers.)

The generateContent call — and native multimodal input

☺ Like you’re 10: Talking to Gemini is like handing a smart friend a folder. Usually the folder has a note in it — but you can also drop in a photo, a voice clip, or a short video, and your friend reads all of it at once before answering. Same folder slot every time.

Everything Gemini does over the API goes through one core method: generateContent. You send contents — a conversation as a list of turns, each with a role (user or model) and one or more parts — plus optional config, and you get a reply back. The big Gemini differentiator is that a part isn’t only text: images, audio, PDFs, and video are first-class parts in the very same request. Multimodality isn’t bolted on; it’s the native shape of the API.

from google import genai

client = genai.Client()                         # reads GEMINI_API_KEY from the environment

resp = client.models.generate_content(
    model="gemini-flash-latest",                   # pick a tier; version names move — true at time of writing
    contents="Explain MCP in one sentence.",
)
print(resp.text)

That string is shorthand for a single user turn with one text part. The full shape is a list of turns, and any turn can mix text with other media — here, a question about an image:

from google import genai
from google.genai import types

client = genai.Client()
img = client.files.upload(file="chart.png")     # large media → upload once, reference by handle

resp = client.models.generate_content(
    model="gemini-flash-latest",
    contents=[
        img,                                    # an image part
        "What trend does this chart show, and what’s the Q4 number?",   # a text part
    ],
    config=types.GenerateContentConfig(
        system_instruction="You are a concise data analyst.",
    ),
)
print(resp.text)

Note the split from Claude/OpenAI vocabulary: Gemini uses contents (not messages), roles are user/model (not user/assistant), and the steering prompt is system_instruction in the config. The ideas are identical — the concept lives provider-neutrally in How models work and Prompting — but the field names differ, so read Google’s spelling carefully. For a deeper tour of image/audio/video handling, see Multimodal.

Function calling: giving Gemini tools

☺ Like you’re 10: A tool is a labelled button you hand Gemini — the label says what it does and what to type first. Gemini can’t press it itself, so instead it hands you a slip saying “please press this button with these words.” You press it, tell it what happened, and it carries on.

Gemini calls this function calling. You describe your functions as declarations — a name, a description, and a JSON-schema of parameters — and pass them as tools. When Gemini decides a function would help, it doesn’t run anything; it returns a functionCall part naming the function and its arguments. Your code runs the real function and sends the result back as a functionResponse part. The model’s description text is what it reads to decide when to call — so write it well.

from google.genai import types

get_weather = types.FunctionDeclaration(
    name="get_weather",
    description="Get the current weather for a city. Use for any 'weather in X' question.",
    parameters={
        "type": "object",
        "properties": {"city": {"type": "string", "description": "City name, e.g. 'Pune'"}},
        "required": ["city"],
    },
)
tools = [types.Tool(function_declarations=[get_weather])]

# When the reply contains a functionCall part:
#   1. read part.function_call.name and .args
#   2. run your real function with .args
#   3. append a Part with a functionResponse (same name, your result)
#   4. call generate_content again — Gemini continues with the result

The SDK can also skip the boilerplate: hand it a plain Python function with type hints and a docstring, and it will build the declaration and execute the call for you (“automatic function calling”). That’s convenient for scripts, but for production you usually want the manual loop so you control what actually runs — the same “programmatic over prompt-based” discipline from across the course. Gemini also offers built-in tools like Google Search grounding and code execution that run server-side.

The agentic loop

☺ Like you’re 10: It’s a board game of turns. Gemini either says “your turn — go fetch me this” or it just writes the final answer. You keep fetching whatever it asks for and handing it back, and you stop only when it stops asking and gives the answer.

You don’t ask Gemini to “be an agent.” You run a loop: send the conversation, look at the reply, and if it contains a function call, run the tool, append the result, and call again — otherwise you’re done. This is the exact same pattern as the Claude and OpenAI loops and the provider-neutral one in Agentic AI. The only Gemini-specific detail is what to check: instead of Claude’s stop_reason == "tool_use", you inspect the reply’s parts for a functionCall.

Structured output: JSON and response schemas

☺ Like you’re 10: Instead of letting Gemini write a messy paragraph, you hand it a form with labelled blanks and say “fill exactly these in.” Because it’s a real form, the answer comes back tidy every time — not a paragraph you have to untangle.

When your app needs data, not prose, don’t beg the prompt for JSON and hope — constrain it. Gemini lets you set a response_mime_type of application/json and, crucially, a response_schema describing the exact shape you want. The model is then guided to emit JSON matching that schema, so you can parse the result directly instead of scraping it out of prose.

from google import genai
from google.genai import types

client = genai.Client()
resp = client.models.generate_content(
    model="gemini-flash-latest",
    contents="Extract the person: 'Ada Lovelace, born 1815, mathematician.'",
    config=types.GenerateContentConfig(
        response_mime_type="application/json",
        response_schema={
            "type": "object",
            "properties": {
                "name": {"type": "string"},
                "born": {"type": "integer"},
                "role": {"type": "string"},
            },
            "required": ["name", "born", "role"],
        },
    ),
)
print(resp.text)   # → {"name": "Ada Lovelace", "born": 1815, "role": "mathematician"}

This is the Gemini spelling of the same principle as Claude’s schema-shaped tool and OpenAI’s structured outputs: enforcing structure in code beats hoping the prompt is obeyed. Still validate what comes back and re-prompt on a mismatch — a schema strongly constrains the model but your parser is the final gate.

Streaming and the “thinking” config

☺ Like you’re 10: Two dials. One makes the answer appear word-by-word so it feels instant instead of waiting for the whole thing (streaming). The other lets Gemini scribble on scratch paper before answering — more scribbling for hard problems, none for easy ones (thinking).

Streaming. For any chat UI, call the streaming variant and print chunks as they arrive over Server-Sent Events, so the reply feels instant instead of appearing all at once after a pause.

for chunk in client.models.generate_content_stream(
        model="gemini-flash-latest",
        contents="Write a haiku about beavers."):
    print(chunk.text, end="")

Thinking. Google’s recent Gemini models are “thinking” models: they can spend internal reasoning tokens before the final answer, which lifts quality on hard, multi-step problems. You control this with a thinking budget in the config — spend more on genuinely hard tasks, and on some tiers dial it down (or off) for cheap, latency-sensitive calls where deep reasoning is wasted. The underlying idea — that letting a model reason before answering improves hard tasks, at the cost of tokens and latency — is covered provider-neutrally in Reasoning; the Gemini-specific lever is that you set the budget explicitly.

The huge context window + embeddings (and RAG)

☺ Like you’re 10: Gemini has an unusually big desk — you can lay out a whole stack of books at once and it reads across all of them. That’s amazing, but re-laying the whole stack for every single question gets expensive, so for a giant, changing library you still keep an index and fetch just the pages you need.

A standout Gemini trait is its very large context window — long enough (on the order of a million-plus tokens, true at time of writing) to hold entire books, long codebases, or hours of transcript in a single call. That genuinely changes what you can do: for a handful of documents that fit, you can just paste everything in and ask across it — the “long context” option from Retrieval & RAG. Pair it with context caching to store a big, reused prefix (a long manual, a codebase) so you don’t re-pay to process it on every call.

But a big window is not a replacement for retrieval. When your knowledge base is large, changing, or private, you still want RAG: fetch only the relevant chunks and put those in the prompt. Google gives you the pieces for both — a Gemini embedding model to turn text into vectors for semantic search, and (on Vertex AI) managed Vertex AI Search / RAG tooling so you don’t hand-roll the retrieval pipeline.

The decision is the same one from Retrieval & RAG — “how big is it, and how often does it change?” Gemini’s huge window just moves the line: more cases fit in long context than with a smaller model. But for a giant, evolving corpus, embeddings-plus-RAG still wins on cost and freshness.

Tokens, cost, and when to build on the API

☺ Like you’re 10: Every word, pixel, and second of audio you send counts like coins into a meter, and the reply costs coins too. A bigger, smarter model charges more per coin; a smaller, faster one is cheaper. So use the small one for easy jobs and only call the big one when you truly need it.

You pay by the token — Gemini bills input and output tokens, and because it’s natively multimodal, images, audio, and video are also metered as tokens (Google publishes conversion rates per media type). Thinking tokens count too. Three habits keep the bill sane:

• Right-size the model. Use a Flash-tier model for high-volume, latency-sensitive, or simple work, and step up to a Pro-tier model only for the hardest tasks. Most apps are a mix, routed per request.
• Cache the stable prefix. If a long system instruction, manual, or codebase repeats across calls, use context caching so you don’t re-pay to process it every time.
• Tune the thinking budget. Deep reasoning is powerful but not free — turn it down for easy calls.

When should you build on the API at all rather than just using the Gemini app or the CLI? Reach for the API when you need to embed Gemini in your own product, run it programmatically at scale, wire in your own tools and data, or enforce structure and governance the chat app can’t. If a human is happily doing the task in a chat window, you probably don’t need the API yet. The moment it needs to run unattended, inside your software, or against your systems — that’s the API’s job.

The broader tradeoffs — build vs buy, which provider, cost and lock-in — sit in The AI landscape and Ecosystem. For turning any of this into a reliable service (evals, monitoring, rate limits, safety), see Production & Ops, AI pipelines, and AI security; for the deeper agent patterns, Building agents and Multi-agent.