Computer Vision & NLP
Before a single model could see and read, two whole fields did those jobs separately: computer vision taught machines to make sense of pixels, and natural language processing taught them to make sense of words. Today’s multimodal models quietly absorbed enormous chunks of both — but the older toolkits didn’t vanish. This page is your orientation: what these fields actually do, how they fold into the LLM-and-agent world you already know, and when a small, specialized model still beats a giant general one.
Computers used to have two separate helpers: one that was good at looking at pictures and one that was good at reading sentences. The looking-helper could spot a cat in a photo; the reading-helper could tell if a review was happy or angry. Now we have one clever helper that can do both at once — but the old specialists are still around, because sometimes a tiny helper that does one thing super fast beats the big brainy one.
Computer vision: teaching machines to see
☺ Like you’re 10: A photo is just millions of tiny colored dots. Computer vision is the trick of turning “here are a bunch of dots” into “that’s a dog, and it’s standing on the left, and it’s wearing a red collar.”
Computer vision (CV) is the field of getting computers to extract meaning from images and video. To a machine, an image is just a grid of numbers — pixel brightness and color values. CV is the set of techniques that turns that raw grid into structured understanding. It’s a whole discipline, but most of it clusters into a few classic tasks, ordered here roughly from “what” to “where” to “exactly which pixels”:
| Task | Question it answers | Output | Everyday example |
|---|---|---|---|
| Image classification | What is this a picture of? | One (or a few) labels for the whole image | “This photo contains a cat” · sorting a photo library |
| Object detection | What objects are here, and where? | A box + label around each object | Boxes around every car and pedestrian in a dashcam frame |
| Segmentation | Which exact pixels belong to what? | A pixel-level mask per object/region | Cutting a person out of the background · a self-driving car tracing the drivable road |
| OCR (optical character recognition) | What text is in this image? | Machine-readable text + positions | Reading a receipt, a license plate, or a scanned form |
Two more show up constantly: pose estimation (finding the joints of a body or hand — think fitness apps and motion capture) and image generation, the reverse direction, where a model produces pixels from a description (covered as diffusion in Deep learning and used in multimodal tools). The through-line: CV is about going from pixels to meaning — and each task hands you a different shape of meaning, from a single label to a per-pixel map.
The CV toolkit: from CNNs to vision transformers
☺ Like you’re 10: Early “seeing” machines learned to notice edges, then shapes, then whole objects — like building up from strokes to letters to words. Newer ones instead cut the picture into puzzle pieces and let every piece “talk” to every other piece to figure out what’s going on.
For roughly a decade, the workhorse of CV was the convolutional neural network (CNN). A CNN slides small filters across an image, building up from simple features (edges, corners) in early layers to complex ones (eyes, wheels, faces) in deeper layers. The 2012 breakthrough that kicked off the deep-learning era — a CNN crushing an image-classification benchmark — is exactly the story told in Deep learning. CNNs are still everywhere, especially where speed and small size matter.
Then came the vision transformer (ViT): take the same transformer architecture that powers language models, chop an image into a grid of patches, treat each patch like a “word,” and let attention relate every patch to every other. ViTs (and hybrids) now lead on many benchmarks and are the backbone of the vision half of most multimodal models — which is a big reason one architecture could suddenly handle both pixels and text. The practical toolbox you’ll actually reach for:
| Tool / family | What it’s for | Notes |
|---|---|---|
| OpenCV | Classic image ops — resizing, filtering, edge detection, camera capture, geometry | Not deep learning per se; the Swiss-army knife you glue around models |
| YOLO-style detectors | Fast, real-time object detection (“You Only Look Once”) | The go-to when you need boxes at video frame rates on modest hardware |
| CNN backbones (ResNet, EfficientNet, MobileNet) | General-purpose feature extractors for classification/detection | MobileNet family is tuned for phones and edge devices |
| Segmentation models (U-Net, Mask R-CNN, SAM-style) | Pixel-level masks; “segment anything” promptable models | Medical imaging, photo editing, robotics |
You reach these through frameworks like PyTorch and TensorFlow (see Deep learning) and hubs like Hugging Face and Ultralytics. The mental model: OpenCV moves and cleans pixels; a neural net turns them into meaning.
The transformer didn’t just take over language — it took over vision too. Once an image is “patches you can attend over,” the same core machinery handles text tokens and image patches alike. That shared architecture is why a single multimodal model can look and read at once.
Classic NLP: language before LLMs
☺ Like you’re 10: Long before chatbots, computers handled language by chopping sentences into pieces and running each piece through a little machine — one machine to find names, another to guess if a review was happy or sad. Lots of small specialists instead of one big talker.
Natural language processing (NLP) is the field of getting computers to work with human language. Today “NLP” and “LLM” feel like synonyms, but for decades NLP meant a pipeline of narrow, well-defined tasks, each solved by its own model. Knowing them makes LLMs less magical and helps you name what you actually need. The classic building blocks:
- Tokenization. Splitting text into units (words, subwords, punctuation) — the first step of nearly every pipeline. LLMs still tokenize; it’s where “how many tokens?” comes from in How models work.
- Embeddings. Turning words or text into vectors so similar meanings land nearby — the same idea you met in Retrieval & RAG. Early versions (word2vec, GloVe) gave each word one fixed vector; modern ones are context-aware.
- Named-entity recognition (NER). Tagging the people, places, organizations, and dates in text. (“Acme Corp shipped to Berlin on June 3.”)
- Sentiment analysis. Classifying text as positive, negative, or neutral — the classic “is this review happy?” task.
- Parsing & part-of-speech tagging. Labeling each word’s grammatical role and how words relate — noun, verb, subject, object.
- TF-IDF. A pre-neural way to score which words matter in a document (frequent here, rare overall) — still the backbone of a lot of keyword search and, blended with embeddings, of hybrid retrieval.
The common thread: each is a specific transformation from text to structure — a label, a tag, a score, a vector. You’d chain several together to build, say, a support-ticket router. The transformer is what eventually let one model do most of these at once instead of a pipeline of separate ones.
The classic NLP toolkit
☺ Like you’re 10: These are the toolboxes people used to build language machines — one is fast and industrial, one is great for learning, and one is the big shared shelf where everyone puts their models.
Three libraries define the pre-LLM (and still very much alive) NLP world. They’re worth knowing because production text systems often still use them for the cheap, deterministic parts — and because a lot of “LLM apps” quietly lean on them for preprocessing.
| Library | Sweet spot | Vibe |
|---|---|---|
| spaCy | Fast, production-grade pipelines: tokenization, NER, POS tagging, parsing at scale | Industrial — built for speed and shipping |
| NLTK | Teaching, experimentation, classic algorithms | Academic — the textbook toolkit |
| Hugging Face Transformers | Downloading and running pretrained transformer models for any task | The shared model hub — bridge from classic NLP to modern LLMs |
Hugging Face is the connective tissue: it’s where you grab a small, fine-tuned model for sentiment or NER without training anything, and it’s where much of the open-model ecosystem you meet in Local & open models lives. A common modern pattern is to let spaCy do fast, rule-based cleanup and entity extraction, then hand the hard, fuzzy reasoning to an LLM.
How LLMs and multimodal models absorbed all this
☺ Like you’re 10: Instead of a whole team of tiny specialists — one for names, one for feelings, one for spotting cats — we trained one enormous student who read and looked at almost everything. Now you just ask it, and it does the specialist’s job without anyone wiring up a pipeline.
Here’s the plot twist that reshaped both fields. A modern large language model — and especially a multimodal one that also takes images — can do most of those classic tasks out of the box, with no task-specific model, just a prompt. Ask any current frontier assistant (Anthropic’s Claude, OpenAI’s GPT/ChatGPT, Google’s Gemini) to “extract every person and company from this email” and you get NER; “is this review positive or negative?” gives you sentiment; “what’s in this photo, and where are the price tags?” exercises classification, detection, and OCR at once. One general model replaced whole pipelines of narrow ones.
| Classic approach | Modern LLM / multimodal approach |
|---|---|
| Train a separate model per task (NER, sentiment, classification…) | One model; switch tasks by changing the prompt |
| Needs labeled training data for each new task | Often works zero-shot or with a few examples |
| OCR model → parser → business logic pipeline | “Read this receipt and return JSON” in one call |
| Retrain to add a task or language | Just describe the new task in words |
Why did this happen? Scale plus the shared transformer architecture. Once text tokens and image patches both became “things you attend over,” a single model trained on enough of both learned representations general enough to cover tasks it was never explicitly built for. This is the same story as the AI landscape: specialized systems giving way to general-purpose foundation models you steer with instructions. The convenience is enormous — but, as the next section shows, it isn’t always the right call.
When specialized CV/NLP still wins
☺ Like you’re 10: A giant brainy helper is amazing, but slow and pricey to ask. If you need an answer 30 times a second, on a tiny camera, for almost no money, a small helper that only knows one trick beats the genius every time.
A frontier model can do object detection, but that doesn’t mean you should ship it for that. General models are big, comparatively slow, and cost per call; a purpose-built model can be tiny, blazing fast, and nearly free to run. Reach for a specialized CV/NLP model when:
| Situation | Why a specialized model wins |
|---|---|
| Real-time / high frame rate (video analytics, robotics) | A YOLO-style detector runs at dozens of frames per second; round-tripping every frame to a big model can’t keep up or afford it |
| Edge & on-device (phones, cameras, sensors) | A few-megabyte MobileNet or quantized model runs offline with no network, no per-call cost, and better privacy |
| Precise measurement (defect detection, dimensions, medical imaging) | A model trained on your exact task is more accurate and consistent than a generalist, and you can validate it rigorously |
| Cost / volume at scale (millions of classifications a day) | A tiny classifier costs a fraction of a frontier API call; at volume the difference is enormous |
| Narrow, stable task (spam filter, language ID, OCR on one form type) | A small model is cheaper, faster, easier to test, and doesn’t drift when a general model updates |
This is the same trade-off you saw with RAG vs fine-tuning and with local vs hosted models: the biggest, most capable option is rarely the most appropriate for a tight, repetitive, latency- or cost-sensitive job. A frequent winning pattern is a cheap specialist as a first-pass filter, escalating only the hard or ambiguous cases to a big general model.
“The multimodal model can do it” is not the same as “the multimodal model is the right tool.” For real-time, edge, high-volume, or safety-critical measurement work, a small specialized model is often faster, cheaper, more accurate, and easier to validate. Match the tool to the constraints, not to the hype.
How CV, NLP, and LLMs combine in real systems
☺ Like you’re 10: The best setups are teams: a fast little “spotter” watches the camera, and only when something interesting happens does it tap the big brainy helper on the shoulder to explain or decide what to do. Everyone does the part they’re best at.
In practice you rarely pick “specialist” or “generalist” — you compose them. The classic tools become the fast, cheap sensors and preprocessors; the LLM becomes the reasoner and orchestrator that ties results into an action. A few patterns you’ll see, all versions of the same idea:
- Detect → describe → decide. A YOLO detector flags objects in a video feed; an LLM (or multimodal model) is called only on the interesting frames to explain the scene, and an agent decides what to do next.
- OCR/parse → reason. A specialized OCR or spaCy pipeline extracts clean text and entities from a stack of documents; the LLM reasons over that structured output, so you don’t pay a giant model to do rote extraction.
- Cheap filter → expensive escalation. A tiny classifier handles the easy 95% (obvious spam, clear sentiment); only ambiguous cases go to the frontier model. Big cost savings, high accuracy where it matters.
- Tools on a leash. Via MCP and tool-calling, an agent invokes a vision model, an OCR service, or a search index as tools — the specialists become callable skills in an agent or pipeline.
The lesson to carry forward: CV and NLP didn’t get replaced by LLMs — they got absorbed for the easy path and kept for the hard constraints. Knowing the older fields lets you build systems that are cheaper, faster, and more reliable than “send everything to the biggest model.” To go deeper, see Deep learning for the networks underneath, Training models and MLOps for building and shipping specialized models, and Multimodal for how one model sees and reads at once.
Cami the Chameleon: Watch — I switch senses. Seeing mode: I scan this photo and box every car. Reading mode: I pull the names and dates out of this contract. Vision and language, my two native tricks.
Foxy: But… doesn’t the big multimodal model just do all of that in one shot? Why keep the old tools around?
Cami the Chameleon: For a photo or a page? Sure, ask the big model. But watch this security camera — 30 frames a second, all day, on a cheap box with no internet. A tiny detector runs right here for pennies; sending every frame to a frontier model would be slow and broke.
Benny the Beaver: So we wire it up as a team: my little detector spots something odd, then it taps the big model to explain just that one frame. Fast where it needs to be fast, smart where it needs to be smart.
Timmy the Turtle: Let me verify the trade: real-time, on-device, high volume → specialist wins on speed and cost; rare, fuzzy, needs reasoning → escalate to the generalist. Match tool to constraint. Approved.
Take one photo with some text in it (a receipt, a sign, a whiteboard). First, hand it to a multimodal assistant and ask it to “list every object and transcribe all the text as JSON” — that’s classification + detection + OCR in one prompt. Then look up a YOLO or OCR demo (e.g. on Hugging Face) and run the same image through a specialized model. Compare: which was faster? Which was more precise on the text? You just felt the generalist-vs-specialist trade-off firsthand.
(1) Name the four classic computer-vision tasks and what each outputs. (2) What changed when the transformer moved from language into vision, and why did that enable one multimodal model? (3) List three classic NLP tasks an LLM can now do from a plain prompt. (4) Give two situations where a small specialized CV/NLP model beats a frontier multimodal model — and explain why.
Check your answers
- The four classic CV tasks: image classification outputs one (or a few) labels for the whole image; object detection outputs a box plus a label around each object; segmentation outputs a pixel-level mask per object or region; and OCR outputs machine-readable text plus its positions. Each hands you a different shape of meaning, from a single label to a per-pixel map.
- Transformer moving into vision: the vision transformer (ViT) chops an image into a grid of patches and treats each patch like a “word,” letting attention relate every patch to every other — the same core machinery that powers language models. Because both text tokens and image patches became “things you attend over,” one shared architecture could handle pixels and text alike, which is why a single multimodal model can look and read at once.
- Three classic NLP tasks an LLM does from a prompt: named-entity recognition (e.g. “extract every person and company from this email”), sentiment analysis (“is this review positive or negative?”), and text classification/routing — all done by changing the prompt instead of training a separate model per task. (Tokenization, embeddings, and POS/parsing are other valid classic tasks.)
- Two situations where a small specialized model wins: real-time / high frame rate work (e.g. video analytics) — a YOLO-style detector runs at dozens of frames per second, whereas round-tripping every frame to a big model can’t keep up or afford it; and edge / on-device use (phones, cameras, sensors) — a few-megabyte model runs offline with no network, no per-call cost, and better privacy. Other valid cases: precise measurement, cost/volume at scale, and narrow stable tasks. The point is to match the tool to the constraint, since “can” isn’t “should.”