AI Advanced · Classical ML & Data Science

Classical ML & Data Science

Long before chatbots, machine learning was quietly running the world — spam filters, credit scoring, fraud detection, demand forecasting, recommendation engines. That whole field, the one that learns patterns from tables of numbers rather than paragraphs of text, is “classical” ML, and it still powers most of the prediction happening in production today. This page is your orientation: what it is, how it thinks, and when a boring little model on a spreadsheet beats the smartest LLM you can buy.

What “classical” ML is — and why it still runs the show

☺ Like you’re 10: The flashy new robot that writes poems isn’t the only robot in town. The quiet older robots that sort your mail, catch cheaters, and guess tomorrow’s weather have been on the job for years — and they’re still the ones doing most of the real work.

When people say “AI” today they usually mean large language models. But machine learning — teaching a computer to find patterns in data instead of hand-coding rules — is a much older and broader field. “Classical ML” is the umbrella term for everything that isn’t a giant deep neural network trained on text or images: the algorithms that learn from structured, tabular data — rows and columns, like a spreadsheet or a database table.

It’s tempting to think LLMs made this obsolete. They didn’t. The overwhelming majority of ML actually deployed in businesses is still classical, and for good reasons you’ll see throughout this page — it’s fast, cheap, explainable, and phenomenally good at the tabular-prediction problems that make up most of real-world “AI.” Some of the biggest jobs it quietly does:

• Fraud & risk. Is this credit-card charge fraudulent? Should this loan be approved? Models score millions of transactions a second — far too fast and too high-stakes to hand to a chatbot.
• Forecasting. How many units will this store sell next week? What will demand, churn, or ad spend look like next quarter? Numbers in, numbers out.
• Recommendation & ranking. Which product, song, or search result goes at the top? Ranking systems are classical ML at massive scale.
• Classification at volume. Spam vs not-spam, defect vs no-defect on a factory line, likely-to-churn vs not — cheap decisions made billions of times.

These are adjacent to the LLM world you already know from the rest of this course, and they connect to it more than you’d expect — modern AI systems routinely mix classical models with LLMs (more on that at the end). Think of this page as a solid map, not a from-scratch course: enough to know how the field thinks, when to reach for it, and where to go deeper.

Supervised vs unsupervised: the two big families

☺ Like you’re 10: Two ways to learn. In the first, a teacher shows you flashcards with the answers on the back until you can guess new ones — that’s supervised. In the second, you get a pile of toys with no labels and you sort them into groups yourself by which ones seem alike — that’s unsupervised.

Almost all classical ML splits into two families, and the difference is simply: did your data come with the answers, or not?

Supervised learning is learning from labeled examples. You have historical data where you already know the outcome — houses with their sold prices, emails marked spam or not — and you train a model to predict that outcome for new, unseen cases. It splits again by what you’re predicting:

• Regression predicts a number: a house price, tomorrow’s temperature, expected revenue. The output is continuous.
• Classification predicts a category: spam or not, fraud or legit, which of five disease types. The output is a label (often with a probability attached).

Unsupervised learning works on data with no labels — no “right answer” to predict. Instead of learning a target, it finds structure hiding in the data itself:

• Clustering groups similar rows together — e.g. segmenting customers into natural buying-behavior groups nobody defined in advance.
• Dimensionality reduction squeezes many columns down to a few that capture most of the information, for visualization or to feed other models. (If you’ve met embeddings, you’ve already seen this idea: turning something complex into a compact vector.)

There’s a third family worth naming so the map is complete: reinforcement learning, where an agent learns by trial and error from rewards rather than from a fixed dataset — that gets its own page. And the deep-learning world (deep learning, computer vision & NLP) is really supervised/unsupervised learning too, just with much bigger, layered models on images and text.

The data-science workflow: the loop every project runs

☺ Like you’re 10: Baking a cake has steps in order — gather ingredients, measure them out, bake, taste-test, then serve. Building a model has the same kind of recipe, and skipping the taste-test (checking it actually works) is how you serve something nobody can eat.

A machine-learning project isn’t “pick an algorithm.” The algorithm is a small part; most of the work is around it. Nearly every project follows the same loop, and the first stages eat most of the time:

1. Data. Collect it, then clean it — fix missing values, remove duplicates, fix bad rows. This unglamorous step is famously ~70–80% of the real work. Bad data quietly ruins everything downstream.
2. Features. Turn raw data into the numeric inputs a model can learn from (its own section below). This is where domain knowledge lives.
3. Train. Pick an algorithm and let it fit the patterns in your training data — adjusting its internal parameters to make good predictions on the examples it sees.
4. Evaluate. Measure how well it does on data it never trained on. This is the honesty check, and it’s where most beginners go wrong (its own section, too).
5. Deploy & monitor. Ship the model so real requests hit it, and watch it over time — because the world drifts and yesterday’s model slowly goes stale.

The loop is a loop for a reason: evaluation usually sends you back to try better features or a different model. And the “deploy & monitor” tail is a whole discipline of its own — the same operational muscle you’ll need for LLM apps lives in MLOps and the broader Production & Ops lesson, while getting clean data in the first place is data engineering.

A tour of the workhorse algorithms

☺ Like you’re 10: These are different “thinking styles” for spotting patterns. One draws a straight line through dots. One plays twenty-questions. One asks a whole crowd of question-askers to vote. You don’t need the math — just the vibe of each, and roughly when it’s the right pick.

You don’t need to derive the math to be dangerous here — you need intuition for what each algorithm does and when it fits. Here are the ones you’ll meet again and again:

• Linear regression. Draws the best straight line (or flat plane) through your data to predict a number. “Price goes up about $200 per square foot” is a linear model. Simple, fast, and beautifully interpretable — you can read the weight on each feature.
• Logistic regression. The classification cousin: instead of a number, it outputs a probability between 0 and 1 (“73% likely to be fraud”). Despite the name, it’s a classifier, and it’s the sensible baseline for yes/no problems everywhere.
• Decision trees. A flowchart of yes/no questions — “Is income > $50k? Then is age < 30?” — ending in a prediction. Wonderfully easy to read, but a single deep tree tends to memorize its training data (overfit).
• Random forests. Train hundreds of different trees on random slices of the data and let them vote. The crowd cancels out any one tree’s quirks, giving a strong, robust model with little fuss. A superb default for tabular problems.
• Gradient boosting. Also many trees, but built one after another, each new tree fixing the mistakes of the ones before. Libraries like XGBoost, LightGBM, and CatBoost implement this — and gradient-boosted trees are, still today, the reigning champions on most tabular prediction tasks, routinely beating deep learning there.
• k-means (clustering). The go-to unsupervised method: pick k groups and shuffle points until each sits with its nearest cluster center. Great for “find natural segments in my customers.”
• Support vector machines (SVM). Finds the boundary that separates two classes with the widest possible margin. Once dominant for classification; still handy on smaller, cleaner datasets.

Features & feature engineering: where projects are won

☺ Like you’re 10: A model can only look at numbers. So if you want it to understand “this happened on a weekend” or “this house is near a school,” you have to turn those ideas into numbers first. Picking and preparing those clues is often what makes one model way smarter than another.

A feature is just one input column the model learns from — square footage, day of week, number of past purchases. Feature engineering is the craft of turning raw data into features that actually help the model, and it’s frequently the difference between a mediocre model and a great one. The algorithm gets the glory; the features do the work. Common moves:

• Encoding categories. A model can’t read the word “Tuesday” or “California” — you convert categories into numbers (e.g. one-hot encoding: a separate 0/1 column per category).
• Scaling numbers. Some algorithms choke when one feature ranges 0–1 and another ranges 0–1,000,000. Rescaling everything to a comparable range keeps them balanced.
• Deriving new signal. From a raw timestamp you can extract day-of-week, is-holiday, hour-of-day. From two columns you can compute a ratio. This is where domain knowledge turns into predictive power.
• Handling missing values. Real data has holes. You decide: fill them with a sensible default, flag them, or drop them — quietly, before the model ever sees them.

This is the closest classical-ML analog to prompting in the LLM world: in both, most of your leverage is in how you present the input, not in the underlying model. And it echoes a theme from across this course — good inputs beat clever models. There’s even a bridge between the two worlds: text and images can be turned into embeddings (dense numeric vectors from a neural network) and then fed as features into a classical model. That’s a common, powerful pattern — let a deep model do the “understanding” and a boosted tree do the fast, cheap prediction.

Evaluation & the overfitting trap

☺ Like you’re 10: If a student memorizes the exact answers to the practice test, they’ll ace the practice test and flunk the real exam. To know if they actually learned, you have to test them on questions they’ve never seen. Models cheat the same way — so you always hide some data and test on that.

Here’s the single most important idea in all of applied ML, and the one beginners most often get wrong. A model’s score on the data it trained on is meaningless — it can just memorize. What matters is how it does on data it has never seen. That’s the whole reason for a train/test split: before training, you hide a slice of your data (say 20%), train on the rest, and then measure the model only on the hidden slice.

Overfitting is when a model learns the training data too well — memorizing its noise and quirks instead of the real pattern — so it looks brilliant on training data and falls apart on new data. A model that scores 99% on training but 70% on the test set isn’t smart; it’s memorizing. The cure is measuring honestly (train/test split, or the sturdier cross-validation where you rotate which slice is held out) and keeping the model simple enough to generalize.

Once you’re measuring on held-out data, you need the right ruler — and “accuracy” alone is a famous trap:

Why accuracy lies: imagine fraud is 1 in 1,000 transactions. A lazy model that predicts “never fraud” is 99.9% accurate — and completely worthless, because it catches zero fraud. That’s why fraud, disease, and other rare-event problems live and die by precision and recall instead. There’s usually a tradeoff: catch more real fraud (higher recall) and you’ll also flag more innocent charges (lower precision). Which you tune toward depends on the cost of each mistake — a missed cancer diagnosis and a false fraud alert are not equally bad.

When classical ML beats an LLM — and how they team up

☺ Like you’re 10: Use the right tool for the job. You wouldn’t hire a world-famous novelist to add up your grocery bill — a $2 calculator is faster, cheaper, and you can check its work. For number-and-table jobs, the little model is the calculator.

The LLMs you’ve studied are extraordinary at language, reasoning, and open-ended tasks. But for a huge class of problems, a classical model is simply the better engineering choice. Reach for classical ML when you have:

• Tabular prediction. Structured rows-and-columns data with a clear target. This is classical ML’s home turf, and it usually wins on accuracy outright.
• Low latency & high volume. Scoring millions of events per second, or needing an answer in single-digit milliseconds. A boosted tree does this trivially; an LLM call cannot.
• Interpretability. When you must explain a decision — why a loan was denied, which factors drove a risk score — regulated industries often require it, and classical models expose it naturally.
• Cost. Predictions can cost a tiny fraction of a cent instead of an LLM API call. At scale, that difference is the whole business case.

And it’s not either/or — the most interesting modern systems combine them. A few patterns you’ll see:

• LLM as feature extractor. Use an embedding model to turn messy text or images into vectors, then feed those vectors as features into a fast classical classifier.
• Classical model as a tool for an agent. An agent can call a trained forecasting or fraud model the way it calls any other tool (often via MCP), getting a precise number the LLM would only have guessed at.
• Classical routing/guardrails around LLMs. Cheap classifiers decide which requests even need an LLM, or screen inputs and outputs — a staple of AI security and pipelines.

Knowing both toolkits — and when to switch between them — is what separates someone who “uses AI” from someone who builds with it well. If you want to go deeper on the internals, the sibling pages branch out from here: the neural-network foundations in deep learning and training models, the perception fields in computer vision & NLP, learning-from-rewards in reinforcement learning, and the operational side in MLOps and data engineering. The further reading page points to full courses if you want to actually build these models yourself.

Check your answers

1. Supervised vs unsupervised, and the two sub-types: Supervised learning trains on labeled examples where you already know the outcome (houses with their sold prices, emails marked spam), while unsupervised learning works on unlabeled data and finds hidden structure instead of predicting a target. Supervised splits further by what you predict: regression outputs a continuous number (a house price, tomorrow’s temperature), and classification outputs a category or label (spam or not, fraud or legit), often with a probability attached.
2. Why training-data scores lie, and what a split fixes: A model’s score on the data it trained on is meaningless because it can simply memorize that data — a model can score 99% on training yet only 70% on new data (overfitting). A train/test split fixes this by hiding a slice (say 20%) before training and measuring the model only on that never-seen data, giving an honest read on how it will generalize.
3. Why 99.9% accuracy can be worthless, and what to look at: If fraud is only 1 in 1,000 transactions, a lazy model that always predicts “never fraud” is 99.9% accurate yet catches zero fraud, so accuracy hides total failure on the rare event that matters. For imbalanced rare-event problems you should look at precision and recall instead, tuning the tradeoff between them based on the real cost of a missed case versus a false alarm.
4. Two reasons for a gradient-boosted tree over an LLM here: House-price prediction is a tabular problem, which is classical ML’s home turf where a boosted tree trained on the rows is more accurate. It also costs a tiny fraction of a cent per prediction and runs in milliseconds (versus a slow, pricier LLM call), and it exposes which features drove each estimate, giving interpretability the LLM can’t — any two of these justify the choice.