Migration Foundations · Data Migration & Data Gravity

Data Migration & Data Gravity

Apps are easy to copy; data is heavy. In this lesson you'll learn how to move databases and files to the cloud safely — online or offline, all at once or continuously — and why the biggest pile of data is usually the hardest, most expensive thing to shift. Meet data gravity and the surprising truth that sometimes a truck really is faster than the internet.

What is data migration?

☺ Like you’re 10: It's carefully moving all your boxes to the new house and then counting them at both doors to make sure every single one arrived.

Data migration means moving your information — databases, files, backups, and media — from an old home (like the on-premises data center we call Fort Rusty) to a new one in the cloud (Cloudville). It sounds like a simple copy-paste, but it's a discipline in its own right, separate from moving the apps that sit on top of it.

Why data — not the apps — is the hard part

An application is mostly a set of instructions. If you lose an app server, you rebuild it from code and redeploy — annoying, but recoverable. Data is different: it is stateful and often irreplaceable. Lose a customer's ten years of order history and there is no re-compiling it back into existence. That asymmetry is the whole reason data gets its own careful process: while you copy, data can be lost, corrupted, half-copied, or changed underneath you — and any one of those can go unnoticed until it's too late to undo.

The three questions every data move answers

Behind all the tooling, every data move comes down to three questions. Answer these well and you've mastered the core of the topic:

• Is the source still being used while we copy? — online vs offline.
• Do we copy once, or keep copying the changes? — one-time bulk vs continuous replication.
• How do we prove it arrived intact? — validation & reconciliation.

☺ Like you’re 10: Are people still using the old house while we pack? Do we make one big trip or keep a conveyor belt running? And how do we check every box made it? Those three answers shape the whole move.

How you copy — online vs offline, bulk vs continuous

☺ Like you’re 10: Two dials to set before moving day: do we keep living in the house while we pack, and do we make one big trip or keep sending boxes as they fill up?

Online vs offline migration

Online migration copies data while the source system stays live and in use. Nothing shuts down, so customers keep working — but the data keeps changing underneath you, so you need a way to capture those changes as they happen. Offline migration takes the source out of service (or freezes writes) first, so the data holds still while you copy it. Offline is simpler and safer to verify, but it means downtime.

• Online — no downtime, more complex, needs change tracking. Best for busy production systems that can't stop.
• Offline — some downtime, much simpler, easy to verify. Best for small systems or scheduled maintenance windows.

The choice usually isn't about which is "better" — it's about how much downtime the business can tolerate. A weekend-only internal tool is happy to go offline; a global checkout that takes money every second cannot, so it must move online.

One-time bulk load vs continuous replication

A one-time bulk load copies everything once — great for data that isn't changing, like archives or a system you're about to retire. Continuous replication keeps the destination in sync with the source by streaming every insert, update, and delete as it happens. Continuous replication is what makes near-zero-downtime cutovers possible: you bulk-load once, then let replication keep the copy fresh until you're ready to switch.

Change data capture (CDC) — how replication stays fresh

Continuous replication is often powered by change data capture (CDC) — a technique that reads the database's own transaction log (the redo log, WAL, or binlog the engine already writes) to catch each change the instant it occurs. Because CDC tails a log the database maintains anyway, it's far lighter than the naïve alternative of repeatedly querying every table to see "what's new," and it never misses a change that happened between polls. That efficiency is why CDC underpins almost every low-downtime cutover.

ETL — reshaping data on the way

Many migrations also involve ETL — Extract, Transform, Load — where data is pulled from the source (extract), reshaped or cleaned along the way (transform), and written to the destination (load). ETL matters when the new home expects the data in a different shape than the old one — merging two customer tables into one, fixing dates, or dropping columns nobody uses anymore. A common cloud variant flips the last two steps into ELT (load the raw data first, then transform it using the destination's own power) — same three jobs, different order.

Database migration

☺ Like you’re 10: Moving between two houses with the same-shaped rooms is easy — furniture just fits. Moving to a house with different rooms means you have to rebuild the shelves before the boxes fit.

Homogeneous vs heterogeneous

When the source and target databases use the same engine — say Oracle to Oracle, or PostgreSQL to PostgreSQL — that's a homogeneous migration. The internal structure matches, so it's mostly a straight copy. When they use different engines — Oracle to PostgreSQL, or SQL Server to MySQL — that's a heterogeneous migration, and it's much harder because the database's blueprint has to be translated.

Heterogeneous moves are common for a very practical reason: teams often migrate off an expensive commercial engine (like Oracle or SQL Server) onto an open-source one (like PostgreSQL or MySQL) to cut licensing costs while they're already in the cloud. The payoff is real, but so is the translation work — which is exactly what the next two ideas handle.

Schema conversion

The blueprint you have to translate is the schema — the definition of tables, columns, data types, keys, and rules. Schema conversion is the work of translating the schema (and things like stored procedures and functions) from the source engine's dialect into the target's. Cloud providers offer schema conversion tools that automate most of this and flag the parts a human must rewrite by hand — because some proprietary features simply have no direct equivalent and need a person to redesign them.

Database Migration Service (DMS)

A Database Migration Service (DMS) is a managed cloud service that handles the moving for you: it connects to the source, performs the initial bulk copy, then keeps the target in sync with continuous replication (CDC) until you cut over. AWS, Azure, and Google Cloud each offer one — AWS Database Migration Service, Azure Database Migration Service, and Google Cloud Database Migration Service. The idea is the same everywhere — let a managed service do the heavy, error-prone streaming so your team doesn't hand-roll it. For a heterogeneous move you typically pair the DMS with a schema conversion tool first, then let the service stream the data. See the toolbox for where these fit among the other migration tools.

Proving it arrived — validation & reconciliation

☺ Like you’re 10: After the truck unloads, you count the boxes at both houses and shake each one to make sure nothing broke inside. Same number, same contents — then you can relax.

Copying data is only half the job; the other half is proving the copy is correct. Validation (also called reconciliation) compares source and destination to confirm they match. Two techniques do most of the work:

Row counts — catch missing or duplicated data

Count the records in each table on both sides. If the source has 4,201,930 rows and the target has 4,201,930, the quantities agree. This catches the loud failures — a table that half-copied, or one that copied twice. But a matching count doesn't prove the contents match: two tables can have the same number of rows and still differ inside.

Checksums & hashes — catch silent corruption

Run a math function over the actual contents to produce a short fingerprint. If the source and target fingerprints match, the content matches, not just the count. This catches silent corruption — a flipped bit or a mangled character — that a row count would sail right past. On enormous tables, teams often checksum in chunks or sample rows so validation itself doesn't take longer than the copy did.

When to validate

Validation isn't a single event at the end. You check after the initial bulk load, again after the final sync at cutover, and you keep the source alive until every check passes. Ellie's rule: never declare a migration done on "it looked fine." Reconcile with counts and checksums, and only then let go of the old copy.

Cutover — the low-downtime switch

☺ Like you’re 10: You keep the conveyor belt running until the last second, then quickly freeze the old house, send the final few boxes, check the count, and flip the sign to point everyone at the new address.

Cutover is the moment you switch from the old system to the new one. It's the same cutover step that sits inside every migration wave, and the trick to minimizing downtime is to combine bulk load, continuous replication, and a short final sync.

The bulk-load + replicate + final-sync recipe

• Bulk load the data once while everything keeps running.
• Replicate changes continuously (CDC) so the target stays close to the source.
• At cutover, stop writes to the source, let the final sync catch the last few changes, run validation, then point traffic at the new system.

Why the downtime is only minutes

Because replication already did the heavy lifting, the source and target are nearly identical before you stop writes. The actual downtime is only the final sync plus the validation check — minutes, not hours. The multi-terabyte copy happened days earlier, quietly, with users none the wiser.

Always keep a rollback path

Keep a rollback path — a way back to the old system — in case validation fails at the last moment. Because you stopped writes rather than deleting the source, the old system is still sitting there, intact, ready to take traffic back. Skipping that safety net is a classic anti-pattern Timmy loves to catch.

Data gravity — why big data is hard to move

☺ Like you’re 10: A giant pile of boxes is like a planet — the bigger it gets, the more it pulls everything toward it, and the harder it is to lift and move somewhere else.

Data behaves like mass

Data gravity is the idea that data behaves like mass: the larger a dataset grows, the more apps and services are drawn to sit close to it (because they need fast access), and the harder — and slower — the whole cluster becomes to move. A small database is a shoebox you carry with one hand. A hundred-terabyte database is a planet with apps orbiting it, and you can't move the planet without moving the orbit.

Why gravity shapes the plan

Data gravity is why teams migrate the data first or design the whole move around it — you don't want an app in Cloudville reaching back across the wire to a database still stuck in Fort Rusty. It's also why cloud-to-cloud moves — leaving a provider you're already on — are so sticky: your data has accumulated gravity where it lives, and everything you've built has quietly gathered around it.

Egress fees — the exit toll

☺ Like you’re 10: Getting into the new house is free, but the old house charges you a fee for every box that leaves. The more boxes, the bigger the exit bill.

Ingress is free; egress is not

Egress means data leaving a provider's network; ingress means data coming in. Most clouds let you upload data for free but charge egress / data-transfer fees to move it out. For a huge dataset, those per-gigabyte fees add up to real money, and they're a big reason data — not apps — is the expensive, sticky part of a migration.

Budget egress before you start

When you plan budgets and total cost (see provisions), egress is a line item you must estimate before you start, especially for cloud-to-cloud moves where you're paying to leave one paid platform for another. Estimate it from the dataset size and the source's published per-GB rate, and add it to the business case up front — an egress bill discovered mid-migration is a nasty surprise.

Moving the bytes — online transfer vs offline appliances

☺ Like you’re 10: If it's a few boxes, mail them over the internet. If it's a whole warehouse, it's actually faster to load a shipping truck and drive it there.

How the bytes physically travel depends on how much there is and how fast your connection is. There are two families of answer.

Online transfer services

Send data over the network. Managed transfer services move files efficiently over the internet or a private link, handling retries, encryption, and verification for you. Examples: AWS DataSync, Azure AzCopy, and Google Storage Transfer Service. Great for small-to-medium datasets and for ongoing syncs where the data keeps trickling in.

Offline appliances

The provider ships you a rugged, encrypted storage device; you copy data onto it locally and mail it back, and they load it into the cloud. Examples: AWS Snowball, Azure Data Box, and Google Transfer Appliance. Built for enormous datasets where the network would take too long — the data is encrypted on the device, so a lost box in transit doesn't mean lost secrets.

When is a truck faster than the internet?

Pip's rule of thumb — "when is a truck faster than the internet?" Do the math: time-over-network ≈ data size ÷ available bandwidth. If pushing your data across the wire would take weeks or months (say, hundreds of terabytes on a modest link), a physical appliance shipped overnight wins easily. This is the modern version of the old joke: never underestimate the bandwidth of a truck full of hard drives.

A common pattern combines both: ship the giant historical bulk on an appliance, then use an online service to sync the changes that piled up while the box was in the mail — so cutover is a quick catch-up, not a fresh multi-week upload.

Check your answers

1. Online copies data while the source stays live and in use (no downtime, but the data keeps changing, so you need change tracking). Offline takes the source out of service first so the data holds still (some downtime, but simpler and easier to verify).
2. A heterogeneous migration moves between different database engines, so the schema's dialect (tables, data types, stored procedures) has to be translated; a homogeneous one uses the same engine and is mostly a straight copy. Schema conversion translates that blueprint into the target engine's format and flags parts a human must rewrite.
3. Row counts compare the number of records on each side — catches missing or duplicated data. Checksums/hashes fingerprint the actual contents — catches silent corruption that a matching row count would miss.
4. Data gravity is the tendency of large datasets to attract apps and services and to resist being moved — the bigger the data, the harder it is to shift. Egress fees reinforce that gravity by charging per gigabyte to move data out of a provider, making big data both expensive and sticky to relocate.