Introduction

Notes from the course introduction, with extra context, examples and industry practice added.

What is data migration?

Data migration is the process of transferring data from one storage system, format or computing environment to another. It is typically a finite, high-stakes project: it has a clear start, a cutover and an end, unlike ongoing data integration (such as nightly ETL feeds), which runs indefinitely.

It is not just copy-and-paste. Copying a file between folders or drives keeps the same format and system. A migration usually changes at least one of storage, format, schema or platform, so the data has to be understood, reshaped and verified.

Common drivers

Cloud adoption: moving on-premises workloads to AWS, Azure or GCP
Server or hardware upgrades: retiring end-of-life infrastructure
Database modernization: for example Oracle to PostgreSQL, or a mainframe to a relational database
Mergers and acquisitions: consolidating two companies' systems into one
Application replacement: for example a legacy CRM to Salesforce, or an old ERP to SAP S/4HANA
Cost, licensing or compliance: data residency rules, or leaving expensive vendor licences

Types of migration

Storage migration: moving data between storage media or arrays, such as a SAN to object storage like S3. The format usually stays the same.
Database migration: moving between database engines or versions. Often needs schema conversion, data type mapping and rewriting stored procedures.
Application migration: moving an application's data to a new application or vendor. Needs mapping between two different data models.
Cloud migration: moving data or workloads from on-premises to the cloud, or between clouds. Often combines the other three.
Business process migration: triggered by reorganizations or mergers, where the data follows changed processes.

The migration lifecycle

In enterprise IT a migration follows a defined pipeline, broadly the ETL pattern plus validation:

Extract: read data out of the source system (full export, change data capture or API reads).
Prepare: profile the data, find duplicates, nulls and invalid values, and cleanse it.
Transform: map source fields to the target schema, convert types, formats and encodings, and apply business rules.
Load: write the data into the target system, often in batches, respecting keys and constraints.
Validate: confirm the data is accurate, complete and usable, using row counts, checksums, field-level comparisons and business reconciliation.

Around this core pipeline sit planning (scope, inventory, strategy) before the move, and cutover, decommissioning and support after it.

Enterprise data migration

Enterprise data migration is a structured process for moving large volumes of data, often terabytes to petabytes across many systems, from legacy systems into modern cloud platforms, centralized data warehouses or consolidated storage.

Legacy systems are older platforms still running critical workloads, often dating from the 1990s: mainframes (COBOL, DB2, VSAM files), older Oracle or SQL Server versions, AS/400 systems and in-house applications. They tend to carry undocumented business logic, inconsistent data and few people who still understand them.

The two overriding goals are:

Data preservation: nothing is lost, altered unintentionally or corrupted.
Business continuity: operations continue with minimal or no downtime.

Migration strategies

Big bang: move everything in one cutover window, such as a weekend. Simpler, but high risk and needs downtime.
Trickle (phased): migrate in stages while the old and new systems run in parallel, often kept in sync with change data capture (CDC). Less downtime, but more complex.

What success looks like

A successful migration lets the business operate smoothly after the move. The migrated data must be:

Consistent: it matches the source, with referential integrity intact
Secure: encrypted in transit and at rest, with access controls carried over
Accessible: available to authorized users and applications when needed

When customer names, invoices, product records or employee data are moved incorrectly, the consequences include:

Reporting errors: wrong dashboards and bad decisions
Financial mistakes: wrong billing, or ledgers that do not balance
Compliance problems: breaches of GDPR, HIPAA or SOX, and failed audits
Operational delays: orders, payroll or support blocked

Industry surveys have repeatedly found that a large share of migration projects overrun their budget or schedule. The main causes are poor data quality, underestimated complexity and weak testing.

Safeguards every major migration needs

Rigorous planning: inventory the data, set scope, identify stakeholders and data owners
Extensive testing: trial migrations and dress rehearsals on production-like data
Strict validation: automated reconciliation between source and target
Rollback strategy: a tested plan, with backups, to return to the source system if cutover fails

Core building blocks

Data types: structured, semi-structured and unstructured data, and how types differ between systems (covered in the next lesson)
Data transformation: cleaning, mapping and reshaping data to fit the target
Source system: where the original data lives today
Target system: where the data needs to end up
Migration pipeline: the secure, repeatable flow from source to target

text

┌───────────────┐   extract → transform → load   ┌───────────────┐
│ Source system │ ─────────────────────────────▶ │ Target system │
└───────────────┘          validate ◀─────────── └───────────────┘

On a diagram the pipeline is a single arrow. In practice it is slow and careful work, especially at enterprise scale, where volume, downtime windows and data quality all constrain it.

Why it matters: digital transformation

Migration is a prerequisite for modernization. Cloud platforms, analytics and AI systems, and modern enterprise applications are only as good as the data in them, following the rule of garbage in, garbage out. A poorly executed migration damages trust in an expensive new system from day one, and users drift back to spreadsheets and old workarounds.

Key takeaways

Migration moves data across systems, formats or environments. It is a finite project, not a copy operation.
Core flow: extract, prepare, transform, load, validate.
Success means the data is consistent, secure and accessible, and the business keeps running.
Planning, testing, validation and rollback are required, not optional.

Next: data types, and how they affect source and target systems.