Data Volume and Complexity Analysis

Section 2: Business Requirements and Data Analysis. Lesson 9.

Once the migration scope is defined, the next step is to analyse the data inside it. Data volume and complexity analysis evaluates the size, structure, relationships, quality and technical characteristics of the data before any migration solution is designed. The goal is to know how much data exists, how complex it is, and what challenges it will create during execution.

Data volume

Data volume is the total amount of data to be migrated, measured in gigabytes (GB), terabytes (TB) or, for the largest estates, petabytes (PB). It is not only bytes: record counts, file counts and the rate at which data grows or changes all count.

Volume directly affects:

• Migration duration: how long extraction, transfer and loading take
• Storage requirements: staging space, target capacity and backups
• Network bandwidth: whether the link can carry the data within the window
• System performance: load on source and target systems during the move

Quick transfer-time estimate

Time is roughly data size divided by effective throughput. For example, 10 TB over a 1 Gbps link at about 70% efficiency (around 87 MB/s) takes about 32 hours, before any transformation or validation. At 100 TB it is nearly two weeks, which is why offline appliances such as AWS Snowball exist.

What to measure

• Size per database, schema and table, and total file storage
• Row counts for the largest tables
• Number and size of files, attachments and binary objects (BLOBs)
• Daily change rate, which determines how much delta must be synced before cutover
• How much is active data and how much can be archived

Data complexity

Complexity is about how hard the data is to move correctly, not how much of it there is. Its main dimensions:

1. Number of tables and objects: tables, views, stored procedures, triggers and files.
2. Relationships between datasets: foreign keys, many-to-many links, hierarchies, and cross-system references that must stay intact.
3. Data quality: duplicates, nulls, inconsistent formats, orphaned records.
4. Business rules: logic such as "an order cannot close with an unpaid invoice", often hidden in application code.
5. Data rules: constraints, validations, formats and allowed values that the target enforces.

Other factors that add complexity: many source systems, heavy transformation or mapping, varied data types (JSON, XML, BLOBs, geospatial), encodings, and data that changes constantly during the migration.

Examples

• A software house: projects, clients, employees, timesheets, invoices and repositories, all linked, so the data architecture is already fairly complex.
• A CRM platform: many more tables (accounts, contacts, leads, opportunities, activities, campaigns, custom objects), deep relationships, strict quality demands, and many business rules and automations.

Important: a small dataset can be more complex than a huge one. A 5 GB ERP with hundreds of interdependent tables and heavy transformation is harder to migrate than a 5 TB archive of flat log files.

Complexity rating

A simple way to classify each dataset or table is to score volume and complexity separately:

• Low volume, low complexity: straightforward. Automate and move early as a pilot.
• High volume, low complexity: mostly a performance problem. Use bulk loading, parallelism and bandwidth planning.
• Low volume, high complexity: mostly a mapping problem. Spend time on transformation rules and testing.
• High volume, high complexity: highest risk. Needs phasing, rehearsals and the most validation.

Profiling tools such as SQL queries against system catalogs, Great Expectations, or vendor assessment tools like AWS Schema Conversion Tool reports speed up this analysis.

Why the analysis matters

Analysing volume and complexity together helps the team:

• Estimate total project cost and timeline realistically
• Select the right migration strategy and tools
• Determine hardware, storage and network requirements
• Identify performance bottlenecks before they appear in production
• Choose between a big bang and a phased migration

Big bang vs phased migration

Big bang migration moves everything at once in a single, highly coordinated operation. The old system is shut down and users switch to the new system immediately.

• Pros: shorter overall project, no need to run two systems in parallel, simpler to keep data in sync.
• Cons: requires a downtime window, all risk lands on one event, and rollback is harder.
• Suits: smaller volumes, low complexity, businesses that can tolerate planned downtime.

Phased (trickle) migration moves data wave by wave over a period of time, for example by module, region or business unit, often with change data capture keeping old and new systems in sync.

• Pros: lower risk per wave, little or no downtime, lessons from early waves improve later ones.
• Cons: longer overall, two systems run in parallel, and sync logic adds complexity.
• Suits: large volumes, high complexity, systems that must stay available (24/7 operations).

Key takeaways

• Analyse volume and complexity before designing the solution.
• Volume (GB, TB, PB) drives duration, storage, bandwidth and performance.
• Complexity comes from tables, relationships, quality, business rules and data rules, and small datasets can be the hardest.
• The analysis feeds cost estimates, resource planning, risk management, and the big bang vs phased decision.

Next: 10. Defining Migration Objectives.