Skip to content
Contact →

Identifying Data Quality Issues

Updated
25 Sept 2026
4 min read
7 sections
Page 13 of 29
Data Migration Masterclass

Section 3: Data Assessment and Preparation. Lesson 13.

Identifying data quality issues is the process of detecting errors, inconsistencies, inaccuracies and anomalies in a dataset before it is migrated to a new system.

One of the biggest mistakes organisations make is moving poor-quality data straight from source to target. If issues are not found and fixed first, the new system inherits the same problems, and much of the value of the migration is lost.

Where quality issues come from

Data degrades over time as it is:

  • Entered manually by many different users, each with their own habits
  • Imported from multiple systems with different formats and rules
  • Modified by changing business processes
  • Stored in legacy applications with weak validation

Machine-generated data (sensor readings, logs, transaction feeds) usually has fewer typing errors and inconsistencies than human-entered data, but it is not immune. It can still carry gaps from outages, clock drift, unit changes after firmware updates, and duplicate events from retries.

Why it matters

Data quality affects reporting, analytics, regulatory compliance and decision-making:

  • Duplicate customer records lead to inaccurate reports and inflated customer counts
  • Missing contact information breaks customer communication
  • Incorrect financial data can cause compliance violations and penalties

A thorough quality assessment raises the migration's success rate, builds trust in the migrated data, reduces operational risk, and means the target system works with accurate information from day one.

Common data quality issues

These map onto the standard data quality dimensions:

  • Completeness: missing values, empty cells, blank rows, deleted records (for example lost global sales figures).
  • Validity: values of the wrong type or outside allowed rules, such as numbers in a text field or a year of 2099.
  • Consistency: the same thing recorded differently, such as PS4, PlayStation 4 and ps-4, or dates in mixed formats.
  • Uniqueness: duplicate records or duplicate keys.
  • Accuracy: values that are well-formed but wrong, such as the wrong publisher on a game.
  • Integrity: broken relationships, such as sales rows pointing at a game_id that no longer exists.
  • Timeliness: outdated values, such as old addresses or discontinued products still marked active.
  • Structural issues: blank or unnamed header columns, merged cells, stray totals rows inside the data.

Hands-on: breaking a dataset on purpose

The lecture takes the video game sales dataset from lesson 12 and deliberately damages it to see what quality issues look like:

  • Clearing values from some cells (completeness)
  • Typing numbers into text columns (validity and consistency)
  • Deleting rows that hold important sales figures (completeness and integrity)

Small inconsistencies like these look harmless in a few rows, but across a full enterprise dataset they break reports, joins and downstream systems. Incorrect formatting is one of the most common causes of large-scale errors.

Assessing quality with AI

Manually assessing a large dataset used to be impractical. AI assistants such as ChatGPT or Claude can now do a first-pass assessment. In the lecture, the damaged file is uploaded with the prompt "Perform a comprehensive data quality assessment." The assistant then:

  • Examines the schema
  • Checks completeness
  • Finds duplicates
  • Validates values and consistency
  • Spots outliers and usability issues

It produced a downloadable Excel assessment report with:

  • Summary metrics: blank game_id rows, duplicate counts, blank header columns
  • Issues grouped by severity: high, medium and low
  • A column profile showing where issues sit
  • A dimensions view: each issue, its impact and the recommended fix
  • Issue samples: the specific invalid values
  • Category distribution: where problems cluster
  • Recommendations with reasons, for example drop or rename four blank header columns that are 100% empty and account for hundreds of thousands of blank cells

A better prompt

text
Perform a data quality assessment of the attached file for a migration.
For each column report: type, % null, distinct count, invalid or out-of-range values.
Find duplicate rows and duplicate keys (game_id).
Flag inconsistent spellings of the same value.
Group issues by severity (high / medium / low) with impact and a recommended fix.
Return the result as an Excel report with a summary tab.

Use AI carefully

  • Never upload confidential or restricted data (PII, financial, health) to a public AI tool. Use an approved enterprise tool, or anonymise the data first. See Data Sensitivity and Compliance.
  • Verify the findings: recompute key counts with SQL or pandas before acting on them, since AI summaries can miss or invent issues.
  • For repeatable checks in a real migration, turn the findings into automated rules with tools such as Great Expectations, dbt tests, Soda, or Informatica and Talend Data Quality.

The same checks in pandas

python
import pandas as pd

df = pd.read_csv("video_games.csv")

# Completeness
print(df.isna().mean().sort_values(ascending=False))   # % missing per column
print(df.isna().all(axis=1).sum(), "fully blank rows")
print([c for c in df.columns if c.startswith("Unnamed")], "blank headers")

# Uniqueness
print(df.duplicated().sum(), "duplicate rows")
print(df["game_id"].duplicated().sum(), "duplicate game_id")

# Validity: numbers where text is expected
numeric_titles = df["title"].astype(str).str.fullmatch(r"\d+(\.\d+)?")
print(numeric_titles.sum(), "numeric values in title")

# Validity: out-of-range years
print(df[(df["year"] < 1970) | (df["year"] > 2026)])

Try it: Game Data Quality Scan

An 11-row sample of the video game file with the lecture's planted problems: a blank title, a number in the title column, a missing game_id, a year of 2099, a negative sales figure, three spellings of PlayStation 4, and an exact duplicate row. Click a quality dimension (completeness, validity, consistency, uniqueness) to highlight only its issues in the table and see each finding's severity and recommended fix.

Game Data Quality ScanOpen full size ↗

Key takeaways

  • Find errors, inconsistencies, inaccuracies and anomalies before migration, or the new system inherits them.
  • Quality degrades through many users, many sources, changing processes and legacy storage.
  • Check completeness, validity, consistency, uniqueness, accuracy, integrity and timeliness.
  • AI tools can produce a fast, structured assessment report, but protect sensitive data and verify the results.

Next: 14. Data Cleansing Strategies.