Ugenz Academy/Tutorial

Datacleaning

6 min read1,033 words

Reading progress

Keep learning

Data Cleaning

Introduction

Data Cleaning is the process of identifying, correcting, removing, or handling incorrect, incomplete, duplicate, and inconsistent data.

It is an important step in the Data Science process because poor-quality data can produce incorrect analysis and unreliable machine learning results.

Data Cleaning converts raw and messy data into clean, consistent, and useful data.


Why is Data Cleaning Important?

Raw data may contain:

  • Missing values
  • Duplicate records
  • Incorrect values
  • Spelling mistakes
  • Inconsistent formats
  • Outliers
  • Unnecessary data
  • Incorrect data types

Clean data helps to:

  • Improve data quality.
  • Produce more accurate analysis.
  • Improve machine learning models.
  • Reduce errors.
  • Make visualization more reliable.
  • Support better decision-making.

Common Data Cleaning Techniques

1. Handling Missing Values

A missing value occurs when information is not available for a particular record.

Example

Name Age City Anuradha 22 Janakpur Ram - Kathmandu Sita 21 -

The missing values can be handled by:

  • Removing the row.
  • Replacing it with the mean.
  • Replacing it with the median.
  • Replacing it with the mode.
  • Using another suitable value.
  • Applying a suitable prediction or imputation method.

Python Example

import pandas as pd data = pd.read_csv("data.csv") data["Age"] = data["Age"].fillna(data["Age"].mean())

2. Removing Duplicate Data

Duplicate data occurs when the same record appears more than once.

Example

ID Name 101 Anuradha 102 Ram 101 Anuradha

The third record is a duplicate.

Python Example

data = data.drop_duplicates()

3. Correcting Inconsistent Data

Sometimes the same value is written in different formats.

Example

Janakpur janakpur JANAKPUR Janakpurdham

These values may need to be standardized depending on the purpose of the analysis.

Python Example

data["City"] = data["City"].str.lower()

4. Handling Incorrect Values

Data may contain values that are impossible or incorrect.

Example

Name Age Anuradha 22 Ram -5 Sita 21

An age of -5 is not valid.

The value should be investigated and corrected, replaced, or treated as missing.


5. Correcting Data Types

Data should have appropriate data types.

Example

A price may be stored as text:

"50000" "60000" "70000"

It should be converted to a numeric type before mathematical calculations.

Python Example

data["Price"] = pd.to_numeric(data["Price"], errors="coerce")

6. Handling Outliers

An outlier is a value that is unusually different from most other values.

Example

Salary: 25000 28000 30000 32000 5000000

5000000 is much larger than the other values and may be an outlier.

Outliers should not automatically be deleted. They should first be investigated to determine whether they are errors or legitimate observations.


7. Removing Unnecessary Columns

Some columns may not be useful for analysis.

Example

ID Name Age City Password

A Password column should not be included in an analysis dataset.

Unnecessary or sensitive information should be removed or excluded appropriately.


8. Standardizing Formats

Data should use consistent formats.

Date Example

Before:

01/02/2026 2026-02-01 Feb 1, 2026

After standardization:

2026-02-01 2026-02-01 2026-02-01

Python Example

data["Date"] = pd.to_datetime(data["Date"])

Data Cleaning Process

A typical data cleaning process is:

Raw Data ↓ Inspect Data ↓ Identify Problems ↓ Handle Missing Values ↓ Remove Duplicates ↓ Correct Errors ↓ Standardize Formats ↓ Handle Outliers ↓ Validate Data ↓ Clean Data

Data Cleaning Using Pandas

Pandas is a popular Python library for data manipulation and analysis.

Load Data

import pandas as pd data = pd.read_csv("sales.csv")

View Data

print(data.head())

Check Missing Values

print(data.isnull().sum())

Remove Duplicates

data = data.drop_duplicates()

Fill Missing Values

data["Price"] = data["Price"].fillna(data["Price"].mean())

Save Clean Data

data.to_csv("cleaned_data.csv", index=False)

Example

Suppose we have the following raw data:

Name Age City Salary Anuradha 22 Janakpur 30000 Ram - Kathmandu 35000 Sita 21 janakpur 32000 Anuradha 22 Janakpur 30000

Problems

  1. Age is missing for Ram.
  2. janakpur uses a different format.
  3. Anuradha appears twice.

After Cleaning

Name Age City Salary Anuradha 22 Janakpur 30000 Ram 22 Kathmandu 35000 Sita 21 Janakpur 32000

Now the dataset is more consistent and suitable for further analysis.


Data Cleaning vs Data Transformation

Data CleaningData Transformation
Fixes errors and quality problemsChanges data into a suitable format
Handles missing valuesConverts or derives variables
Removes duplicatesAggregates data
Corrects inconsistent valuesNormalizes or scales data
Improves data qualityMakes data suitable for analysis or modeling

Both activities are often part of data preprocessing.


Advantages of Data Cleaning

  • Improves data quality.
  • Reduces errors.
  • Improves analysis accuracy.
  • Helps produce reliable visualizations.
  • Improves machine learning performance.
  • Makes data easier to understand.
  • Supports better decision-making.

Challenges of Data Cleaning

  • Large amounts of data can take time to clean.
  • Missing values may be difficult to handle.
  • Some errors require domain knowledge.
  • Different data sources may use different formats.
  • Outliers may be legitimate values.
  • Cleaning rules can sometimes introduce unintended changes.

Best Practices

When cleaning data:

  1. Always keep a copy of the original data.
  2. Understand the dataset before modifying it.
  3. Check for missing values.
  4. Remove or investigate duplicates.
  5. Standardize formats.
  6. Validate corrected values.
  7. Document important cleaning steps.
  8. Protect sensitive information.
  9. Avoid deleting data without understanding why it is incorrect.
  10. Recheck the dataset after cleaning.

Conclusion

Data Cleaning is an essential step in Data Science. It involves identifying and handling missing values, duplicates, errors, inconsistent formats, incorrect data types, and unusual values.

Clean and reliable data provides a strong foundation for data analysis, visualization, and machine learning.

Garbage in, garbage out: if poor-quality data is used as input, the resulting analysis or model may also be unreliable.

Keep learning. Keep building. Keep growing.

© 2026 Ugenz Academy