Data Cleaning
Introduction
Data Cleaning is the process of identifying, correcting, removing, or handling incorrect, incomplete, duplicate, and inconsistent data.
It is an important step in the Data Science process because poor-quality data can produce incorrect analysis and unreliable machine learning results.
Data Cleaning converts raw and messy data into clean, consistent, and useful data.
Why is Data Cleaning Important?
Raw data may contain:
- Missing values
- Duplicate records
- Incorrect values
- Spelling mistakes
- Inconsistent formats
- Outliers
- Unnecessary data
- Incorrect data types
Clean data helps to:
- Improve data quality.
- Produce more accurate analysis.
- Improve machine learning models.
- Reduce errors.
- Make visualization more reliable.
- Support better decision-making.
Common Data Cleaning Techniques
1. Handling Missing Values
A missing value occurs when information is not available for a particular record.
Example
Name Age City Anuradha 22 Janakpur Ram - Kathmandu Sita 21 -
The missing values can be handled by:
- Removing the row.
- Replacing it with the mean.
- Replacing it with the median.
- Replacing it with the mode.
- Using another suitable value.
- Applying a suitable prediction or imputation method.
Python Example
import pandas as pd data = pd.read_csv("data.csv") data["Age"] = data["Age"].fillna(data["Age"].mean())
2. Removing Duplicate Data
Duplicate data occurs when the same record appears more than once.
Example
ID Name 101 Anuradha 102 Ram 101 Anuradha
The third record is a duplicate.
Python Example
data = data.drop_duplicates()
3. Correcting Inconsistent Data
Sometimes the same value is written in different formats.
Example
Janakpur janakpur JANAKPUR Janakpurdham
These values may need to be standardized depending on the purpose of the analysis.
Python Example
data["City"] = data["City"].str.lower()
4. Handling Incorrect Values
Data may contain values that are impossible or incorrect.
Example
Name Age Anuradha 22 Ram -5 Sita 21
An age of -5 is not valid.
The value should be investigated and corrected, replaced, or treated as missing.
5. Correcting Data Types
Data should have appropriate data types.
Example
A price may be stored as text:
"50000" "60000" "70000"
It should be converted to a numeric type before mathematical calculations.
Python Example
data["Price"] = pd.to_numeric(data["Price"], errors="coerce")
6. Handling Outliers
An outlier is a value that is unusually different from most other values.
Example
Salary: 25000 28000 30000 32000 5000000
5000000 is much larger than the other values and may be an outlier.
Outliers should not automatically be deleted. They should first be investigated to determine whether they are errors or legitimate observations.
7. Removing Unnecessary Columns
Some columns may not be useful for analysis.
Example
ID Name Age City Password
A Password column should not be included in an analysis dataset.
Unnecessary or sensitive information should be removed or excluded appropriately.
8. Standardizing Formats
Data should use consistent formats.
Date Example
Before:
01/02/2026 2026-02-01 Feb 1, 2026
After standardization:
2026-02-01 2026-02-01 2026-02-01
Python Example
data["Date"] = pd.to_datetime(data["Date"])
Data Cleaning Process
A typical data cleaning process is:
Raw Data ↓ Inspect Data ↓ Identify Problems ↓ Handle Missing Values ↓ Remove Duplicates ↓ Correct Errors ↓ Standardize Formats ↓ Handle Outliers ↓ Validate Data ↓ Clean Data
Data Cleaning Using Pandas
Pandas is a popular Python library for data manipulation and analysis.
Load Data
import pandas as pd data = pd.read_csv("sales.csv")
View Data
print(data.head())
Check Missing Values
print(data.isnull().sum())
Remove Duplicates
data = data.drop_duplicates()
Fill Missing Values
data["Price"] = data["Price"].fillna(data["Price"].mean())
Save Clean Data
data.to_csv("cleaned_data.csv", index=False)
Example
Suppose we have the following raw data:
Name Age City Salary Anuradha 22 Janakpur 30000 Ram - Kathmandu 35000 Sita 21 janakpur 32000 Anuradha 22 Janakpur 30000
Problems
Ageis missing for Ram.janakpuruses a different format.- Anuradha appears twice.
After Cleaning
Name Age City Salary Anuradha 22 Janakpur 30000 Ram 22 Kathmandu 35000 Sita 21 Janakpur 32000
Now the dataset is more consistent and suitable for further analysis.
Data Cleaning vs Data Transformation
| Data Cleaning | Data Transformation |
|---|---|
| Fixes errors and quality problems | Changes data into a suitable format |
| Handles missing values | Converts or derives variables |
| Removes duplicates | Aggregates data |
| Corrects inconsistent values | Normalizes or scales data |
| Improves data quality | Makes data suitable for analysis or modeling |
Both activities are often part of data preprocessing.
Advantages of Data Cleaning
- Improves data quality.
- Reduces errors.
- Improves analysis accuracy.
- Helps produce reliable visualizations.
- Improves machine learning performance.
- Makes data easier to understand.
- Supports better decision-making.
Challenges of Data Cleaning
- Large amounts of data can take time to clean.
- Missing values may be difficult to handle.
- Some errors require domain knowledge.
- Different data sources may use different formats.
- Outliers may be legitimate values.
- Cleaning rules can sometimes introduce unintended changes.
Best Practices
When cleaning data:
- Always keep a copy of the original data.
- Understand the dataset before modifying it.
- Check for missing values.
- Remove or investigate duplicates.
- Standardize formats.
- Validate corrected values.
- Document important cleaning steps.
- Protect sensitive information.
- Avoid deleting data without understanding why it is incorrect.
- Recheck the dataset after cleaning.
Conclusion
Data Cleaning is an essential step in Data Science. It involves identifying and handling missing values, duplicates, errors, inconsistent formats, incorrect data types, and unusual values.
Clean and reliable data provides a strong foundation for data analysis, visualization, and machine learning.
Garbage in, garbage out: if poor-quality data is used as input, the resulting analysis or model may also be unreliable.