Data Preprocessing
Introduction
Data Preprocessing is the process of preparing raw data so that it can be used effectively for data analysis, visualization, and machine learning.
Raw data may contain missing values, duplicate records, inconsistent formats, categorical values, and unnecessary information. Data preprocessing transforms this raw data into a clean and suitable format.
Data preprocessing is the process of converting raw data into a suitable format for analysis and machine learning.
Why is Data Preprocessing Important?
Data preprocessing is important because real-world data is often messy and incomplete.
It helps to:
- Improve data quality.
- Handle missing values.
- Remove duplicate records.
- Convert data into suitable formats.
- Reduce noise.
- Handle outliers.
- Convert categorical data into numerical data.
- Scale numerical features.
- Improve machine learning performance.
Data Preprocessing Steps
A typical data preprocessing workflow is:
Raw Data ↓ Data Collection ↓ Data Cleaning ↓ Data Integration ↓ Data Transformation ↓ Feature Selection ↓ Feature Engineering ↓ Data Reduction ↓ Data Splitting ↓ Ready for Analysis / Machine Learning
1. Data Cleaning
Data cleaning identifies and handles problems in the dataset.
Common Tasks
- Handling missing values
- Removing duplicates
- Correcting incorrect values
- Handling inconsistent data
- Handling outliers
Example
Before cleaning:
Name Age Anuradha 22 Ram - Sita 21
The missing age for Ram needs to be handled appropriately.
2. Handling Missing Values
Missing values can be handled using different methods.
Method 1: Remove Rows
data = data.dropna()
Method 2: Fill With Mean
data["Age"] = data["Age"].fillna(data["Age"].mean())
Method 3: Fill With Median
data["Age"] = data["Age"].fillna(data["Age"].median())
Method 4: Fill With Mode
data["City"] = data["City"].fillna(data["City"].mode()[0])
The appropriate method depends on the dataset and the reason values are missing.
3. Removing Duplicate Data
Duplicate records can affect analysis and machine learning.
Example
ID Name 101 Anuradha 102 Ram 101 Anuradha
Remove duplicates using:
data = data.drop_duplicates()
4. Handling Outliers
An outlier is a value that is unusually different from the other observations.
Example
Salary: 25000 28000 30000 32000 5000000
The value 5000000 may be an outlier.
Outliers should be investigated before deciding whether to remove, transform, or keep them.
5. Data Integration
Data Integration means combining data from multiple sources into a single dataset.
Example
A company may have:
Customer Database + Sales Database + Product Database ↓ Integrated Dataset
Python Example
data = pd.merge(customers, sales, on="CustomerID")
6. Data Transformation
Data transformation changes data into a suitable format.
Common transformations include:
- Scaling
- Normalization
- Standardization
- Encoding
- Aggregation
- Log transformation
7. Encoding Categorical Data
Machine learning algorithms often require numerical input. Therefore, categorical values may need to be converted into numbers.
Example
Original data:
Gender Male Female Female Male
Label Encoding
Male → 0 Female → 1
One-Hot Encoding
Gender_Male Gender_Female 1 0 0 1 0 1 1 0
Python Example
data = pd.get_dummies(data, columns=["Gender"])
8. Feature Scaling
Features may have very different numerical ranges.
Example
Age = 22 Salary = 50000
Some machine learning algorithms can be affected by these differences in scale.
Two common methods are:
Min-Max Normalization
It generally converts values to a fixed range, often 0 to 1.
Formula:
X' = (X - Xmin) / (Xmax - Xmin)
Standardization
Standardization transforms data using the mean and standard deviation.
Formula:
Z = (X - μ) / σ
9. Feature Selection
Feature Selection means selecting the most useful variables for analysis or machine learning.
Example
Suppose a dataset contains:
Age Gender Income City Customer_ID Purchase_History Favorite_Color
For a purchase prediction task, some variables may be more useful than others.
Feature selection can:
- Reduce unnecessary information.
- Reduce model complexity.
- Improve training efficiency.
- Sometimes improve model performance.
10. Feature Engineering
Feature Engineering means creating new useful features from existing data.
Example
Given:
Date of Birth Current Date
We can create:
Age
Another example:
Quantity × Price = Total Sales
Python:
data["Total_Sales"] = data["Quantity"] * data["Price"]
11. Data Reduction
Data Reduction reduces the size or complexity of data while trying to preserve important information.
Techniques
- Feature selection
- Sampling
- Aggregation
- Dimensionality reduction
Example
A dataset containing millions of records may be sampled for exploratory analysis when using the full dataset is unnecessary.
12. Train-Test Split
For machine learning, data is commonly divided into training and testing sets.
Training Data
Used to train the machine learning model.
Testing Data
Used to evaluate how the trained model performs on unseen data.
Example:
Dataset ↓ Training Data → 80% Testing Data → 20%
Python Example
from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42 )
The exact split depends on the problem and dataset.
Example of Data Preprocessing
Suppose we have:
Name Age Gender Salary Anuradha 22 Female 30000 Ram - Male 35000 Sita 21 Female 32000 Anuradha 22 Female 30000
Problems
- Age is missing for Ram.
- Anuradha's record is duplicated.
- Gender is categorical.
Preprocessing
Step 1: Handle missing age
Ram → Replace missing Age with an appropriate value
Step 2: Remove duplicate
Remove the repeated Anuradha record.
Step 3: Encode Gender
Female → 0 Male → 1
Step 4: Scale numerical features if required
Age and Salary can be scaled depending on the machine learning algorithm.
The resulting data is now more suitable for machine learning.
Data Preprocessing Using Python
A simple example using Pandas and Scikit-learn:
import pandas as pd from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler data = pd.read_csv("data.csv") # Remove duplicate records data = data.drop_duplicates() # Fill missing numerical values data["Age"] = data["Age"].fillna(data["Age"].median()) # Select features and target X = data[["Age", "Salary"]] y = data["Purchased"] # Split data X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42 ) # Standardize features scaler = StandardScaler() X_train = scaler.fit_transform(X_train) X_test = scaler.transform(X_test)
Data Cleaning vs Data Preprocessing
| Data Cleaning | Data Preprocessing |
|---|---|
| Focuses mainly on improving data quality | Broader process of preparing data |
| Handles missing values | Includes cleaning, transformation, encoding, scaling, etc. |
| Removes duplicates | Can include feature engineering |
| Corrects errors | Can include train-test splitting |
| Is a part of preprocessing | Includes multiple preparation steps |
Data cleaning is a part of data preprocessing.
Common Tools
| Tool | Use |
|---|---|
| Python | Data preprocessing and automation |
| Pandas | Data cleaning and manipulation |
| NumPy | Numerical operations |
| Scikit-learn | Encoding, scaling, splitting, and preprocessing |
| Excel | Basic data cleaning |
| SQL | Data extraction and transformation |
| Power BI | Data transformation and preparation |
Best Practices
When preprocessing data:
- Understand the dataset first.
- Keep the original data unchanged.
- Handle missing values carefully.
- Check for duplicates.
- Investigate outliers.
- Encode categorical variables appropriately.
- Scale features when required.
- Select useful features.
- Split data appropriately for machine learning.
- Avoid data leakage.
What is Data Leakage?
Data leakage occurs when information that should not be available during model training is accidentally used by the model.
For example, information from the test set should not be used to calculate training transformations.
Conclusion
Data Preprocessing is an essential stage of Data Science and Machine Learning. It converts raw data into a clean, consistent, and suitable form for analysis and modeling.
The major preprocessing activities include:
- Data cleaning
- Missing-value handling
- Duplicate removal
- Outlier handling
- Data integration
- Data transformation
- Encoding
- Feature scaling
- Feature selection
- Feature engineering
- Data reduction
- Train-test splitting
Better-prepared data provides a stronger foundation for reliable analysis and machine learning.