Ugenz Academy/Tutorial

Datapreprocessing

8 min read1,403 words

Reading progress

Keep learning

Data Preprocessing

Introduction

Data Preprocessing is the process of preparing raw data so that it can be used effectively for data analysis, visualization, and machine learning.

Raw data may contain missing values, duplicate records, inconsistent formats, categorical values, and unnecessary information. Data preprocessing transforms this raw data into a clean and suitable format.

Data preprocessing is the process of converting raw data into a suitable format for analysis and machine learning.


Why is Data Preprocessing Important?

Data preprocessing is important because real-world data is often messy and incomplete.

It helps to:

  • Improve data quality.
  • Handle missing values.
  • Remove duplicate records.
  • Convert data into suitable formats.
  • Reduce noise.
  • Handle outliers.
  • Convert categorical data into numerical data.
  • Scale numerical features.
  • Improve machine learning performance.

Data Preprocessing Steps

A typical data preprocessing workflow is:

Raw Data ↓ Data Collection ↓ Data Cleaning ↓ Data Integration ↓ Data Transformation ↓ Feature Selection ↓ Feature Engineering ↓ Data Reduction ↓ Data Splitting ↓ Ready for Analysis / Machine Learning

1. Data Cleaning

Data cleaning identifies and handles problems in the dataset.

Common Tasks

  • Handling missing values
  • Removing duplicates
  • Correcting incorrect values
  • Handling inconsistent data
  • Handling outliers

Example

Before cleaning:

Name Age Anuradha 22 Ram - Sita 21

The missing age for Ram needs to be handled appropriately.


2. Handling Missing Values

Missing values can be handled using different methods.

Method 1: Remove Rows

data = data.dropna()

Method 2: Fill With Mean

data["Age"] = data["Age"].fillna(data["Age"].mean())

Method 3: Fill With Median

data["Age"] = data["Age"].fillna(data["Age"].median())

Method 4: Fill With Mode

data["City"] = data["City"].fillna(data["City"].mode()[0])

The appropriate method depends on the dataset and the reason values are missing.


3. Removing Duplicate Data

Duplicate records can affect analysis and machine learning.

Example

ID Name 101 Anuradha 102 Ram 101 Anuradha

Remove duplicates using:

data = data.drop_duplicates()

4. Handling Outliers

An outlier is a value that is unusually different from the other observations.

Example

Salary: 25000 28000 30000 32000 5000000

The value 5000000 may be an outlier.

Outliers should be investigated before deciding whether to remove, transform, or keep them.


5. Data Integration

Data Integration means combining data from multiple sources into a single dataset.

Example

A company may have:

Customer Database + Sales Database + Product Database ↓ Integrated Dataset

Python Example

data = pd.merge(customers, sales, on="CustomerID")

6. Data Transformation

Data transformation changes data into a suitable format.

Common transformations include:

  • Scaling
  • Normalization
  • Standardization
  • Encoding
  • Aggregation
  • Log transformation

7. Encoding Categorical Data

Machine learning algorithms often require numerical input. Therefore, categorical values may need to be converted into numbers.

Example

Original data:

Gender Male Female Female Male

Label Encoding

Male → 0 Female → 1

One-Hot Encoding

Gender_Male Gender_Female 1 0 0 1 0 1 1 0

Python Example

data = pd.get_dummies(data, columns=["Gender"])

8. Feature Scaling

Features may have very different numerical ranges.

Example

Age = 22 Salary = 50000

Some machine learning algorithms can be affected by these differences in scale.

Two common methods are:

Min-Max Normalization

It generally converts values to a fixed range, often 0 to 1.

Formula:

X' = (X - Xmin) / (Xmax - Xmin)

Standardization

Standardization transforms data using the mean and standard deviation.

Formula:

Z = (X - μ) / σ

9. Feature Selection

Feature Selection means selecting the most useful variables for analysis or machine learning.

Example

Suppose a dataset contains:

Age Gender Income City Customer_ID Purchase_History Favorite_Color

For a purchase prediction task, some variables may be more useful than others.

Feature selection can:

  • Reduce unnecessary information.
  • Reduce model complexity.
  • Improve training efficiency.
  • Sometimes improve model performance.

10. Feature Engineering

Feature Engineering means creating new useful features from existing data.

Example

Given:

Date of Birth Current Date

We can create:

Age

Another example:

Quantity × Price = Total Sales

Python:

data["Total_Sales"] = data["Quantity"] * data["Price"]

11. Data Reduction

Data Reduction reduces the size or complexity of data while trying to preserve important information.

Techniques

  • Feature selection
  • Sampling
  • Aggregation
  • Dimensionality reduction

Example

A dataset containing millions of records may be sampled for exploratory analysis when using the full dataset is unnecessary.


12. Train-Test Split

For machine learning, data is commonly divided into training and testing sets.

Training Data

Used to train the machine learning model.

Testing Data

Used to evaluate how the trained model performs on unseen data.

Example:

Dataset ↓ Training Data → 80% Testing Data → 20%

Python Example

from sklearn.model_selection import train_test_split X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42 )

The exact split depends on the problem and dataset.


Example of Data Preprocessing

Suppose we have:

Name Age Gender Salary Anuradha 22 Female 30000 Ram - Male 35000 Sita 21 Female 32000 Anuradha 22 Female 30000

Problems

  1. Age is missing for Ram.
  2. Anuradha's record is duplicated.
  3. Gender is categorical.

Preprocessing

Step 1: Handle missing age

Ram → Replace missing Age with an appropriate value

Step 2: Remove duplicate

Remove the repeated Anuradha record.

Step 3: Encode Gender

Female → 0 Male → 1

Step 4: Scale numerical features if required

Age and Salary can be scaled depending on the machine learning algorithm.

The resulting data is now more suitable for machine learning.


Data Preprocessing Using Python

A simple example using Pandas and Scikit-learn:

import pandas as pd from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler data = pd.read_csv("data.csv") # Remove duplicate records data = data.drop_duplicates() # Fill missing numerical values data["Age"] = data["Age"].fillna(data["Age"].median()) # Select features and target X = data[["Age", "Salary"]] y = data["Purchased"] # Split data X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42 ) # Standardize features scaler = StandardScaler() X_train = scaler.fit_transform(X_train) X_test = scaler.transform(X_test)

Data Cleaning vs Data Preprocessing

Data CleaningData Preprocessing
Focuses mainly on improving data qualityBroader process of preparing data
Handles missing valuesIncludes cleaning, transformation, encoding, scaling, etc.
Removes duplicatesCan include feature engineering
Corrects errorsCan include train-test splitting
Is a part of preprocessingIncludes multiple preparation steps

Data cleaning is a part of data preprocessing.


Common Tools

ToolUse
PythonData preprocessing and automation
PandasData cleaning and manipulation
NumPyNumerical operations
Scikit-learnEncoding, scaling, splitting, and preprocessing
ExcelBasic data cleaning
SQLData extraction and transformation
Power BIData transformation and preparation

Best Practices

When preprocessing data:

  1. Understand the dataset first.
  2. Keep the original data unchanged.
  3. Handle missing values carefully.
  4. Check for duplicates.
  5. Investigate outliers.
  6. Encode categorical variables appropriately.
  7. Scale features when required.
  8. Select useful features.
  9. Split data appropriately for machine learning.
  10. Avoid data leakage.

What is Data Leakage?

Data leakage occurs when information that should not be available during model training is accidentally used by the model.

For example, information from the test set should not be used to calculate training transformations.


Conclusion

Data Preprocessing is an essential stage of Data Science and Machine Learning. It converts raw data into a clean, consistent, and suitable form for analysis and modeling.

The major preprocessing activities include:

  • Data cleaning
  • Missing-value handling
  • Duplicate removal
  • Outlier handling
  • Data integration
  • Data transformation
  • Encoding
  • Feature scaling
  • Feature selection
  • Feature engineering
  • Data reduction
  • Train-test splitting

Better-prepared data provides a stronger foundation for reliable analysis and machine learning.

Keep learning. Keep building. Keep growing.

© 2026 Ugenz Academy