Exploratory Data Analysis (EDA)
Introduction
Exploratory Data Analysis (EDA) is the process of examining, summarizing, and visualizing a dataset to understand its main characteristics, patterns, relationships, and unusual values.
EDA is an important step in Data Science because it helps us understand the data before applying statistical methods or building machine learning models.
EDA is the process of exploring and understanding data using statistics and visualizations.
Why is EDA Important?
EDA helps Data Scientists to:
- Understand the structure of data.
- Find patterns and trends.
- Identify missing values.
- Detect duplicate records.
- Find outliers.
- Understand relationships between variables.
- Discover errors or unusual values.
- Select useful features.
- Choose appropriate analysis or machine learning techniques.
EDA Process
A typical EDA process is:
Collect Data ↓ Load Data ↓ Understand Data ↓ Clean Data ↓ Statistical Analysis ↓ Data Visualization ↓ Find Patterns & Relationships ↓ Draw Insights
1. Understand the Dataset
The first step is to understand what the dataset contains.
Important questions include:
- How many rows are there?
- How many columns are there?
- What are the column names?
- What type of data does each column contain?
- Are there missing values?
- Are there duplicate records?
Python Example
import pandas as pd data = pd.read_csv("sales.csv") print(data.head()) print(data.shape) print(data.columns) print(data.info())
2. View the First and Last Records
First Five Rows
print(data.head())
Last Five Rows
print(data.tail())
These functions help us quickly inspect the dataset.
3. Check Dataset Size
The shape property gives the number of rows and columns.
print(data.shape)
Example output:
(1000, 6)
This means:
- 1000 rows
- 6 columns
4. Check Data Types
Different columns can contain different types of data.
print(data.dtypes)
Common data types include:
- Integer
- Float
- String
- Boolean
- Date/Time
Correct data types are important for analysis.
5. Check Missing Values
Missing values can affect analysis.
print(data.isnull().sum())
Example:
Name 0 Age 5 City 2 Salary 0
This tells us that:
- Age has 5 missing values.
- City has 2 missing values.
6. Statistical Summary
The describe() function provides statistical information about numerical columns.
print(data.describe())
It can provide:
- Count
- Mean
- Standard deviation
- Minimum
- 25th percentile
- Median
- 75th percentile
- Maximum
Example:
Age Salary count 100.0 100.0 mean 23.5 35000.0 min 18.0 20000.0 max 40.0 80000.0
7. Measures of Central Tendency
EDA often uses measures that describe the center of the data.
Mean
The mean is the average value.
Formula:
Mean = Sum of all values / Number of values
Python:
print(data["Salary"].mean())
Median
The median is the middle value after sorting the data.
print(data["Salary"].median())
Mode
The mode is the most frequently occurring value.
print(data["City"].mode())
8. Measures of Dispersion
Dispersion describes how spread out the data is.
Common measures include:
- Range
- Variance
- Standard deviation
Standard Deviation
print(data["Salary"].std())
A larger standard deviation generally indicates greater variation in the values.
9. Data Visualization
Visualization is an important part of EDA.
It helps us understand data more easily through graphs and charts.
Common visualizations include:
- Bar chart
- Line chart
- Histogram
- Box plot
- Scatter plot
- Pie chart
- Heatmap
10. Histogram
A histogram shows the distribution of numerical data.
Example
import matplotlib.pyplot as plt plt.hist(data["Age"]) plt.xlabel("Age") plt.ylabel("Frequency") plt.title("Age Distribution") plt.show()
A histogram can help identify the shape and spread of a distribution.
11. Bar Chart
A bar chart is useful for comparing categories.
Example
data["City"].value_counts().plot(kind="bar") plt.xlabel("City") plt.ylabel("Number of Customers") plt.title("Customers by City") plt.show()
12. Line Chart
A line chart is useful for showing changes over time.
Example
plt.plot(data["Date"], data["Sales"]) plt.xlabel("Date") plt.ylabel("Sales") plt.title("Sales Trend") plt.show()
13. Scatter Plot
A scatter plot helps examine the relationship between two numerical variables.
Example
plt.scatter(data["Age"], data["Salary"]) plt.xlabel("Age") plt.ylabel("Salary") plt.title("Age vs Salary") plt.show()
It can help identify whether there is a relationship or pattern between two variables.
14. Box Plot
A box plot helps understand the distribution of data and identify potential outliers.
plt.boxplot(data["Salary"]) plt.ylabel("Salary") plt.title("Salary Distribution") plt.show()
15. Correlation Analysis
Correlation measures the strength and direction of a relationship between numerical variables.
The correlation coefficient generally ranges from:
-1 to +1
+1→ Strong positive relationship0→ Little or no linear relationship-1→ Strong negative relationship
Python Example
print(data.corr(numeric_only=True))
16. Correlation Heatmap
A heatmap provides a visual representation of correlations.
import seaborn as sns import matplotlib.pyplot as plt sns.heatmap( data.corr(numeric_only=True), annot=True ) plt.title("Correlation Heatmap") plt.show()
17. Detecting Outliers
Outliers are values that are unusually different from the rest of the data.
Example
Salary: 25000 28000 30000 32000 5000000
The value 5000000 may be an outlier.
A box plot is commonly used to identify potential outliers.
plt.boxplot(data["Salary"]) plt.show()
Outliers should be investigated before deciding whether to remove or transform them.
18. Analyzing Categorical Data
Categorical variables contain groups or labels.
Examples:
Gender City Product Department
We can count categories using:
print(data["City"].value_counts())
Example:
Janakpur 50 Kathmandu 40 Birgunj 30
19. Example Dataset
Consider the following sales dataset:
| Date | Product | City | Quantity | Price | Total |
|---|---|---|---|---|---|
| 3-Apr | Laptop | Janakpur | 2 | 50000 | 100000 |
| 6-Jun | Mouse | Kathmandu | 10 | 1000 | 10000 |
| 2-Mar | Laptop | Birgunj | 1 | 80000 | 80000 |
| 2-Apr | Keyboard | Janakpur | 5 | 12000 | 60000 |
Using EDA, we can find:
- Which product has the highest sales.
- Which city has the highest sales.
- Total quantity sold.
- Average product price.
- Sales trends.
- Relationship between quantity and total sales.
20. Simple EDA Example in Python
import pandas as pd import matplotlib.pyplot as plt # Load data data = pd.read_csv("sales.csv") # Basic information print(data.head()) print(data.shape) print(data.info()) # Check missing values print(data.isnull().sum()) # Statistical summary print(data.describe()) # Product frequency print(data["Product"].value_counts()) # Total sales by product sales_by_product = data.groupby("Product")["Total"].sum() print(sales_by_product) # Visualization sales_by_product.plot(kind="bar") plt.xlabel("Product") plt.ylabel("Total Sales") plt.title("Sales by Product") plt.show()
EDA Tools
Common tools used for Exploratory Data Analysis include:
| Tool | Purpose |
|---|---|
| Python | Data analysis and visualization |
| Pandas | Data manipulation |
| NumPy | Numerical calculations |
| Matplotlib | Data visualization |
| Seaborn | Statistical visualization |
| Excel | Basic data exploration |
| Power BI | Interactive analysis and dashboards |
| R | Statistical analysis and visualization |
EDA vs Data Cleaning
| EDA | Data Cleaning |
|---|---|
| Focuses on understanding the data | Focuses on improving data quality |
| Finds patterns and relationships | Handles missing and incorrect values |
| Uses statistics and visualization | Corrects errors and inconsistencies |
| Helps generate insights | Prepares data for analysis |
| Can identify potential problems | Resolves identified data-quality problems |
Both activities are closely related and may be performed iteratively.
Benefits of EDA
- Helps understand the dataset.
- Identifies patterns and trends.
- Detects missing values.
- Finds potential outliers.
- Reveals relationships between variables.
- Helps select useful features.
- Supports better model selection.
- Improves understanding before machine learning.
Conclusion
Exploratory Data Analysis (EDA) is an important step in Data Science that helps us understand data before making conclusions or building machine learning models.
EDA combines statistical techniques, data summaries, and visualizations to discover patterns, relationships, trends, missing values, and unusual observations.
EDA helps answer the question: "What does my data tell me?"