Decision Trees
1. Introduction
A Decision Tree is a supervised machine learning algorithm used for both classification and regression problems.
It makes predictions by repeatedly splitting data based on conditions or rules.
A decision tree looks similar to a flowchart, where:
- Root node represents the starting condition.
- Decision nodes represent questions or conditions.
- Branches represent possible answers.
- Leaf nodes represent the final prediction.
Simple Definition
A Decision Tree is a machine learning algorithm that makes predictions by dividing data into smaller groups using a sequence of decision rules.
2. Simple Example
Suppose we want to predict whether a student will pass based on study hours.
Study Hours > 4? / \ No Yes / \ Fail Pass
The tree asks a question:
Is Study Hours greater than 4?
If the answer is Yes, the prediction is Pass.
If the answer is No, the prediction is Fail.
3. Main Components of a Decision Tree
A Decision Tree contains several important components.
3.1 Root Node
The root node is the first and topmost node of the tree.
It represents the first condition used to divide the data.
Example:
Study Hours > 4?
3.2 Decision Node
A decision node contains a condition used to make another split.
Example:
Attendance > 75%?
3.3 Branch
A branch connects nodes and represents the result of a condition.
Example:
Yes No
3.4 Leaf Node
A leaf node is the final node that contains the prediction.
Example:
Pass Fail
4. Structure of a Decision Tree
A simple decision tree can be represented as:
Root | Is condition true? / \ Yes No | | Decision Decision / \ / \ / \ / \ Result Result Result Result
5. Types of Decision Trees
Decision Trees are mainly used for two types of machine learning tasks.
5.1 Classification Tree
A Classification Tree predicts a categorical class.
Examples
Pass / Fail Yes / No Spam / Not Spam Fraud / Not Fraud
Example
Income > 50,000? / \ Yes No / \ Approved Rejected
5.2 Regression Tree
A Regression Tree predicts a numerical value.
Examples
- House price
- Sales
- Temperature
- Salary
- Product demand
Example:
House Size > 1000? / \ Yes No / \ 5,000,000 3,000,000
6. How Decision Trees Work
The general process is:
Collect Data ↓ Clean Data ↓ Select Features ↓ Choose Best Split ↓ Create Branches ↓ Repeat Splitting ↓ Create Leaf Nodes ↓ Make Prediction
The algorithm keeps splitting the data until a stopping condition is reached.
7. Example Dataset
Consider the following student dataset:
| Study Hours | Attendance | Result |
|---|---|---|
| 2 | 60 | Fail |
| 3 | 65 | Fail |
| 4 | 70 | Fail |
| 5 | 80 | Pass |
| 6 | 85 | Pass |
| 8 | 90 | Pass |
A Decision Tree may learn:
Study Hours > 4? / \ No Yes / \ Fail Pass
The model can then use this rule to classify a new student.
8. Decision Tree Splitting
A Decision Tree needs to decide:
Which feature should be used for the next split?
The algorithm evaluates possible splits using measures such as:
- Gini Impurity
- Entropy
- Information Gain
- Variance Reduction
9. Entropy
Entropy measures the impurity or uncertainty in a dataset.
For classification, the formula is:
Entropy = -Σ pᵢ log₂(pᵢ)
Where:
pᵢ= probability of classi
Interpretation
Entropy = 0
means the node is completely pure.
A higher entropy indicates greater uncertainty.
10. Example of Entropy
Suppose a dataset contains:
5 Pass 0 Fail
The node contains only one class.
Therefore:
Entropy = 0
Now suppose:
5 Pass 5 Fail
The classes are evenly mixed, so the uncertainty is higher.
For two equally likely classes:
Entropy = 1
11. Information Gain
Information Gain measures how much a split reduces uncertainty.
Formula
Information Gain = Parent Entropy - Weighted Child Entropy
A split with higher information gain provides a greater reduction in uncertainty.
Decision Tree algorithms can use Information Gain to help choose a split.
12. Gini Impurity
Gini Impurity is another common measure used to select classification splits.
Formula
Gini = 1 - Σpᵢ²
Where:
pᵢ= proportion of observations belonging to classi
A lower Gini value means the node is more pure.
13. Entropy vs Gini Impurity
| Entropy | Gini Impurity |
|---|---|
| Measures uncertainty | Measures impurity |
| Uses logarithm | Uses squared probabilities |
| Based on information theory | Computationally simpler |
| Used by some tree algorithms | Commonly used by CART implementations |
| Lower value means greater purity | Lower value means greater purity |
14. Splitting in Regression Trees
For regression problems, Decision Trees use criteria based on numerical prediction error.
A common approach is to minimize:
Mean Squared Error (MSE)
The algorithm searches for splits that make the resulting groups more homogeneous in their target values.
15. Stopping Conditions
A tree should not continue splitting forever.
Common stopping conditions include:
Maximum Depth
Limits how deep the tree can grow.
max_depth=5
Minimum Samples Split
Specifies the minimum number of samples required to split an internal node.
min_samples_split=10
Minimum Samples Leaf
Specifies the minimum number of samples required in a leaf.
min_samples_leaf=5
These parameters help control model complexity.
16. Overfitting in Decision Trees
A Decision Tree can become too complex and learn the training data too closely.
This is called overfitting.
Example
An extremely deep tree may memorize individual training observations rather than learning general patterns.
Simple Tree ↓ General Rules ↓ Better Generalization
But:
Very Deep Tree ↓ Complex Rules ↓ Possible Overfitting
17. Underfitting
Underfitting occurs when the tree is too simple to capture important patterns in the data.
For example:
Very Small Tree ↓ Too Few Rules ↓ Important Patterns Missed
Therefore, the goal is to find an appropriate level of complexity.
18. Pruning
Pruning is the process of reducing unnecessary branches from a Decision Tree.
It can help reduce overfitting.
There are two general approaches:
Pre-Pruning
Stop the tree from becoming too complex during training.
Examples:
max_depth min_samples_split min_samples_leaf
Post-Pruning
Build a larger tree and then remove branches that do not improve generalization.
19. Decision Tree Example
Suppose a bank wants to classify whether a loan application should be approved.
Features:
Income Credit Score Age Employment Status
Target:
Approved / Rejected
A possible tree:
Credit Score > 650? / \ Yes No / \ Income > 40K? Reject / \ Yes No / \ Approve Reject
The tree makes a prediction by following the appropriate branches.
20. Decision Tree Using Python
The scikit-learn library provides Decision Tree algorithms.
Installation
pip install scikit-learn
Simple Classification Example
from sklearn.tree import DecisionTreeClassifier # Features X = [ [1], [2], [3], [4], [5], [6] ] # Target # 0 = Fail # 1 = Pass y = [0, 0, 0, 1, 1, 1] # Create model model = DecisionTreeClassifier() # Train model model.fit(X, y) # Make prediction prediction = model.predict([[5]]) print("Prediction:", prediction[0])
Output:
Prediction: 1
Here:
1 = Pass
21. Decision Tree with Train-Test Split
import pandas as pd from sklearn.model_selection import train_test_split from sklearn.tree import DecisionTreeClassifier from sklearn.metrics import accuracy_score # Load dataset data = pd.read_csv("students.csv") # Features X = data[["StudyHours", "Attendance"]] # Target y = data["Pass"] # Split dataset X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42 ) # Create model model = DecisionTreeClassifier( max_depth=4, random_state=42 ) # Train model model.fit(X_train, y_train) # Predict y_pred = model.predict(X_test) # Evaluate accuracy = accuracy_score(y_test, y_pred) print("Accuracy:", accuracy)
22. Decision Tree for Regression
For numerical prediction, use DecisionTreeRegressor.
from sklearn.tree import DecisionTreeRegressor X = [ [500], [700], [900], [1100], [1300] ] y = [ 2000000, 2800000, 3500000, 4300000, 5000000 ] model = DecisionTreeRegressor( max_depth=3, random_state=42 ) model.fit(X, y) prediction = model.predict([[1000]]) print("Predicted Price:", prediction[0])
Here:
X = House Size Y = House Price
23. Visualizing a Decision Tree
A Decision Tree can be visualized using plot_tree().
import matplotlib.pyplot as plt from sklearn.tree import plot_tree plt.figure(figsize=(12, 8)) plot_tree( model, filled=True, feature_names=["StudyHours"], class_names=["Fail", "Pass"] ) plt.show()
Visualization makes the decision rules easier to understand.
24. Important Parameters
Some important parameters in DecisionTreeClassifier and DecisionTreeRegressor include:
| Parameter | Purpose |
|---|---|
criterion | Defines the split-quality measure |
max_depth | Controls maximum tree depth |
min_samples_split | Minimum samples needed to split a node |
min_samples_leaf | Minimum samples allowed in a leaf |
max_features | Controls features considered for a split |
random_state | Makes results reproducible |
25. Advantages of Decision Trees
- Easy to understand.
- Easy to visualize.
- Can be used for classification.
- Can be used for regression.
- Requires relatively little data preprocessing.
- Can model nonlinear relationships.
- Can handle numerical and categorical features after appropriate encoding.
- Does not require feature scaling.
- Produces interpretable decision rules.
26. Limitations of Decision Trees
- Can easily overfit.
- Small changes in data can produce a different tree.
- Very deep trees can become difficult to interpret.
- Individual trees may have lower predictive performance than ensemble methods on some datasets.
- Some implementations require categorical variables to be encoded numerically.
- Large trees can become complex.
27. Applications of Decision Trees
Healthcare
- Patient classification
- Medical diagnosis support
- Risk classification
Banking
- Loan classification
- Fraud classification
- Customer segmentation
Education
- Student performance classification
- Pass/fail prediction
- Student dropout analysis
Marketing
- Customer classification
- Purchase prediction
- Customer segmentation
E-Commerce
- Product recommendation support
- Customer behavior classification
- Purchase prediction
Cybersecurity
- Threat classification
- Intrusion detection
- Suspicious activity classification
28. Decision Tree vs Linear Regression
| Decision Tree | Linear Regression |
|---|---|
| Classification and regression | Mainly regression |
| Uses decision rules | Uses a mathematical linear relationship |
| Can model nonlinear patterns | Models linear relationships |
| Does not require feature scaling | Scaling usually not required |
| Can easily overfit | Can underfit nonlinear relationships |
| Easy to visualize | Equation is easy to interpret |
29. Decision Tree vs Logistic Regression
| Decision Tree | Logistic Regression |
|---|---|
| Classification and regression | Mainly classification |
| Uses tree-based rules | Uses logistic function |
| Can model nonlinear decision boundaries | Basic form has a linear decision boundary |
| Easy to visualize | Coefficients can be interpreted |
| Does not require feature scaling | Scaling can be useful in some settings |
| Can overfit easily | Usually less flexible than a deep tree |
30. Random Forest and Decision Trees
A Random Forest is an ensemble learning algorithm that combines predictions from many Decision Trees.
Decision Tree 1 ──┐ Decision Tree 2 ──┤ Decision Tree 3 ──┤ Decision Tree 4 ──┤──> Combined Prediction Decision Tree 5 ──┘
Instead of relying on one tree, Random Forest uses multiple trees to improve robustness and often reduce overfitting.
31. Real-Life Example
Suppose an online store wants to predict whether a customer will purchase a product.
The available features are:
Age Income Previous Purchases Website Visits Time Spent
Target:
Purchase = Yes / No
A Decision Tree can create rules such as:
Website Visits > 5? / \ Yes No / \ Previous Purchase? No / \ Yes No / \ Yes Maybe
The exact tree depends on the training data and selected parameters.
32. Decision Tree Workflow
Dataset ↓ Data Cleaning ↓ Feature Selection ↓ Train/Test Split ↓ Choose Best Split ↓ Build Decision Tree ↓ Make Predictions ↓ Evaluate Performance ↓ Tune the Parameters
33. Key Points to Remember
- Decision Tree is a supervised machine learning algorithm.
- It can solve both classification and regression problems.
- It works using a sequence of decision rules.
- The first node is called the root node.
- Internal nodes represent decisions.
- Branches represent outcomes.
- Leaf nodes represent final predictions.
- Common classification split measures include Gini Impurity and Entropy.
- Information Gain can be used to evaluate classification splits.
- Decision Trees can overfit if they become too deep.
- Pruning and tree-depth controls can help reduce overfitting.
- Decision Trees do not generally require feature scaling.
Conclusion
Decision Trees are simple and powerful machine learning algorithms that make predictions through a sequence of decision rules. They can be used for both classification and regression and are easy to understand and visualize. However, individual trees can overfit, so parameters such as max_depth, min_samples_split, and min_samples_leaf are often used to control their complexity.