Ugenz Academy/Tutorial

Decisiontrees

12 min read2,311 words

Reading progress

Keep learning

Decision Trees

1. Introduction

A Decision Tree is a supervised machine learning algorithm used for both classification and regression problems.

It makes predictions by repeatedly splitting data based on conditions or rules.

A decision tree looks similar to a flowchart, where:

  • Root node represents the starting condition.
  • Decision nodes represent questions or conditions.
  • Branches represent possible answers.
  • Leaf nodes represent the final prediction.

Simple Definition

A Decision Tree is a machine learning algorithm that makes predictions by dividing data into smaller groups using a sequence of decision rules.


2. Simple Example

Suppose we want to predict whether a student will pass based on study hours.

Study Hours > 4? / \ No Yes / \ Fail Pass

The tree asks a question:

Is Study Hours greater than 4?

If the answer is Yes, the prediction is Pass.

If the answer is No, the prediction is Fail.


3. Main Components of a Decision Tree

A Decision Tree contains several important components.

3.1 Root Node

The root node is the first and topmost node of the tree.

It represents the first condition used to divide the data.

Example:

Study Hours > 4?

3.2 Decision Node

A decision node contains a condition used to make another split.

Example:

Attendance > 75%?

3.3 Branch

A branch connects nodes and represents the result of a condition.

Example:

Yes No

3.4 Leaf Node

A leaf node is the final node that contains the prediction.

Example:

Pass Fail

4. Structure of a Decision Tree

A simple decision tree can be represented as:

Root | Is condition true? / \ Yes No | | Decision Decision / \ / \ / \ / \ Result Result Result Result

5. Types of Decision Trees

Decision Trees are mainly used for two types of machine learning tasks.

5.1 Classification Tree

A Classification Tree predicts a categorical class.

Examples

Pass / Fail Yes / No Spam / Not Spam Fraud / Not Fraud

Example

Income > 50,000? / \ Yes No / \ Approved Rejected

5.2 Regression Tree

A Regression Tree predicts a numerical value.

Examples

  • House price
  • Sales
  • Temperature
  • Salary
  • Product demand

Example:

House Size > 1000? / \ Yes No / \ 5,000,000 3,000,000

6. How Decision Trees Work

The general process is:

Collect Data ↓ Clean Data ↓ Select Features ↓ Choose Best Split ↓ Create Branches ↓ Repeat Splitting ↓ Create Leaf Nodes ↓ Make Prediction

The algorithm keeps splitting the data until a stopping condition is reached.


7. Example Dataset

Consider the following student dataset:

Study HoursAttendanceResult
260Fail
365Fail
470Fail
580Pass
685Pass
890Pass

A Decision Tree may learn:

Study Hours > 4? / \ No Yes / \ Fail Pass

The model can then use this rule to classify a new student.


8. Decision Tree Splitting

A Decision Tree needs to decide:

Which feature should be used for the next split?

The algorithm evaluates possible splits using measures such as:

  • Gini Impurity
  • Entropy
  • Information Gain
  • Variance Reduction

9. Entropy

Entropy measures the impurity or uncertainty in a dataset.

For classification, the formula is:

Entropy = -Σ pᵢ log₂(pᵢ)

Where:

  • pᵢ = probability of class i

Interpretation

Entropy = 0

means the node is completely pure.

A higher entropy indicates greater uncertainty.


10. Example of Entropy

Suppose a dataset contains:

5 Pass 0 Fail

The node contains only one class.

Therefore:

Entropy = 0

Now suppose:

5 Pass 5 Fail

The classes are evenly mixed, so the uncertainty is higher.

For two equally likely classes:

Entropy = 1

11. Information Gain

Information Gain measures how much a split reduces uncertainty.

Formula

Information Gain = Parent Entropy - Weighted Child Entropy

A split with higher information gain provides a greater reduction in uncertainty.

Decision Tree algorithms can use Information Gain to help choose a split.


12. Gini Impurity

Gini Impurity is another common measure used to select classification splits.

Formula

Gini = 1 - Σpᵢ²

Where:

  • pᵢ = proportion of observations belonging to class i

A lower Gini value means the node is more pure.


13. Entropy vs Gini Impurity

EntropyGini Impurity
Measures uncertaintyMeasures impurity
Uses logarithmUses squared probabilities
Based on information theoryComputationally simpler
Used by some tree algorithmsCommonly used by CART implementations
Lower value means greater purityLower value means greater purity

14. Splitting in Regression Trees

For regression problems, Decision Trees use criteria based on numerical prediction error.

A common approach is to minimize:

Mean Squared Error (MSE)

The algorithm searches for splits that make the resulting groups more homogeneous in their target values.


15. Stopping Conditions

A tree should not continue splitting forever.

Common stopping conditions include:

Maximum Depth

Limits how deep the tree can grow.

max_depth=5

Minimum Samples Split

Specifies the minimum number of samples required to split an internal node.

min_samples_split=10

Minimum Samples Leaf

Specifies the minimum number of samples required in a leaf.

min_samples_leaf=5

These parameters help control model complexity.


16. Overfitting in Decision Trees

A Decision Tree can become too complex and learn the training data too closely.

This is called overfitting.

Example

An extremely deep tree may memorize individual training observations rather than learning general patterns.

Simple Tree ↓ General Rules ↓ Better Generalization

But:

Very Deep Tree ↓ Complex Rules ↓ Possible Overfitting

17. Underfitting

Underfitting occurs when the tree is too simple to capture important patterns in the data.

For example:

Very Small Tree ↓ Too Few Rules ↓ Important Patterns Missed

Therefore, the goal is to find an appropriate level of complexity.


18. Pruning

Pruning is the process of reducing unnecessary branches from a Decision Tree.

It can help reduce overfitting.

There are two general approaches:

Pre-Pruning

Stop the tree from becoming too complex during training.

Examples:

max_depth min_samples_split min_samples_leaf

Post-Pruning

Build a larger tree and then remove branches that do not improve generalization.


19. Decision Tree Example

Suppose a bank wants to classify whether a loan application should be approved.

Features:

Income Credit Score Age Employment Status

Target:

Approved / Rejected

A possible tree:

Credit Score > 650? / \ Yes No / \ Income > 40K? Reject / \ Yes No / \ Approve Reject

The tree makes a prediction by following the appropriate branches.


20. Decision Tree Using Python

The scikit-learn library provides Decision Tree algorithms.

Installation

pip install scikit-learn

Simple Classification Example

from sklearn.tree import DecisionTreeClassifier # Features X = [ [1], [2], [3], [4], [5], [6] ] # Target # 0 = Fail # 1 = Pass y = [0, 0, 0, 1, 1, 1] # Create model model = DecisionTreeClassifier() # Train model model.fit(X, y) # Make prediction prediction = model.predict([[5]]) print("Prediction:", prediction[0])

Output:

Prediction: 1

Here:

1 = Pass

21. Decision Tree with Train-Test Split

import pandas as pd from sklearn.model_selection import train_test_split from sklearn.tree import DecisionTreeClassifier from sklearn.metrics import accuracy_score # Load dataset data = pd.read_csv("students.csv") # Features X = data[["StudyHours", "Attendance"]] # Target y = data["Pass"] # Split dataset X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42 ) # Create model model = DecisionTreeClassifier( max_depth=4, random_state=42 ) # Train model model.fit(X_train, y_train) # Predict y_pred = model.predict(X_test) # Evaluate accuracy = accuracy_score(y_test, y_pred) print("Accuracy:", accuracy)

22. Decision Tree for Regression

For numerical prediction, use DecisionTreeRegressor.

from sklearn.tree import DecisionTreeRegressor X = [ [500], [700], [900], [1100], [1300] ] y = [ 2000000, 2800000, 3500000, 4300000, 5000000 ] model = DecisionTreeRegressor( max_depth=3, random_state=42 ) model.fit(X, y) prediction = model.predict([[1000]]) print("Predicted Price:", prediction[0])

Here:

X = House Size Y = House Price

23. Visualizing a Decision Tree

A Decision Tree can be visualized using plot_tree().

import matplotlib.pyplot as plt from sklearn.tree import plot_tree plt.figure(figsize=(12, 8)) plot_tree( model, filled=True, feature_names=["StudyHours"], class_names=["Fail", "Pass"] ) plt.show()

Visualization makes the decision rules easier to understand.


24. Important Parameters

Some important parameters in DecisionTreeClassifier and DecisionTreeRegressor include:

ParameterPurpose
criterionDefines the split-quality measure
max_depthControls maximum tree depth
min_samples_splitMinimum samples needed to split a node
min_samples_leafMinimum samples allowed in a leaf
max_featuresControls features considered for a split
random_stateMakes results reproducible

25. Advantages of Decision Trees

  1. Easy to understand.
  2. Easy to visualize.
  3. Can be used for classification.
  4. Can be used for regression.
  5. Requires relatively little data preprocessing.
  6. Can model nonlinear relationships.
  7. Can handle numerical and categorical features after appropriate encoding.
  8. Does not require feature scaling.
  9. Produces interpretable decision rules.

26. Limitations of Decision Trees

  1. Can easily overfit.
  2. Small changes in data can produce a different tree.
  3. Very deep trees can become difficult to interpret.
  4. Individual trees may have lower predictive performance than ensemble methods on some datasets.
  5. Some implementations require categorical variables to be encoded numerically.
  6. Large trees can become complex.

27. Applications of Decision Trees

Healthcare

  • Patient classification
  • Medical diagnosis support
  • Risk classification

Banking

  • Loan classification
  • Fraud classification
  • Customer segmentation

Education

  • Student performance classification
  • Pass/fail prediction
  • Student dropout analysis

Marketing

  • Customer classification
  • Purchase prediction
  • Customer segmentation

E-Commerce

  • Product recommendation support
  • Customer behavior classification
  • Purchase prediction

Cybersecurity

  • Threat classification
  • Intrusion detection
  • Suspicious activity classification

28. Decision Tree vs Linear Regression

Decision TreeLinear Regression
Classification and regressionMainly regression
Uses decision rulesUses a mathematical linear relationship
Can model nonlinear patternsModels linear relationships
Does not require feature scalingScaling usually not required
Can easily overfitCan underfit nonlinear relationships
Easy to visualizeEquation is easy to interpret

29. Decision Tree vs Logistic Regression

Decision TreeLogistic Regression
Classification and regressionMainly classification
Uses tree-based rulesUses logistic function
Can model nonlinear decision boundariesBasic form has a linear decision boundary
Easy to visualizeCoefficients can be interpreted
Does not require feature scalingScaling can be useful in some settings
Can overfit easilyUsually less flexible than a deep tree

30. Random Forest and Decision Trees

A Random Forest is an ensemble learning algorithm that combines predictions from many Decision Trees.

Decision Tree 1 ──┐ Decision Tree 2 ──┤ Decision Tree 3 ──┤ Decision Tree 4 ──┤──> Combined Prediction Decision Tree 5 ──┘

Instead of relying on one tree, Random Forest uses multiple trees to improve robustness and often reduce overfitting.


31. Real-Life Example

Suppose an online store wants to predict whether a customer will purchase a product.

The available features are:

Age Income Previous Purchases Website Visits Time Spent

Target:

Purchase = Yes / No

A Decision Tree can create rules such as:

Website Visits > 5? / \ Yes No / \ Previous Purchase? No / \ Yes No / \ Yes Maybe

The exact tree depends on the training data and selected parameters.


32. Decision Tree Workflow

Dataset ↓ Data Cleaning ↓ Feature Selection ↓ Train/Test Split ↓ Choose Best Split ↓ Build Decision Tree ↓ Make Predictions ↓ Evaluate Performance ↓ Tune the Parameters

33. Key Points to Remember

  • Decision Tree is a supervised machine learning algorithm.
  • It can solve both classification and regression problems.
  • It works using a sequence of decision rules.
  • The first node is called the root node.
  • Internal nodes represent decisions.
  • Branches represent outcomes.
  • Leaf nodes represent final predictions.
  • Common classification split measures include Gini Impurity and Entropy.
  • Information Gain can be used to evaluate classification splits.
  • Decision Trees can overfit if they become too deep.
  • Pruning and tree-depth controls can help reduce overfitting.
  • Decision Trees do not generally require feature scaling.

Conclusion

Decision Trees are simple and powerful machine learning algorithms that make predictions through a sequence of decision rules. They can be used for both classification and regression and are easy to understand and visualize. However, individual trees can overfit, so parameters such as max_depth, min_samples_split, and min_samples_leaf are often used to control their complexity.

Keep learning. Keep building. Keep growing.

© 2026 Ugenz Academy