Logistic Regression
1. Introduction
Logistic Regression is a supervised machine learning algorithm used mainly for classification problems.
Unlike Linear Regression, which predicts continuous numerical values, Logistic Regression predicts the probability that an observation belongs to a particular class.
Simple Definition
Logistic Regression is a classification algorithm that predicts the probability of an outcome using a logistic (sigmoid) function.
Examples
Logistic Regression can be used to predict:
- Pass or Fail
- Yes or No
- Spam or Not Spam
- Disease or No Disease
- Fraud or Not Fraud
- Customer Churn or No Churn
2. Linear Regression vs Logistic Regression
Linear Regression predicts a continuous numerical value.
Example:
House Price = Rs. 5,000,000
Logistic Regression predicts a class or probability.
Example:
Pass Probability = 0.85
Therefore:
| Linear Regression | Logistic Regression |
|---|---|
| Used for regression | Used mainly for classification |
| Predicts continuous values | Predicts probabilities/classes |
| Output can be any numerical value | Output probability is between 0 and 1 |
| Uses a straight-line model | Uses a logistic/sigmoid function |
| Example: Predict salary | Example: Predict pass/fail |
3. Classification
Classification is a machine learning task where the model assigns data to predefined categories.
Example
Suppose we want to predict whether a student will pass.
Input: Study Hours Output: Pass / Fail
The classes are:
0 = Fail 1 = Pass
4. Binary Classification
Logistic Regression is commonly used for binary classification.
Binary classification has two possible classes.
Examples
Yes / No True / False Pass / Fail Spam / Not Spam Fraud / Not Fraud
For example:
0 → Fail 1 → Pass
5. Logistic Regression Formula
The logistic regression model uses the sigmoid function.
The sigmoid function is:
P(Y = 1) = 1 / (1 + e^(-z))
Where:
z = β₀ + β₁X₁ + β₂X₂ + ... + βₙXₙ
Therefore:
P(Y = 1) = 1 / (1 + e^-(β₀ + β₁X₁ + ... + βₙXₙ))
Where:
P(Y = 1)= probability of class 1β₀= interceptβ₁, β₂, ...= model coefficientsX₁, X₂, ...= input variablese= Euler's number
6. Sigmoid Function
The sigmoid function converts any real-valued input into a value between 0 and 1.
Formula
σ(z) = 1 / (1 + e^-z)
Important properties
z → -∞ → sigmoid → 0 z = 0 → sigmoid = 0.5 z → +∞ → sigmoid → 1
The curve looks approximately like:
Probability 1.0 | ______ | __/ 0.5 |--------------__/ | ___/ 0.0 |__________/ +--------------------------> z
7. Why Sigmoid Function is Used
A linear equation can produce values less than 0 or greater than 1.
For example:
-2 1.5 5
These values cannot represent probabilities.
The sigmoid function converts the result into:
0 ≤ Probability ≤ 1
Therefore, it is useful for classification.
8. Probability and Classification
Suppose the model predicts:
Probability = 0.80
This means the model estimates an 80% probability for class 1.
A classification threshold is then used to convert the probability into a class.
A common threshold is:
0.5
Example
Probability >= 0.5 → Class 1 Probability < 0.5 → Class 0
Therefore:
| Probability | Predicted Class |
|---|---|
| 0.20 | 0 |
| 0.40 | 0 |
| 0.50 | 1 |
| 0.75 | 1 |
| 0.90 | 1 |
The threshold can be changed depending on the application.
9. Example: Student Pass Prediction
Suppose we want to predict whether a student will pass based on study hours.
| Study Hours | Result |
|---|---|
| 1 | Fail |
| 2 | Fail |
| 3 | Fail |
| 5 | Pass |
| 6 | Pass |
| 8 | Pass |
Convert the classes into numbers:
Fail = 0 Pass = 1
The model learns the relationship between:
Study Hours → Probability of Passing
If a student studies for 6 hours and the model predicts:
Probability = 0.92
Using a threshold of 0.5:
0.92 >= 0.5
Therefore:
Prediction = Pass
10. Types of Logistic Regression
There are three commonly discussed types.
10.1 Binary Logistic Regression
Used when there are two classes.
Example
Pass / Fail
10.2 Multinomial Logistic Regression
Used when there are more than two classes that do not have a natural order.
Example
Predicting transportation choice:
Bus Car Bike Train
10.3 Ordinal Logistic Regression
Used when the classes have a natural order.
Example
Customer satisfaction:
Poor Average Good Excellent
The categories have an order.
11. Logistic Regression Workflow
Collect Data ↓ Clean Data ↓ Prepare Data ↓ Select Features ↓ Split Dataset ↓ Train Logistic Regression ↓ Predict Probabilities ↓ Apply Classification Threshold ↓ Evaluate Model
12. Steps to Build a Logistic Regression Model
Step 1: Collect Data
Collect relevant data from sources such as:
- Databases
- CSV files
- Excel files
- APIs
- Surveys
- Applications
Step 2: Clean the Data
Handle:
- Missing values
- Duplicate records
- Incorrect values
- Outliers
- Incorrect data types
Step 3: Select Features
Choose variables that may help predict the target.
Example:
Study Hours Attendance Previous Score
Target:
Pass / Fail
Step 4: Encode the Target
Convert categories into numerical values.
Example:
Fail = 0 Pass = 1
Step 5: Split the Dataset
The data can be divided into:
Training Data Testing Data
For example:
80% → Training 20% → Testing
The exact split can vary depending on the dataset.
Step 6: Train the Model
The model learns the relationship between the features and the target classes.
Step 7: Make Predictions
The model can produce:
Probability
and then:
Predicted Class
Step 8: Evaluate the Model
Common evaluation measures include:
- Accuracy
- Precision
- Recall
- F1-score
- Confusion Matrix
- ROC-AUC
13. Confusion Matrix
A confusion matrix summarizes classification predictions.
For binary classification:
| Predicted Positive | Predicted Negative | |
|---|---|---|
| Actual Positive | True Positive | False Negative |
| Actual Negative | False Positive | True Negative |
True Positive (TP)
Actual class is positive and prediction is positive.
True Negative (TN)
Actual class is negative and prediction is negative.
False Positive (FP)
Actual class is negative but prediction is positive.
False Negative (FN)
Actual class is positive but prediction is negative.
14. Accuracy
Accuracy measures the proportion of predictions that are correct.
Formula
Accuracy = (TP + TN) / (TP + TN + FP + FN)
Example
Suppose:
TP = 40 TN = 45 FP = 5 FN = 10
Then:
Accuracy = (40 + 45) / (40 + 45 + 5 + 10) Accuracy = 85 / 100 Accuracy = 0.85
Therefore:
Accuracy = 85%
15. Precision
Precision measures how many of the observations predicted as positive were actually positive.
Formula
Precision = TP / (TP + FP)
Example
TP = 40 FP = 5
Then:
Precision = 40 / (40 + 5) = 40 / 45 ≈ 0.889
Therefore:
Precision ≈ 88.9%
16. Recall
Recall measures how many of the actual positive cases were correctly identified.
Formula
Recall = TP / (TP + FN)
Example
TP = 40 FN = 10
Then:
Recall = 40 / (40 + 10) = 0.80
Therefore:
Recall = 80%
17. F1-Score
F1-score combines Precision and Recall using their harmonic mean.
Formula
F1 = 2 × (Precision × Recall) / (Precision + Recall)
F1-score is useful when both precision and recall are important.
18. ROC Curve and AUC
The ROC curve shows the trade-off between:
- True Positive Rate
- False Positive Rate
for different classification thresholds.
AUC
AUC stands for:
Area Under the Curve
AUC summarizes the model's ability to distinguish between classes across thresholds.
19. Cost Function
Logistic Regression commonly uses log loss, also called cross-entropy loss, to measure prediction error.
For binary classification:
Log Loss = -[y log(p) + (1-y) log(1-p)]
Where:
y= actual classp= predicted probability
The model learns coefficients that reduce the overall loss on the training data.
20. Logistic Regression Using Python
The scikit-learn library provides Logistic Regression.
Installation
pip install scikit-learn
Simple Example
import numpy as np from sklearn.linear_model import LogisticRegression # Study hours X = np.array([[1], [2], [3], [4], [5], [6], [7], [8]]) # 0 = Fail, 1 = Pass y = np.array([0, 0, 0, 0, 1, 1, 1, 1]) # Create model model = LogisticRegression() # Train model model.fit(X, y) # Predict class prediction = model.predict([[6]]) print("Prediction:", prediction[0])
Output:
Prediction: 1
Here:
1 = Pass
21. Predicting Probability
We can use predict_proba() to obtain probabilities.
probability = model.predict_proba([[6]]) print(probability)
The output contains probabilities for each class:
[Probability of Class 0, Probability of Class 1]
For example:
[0.08, 0.92]
This means:
Fail = 8% Pass = 92%
22. Logistic Regression with Train-Test Split
import pandas as pd from sklearn.model_selection import train_test_split from sklearn.linear_model import LogisticRegression from sklearn.metrics import accuracy_score # Load dataset data = pd.read_csv("students.csv") # Features X = data[["Hours"]] # Target y = data["Pass"] # Split data X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42 ) # Create model model = LogisticRegression() # Train model model.fit(X_train, y_train) # Make predictions y_pred = model.predict(X_test) # Calculate accuracy accuracy = accuracy_score(y_test, y_pred) print("Accuracy:", accuracy)
23. Multiple Features
Logistic Regression can use multiple input variables.
For example, student performance can be predicted using:
Study Hours Attendance Previous Score Assignment Score
Python example:
X = data[ [ "StudyHours", "Attendance", "PreviousScore", "AssignmentScore" ] ] y = data["Pass"]
The model uses all selected features to estimate the probability of passing.
24. Advantages of Logistic Regression
- Simple and easy to understand.
- Easy to implement.
- Computationally efficient.
- Produces probability estimates.
- Works well for many binary classification problems.
- Coefficients can provide interpretable information about predictors.
- Can handle multiple input variables.
- Provides a useful baseline classification model.
25. Limitations of Logistic Regression
- It assumes a suitable relationship between predictors and the log-odds of the outcome.
- It may perform poorly for highly nonlinear decision boundaries unless features are transformed or additional features are added.
- It can be sensitive to multicollinearity.
- Outliers can affect the fitted model.
- Feature scaling can be useful, especially when regularization is used.
- Class imbalance can make accuracy misleading.
- It may not perform well when the classes cannot be adequately separated by the available features.
26. Applications of Logistic Regression
Healthcare
Used for classification tasks such as:
- Disease risk classification
- Patient outcome analysis
- Medical screening
Banking
Used for:
- Fraud classification
- Loan default analysis
- Customer response prediction
Marketing
Used for:
- Customer purchase prediction
- Campaign response prediction
- Customer churn classification
Education
Used for:
- Pass/fail prediction
- Student dropout analysis
- Admission prediction
Cybersecurity
Used for:
- Spam classification
- Suspicious activity classification
- Threat detection
E-Commerce
Used for:
- Purchase prediction
- Customer churn
- Product response classification
27. Logistic Regression Example in Real Life
Suppose an online shopping company wants to predict whether a customer will purchase a product.
Features:
Age Time spent on website Number of pages visited Previous purchases
Target:
Purchase
Convert the target:
No Purchase = 0 Purchase = 1
The model might produce:
Probability = 0.78
Using a threshold of 0.5:
0.78 >= 0.5
Therefore:
Predicted Class = Purchase
The company can use this information as part of its customer analysis.
28. Important Difference: Linear vs Logistic Regression
| Feature | Linear Regression | Logistic Regression |
|---|---|---|
| Main task | Regression | Classification |
| Output | Continuous value | Probability/class |
| Typical target | Numerical | Categorical |
| Function | Linear equation | Sigmoid/logistic function |
| Output range | Not restricted to 0–1 | Probability between 0 and 1 |
| Example | Predict house price | Predict loan default |
| Common metrics | MAE, MSE, RMSE, R² | Accuracy, Precision, Recall, F1, ROC-AUC |
29. Important Formulas
Sigmoid Function
σ(z) = 1 / (1 + e^-z)
Logistic Model
z = β₀ + β₁X₁ + β₂X₂ + ... + βₙXₙ
Probability
P(Y = 1) = 1 / (1 + e^-z)
Accuracy
Accuracy = (TP + TN) / (TP + TN + FP + FN)
Precision
Precision = TP / (TP + FP)
Recall
Recall = TP / (TP + FN)
F1-Score
F1 = 2 × (Precision × Recall) / (Precision + Recall)
30. Key Points to Remember
- Logistic Regression is a supervised machine learning algorithm.
- It is mainly used for classification.
- Binary Logistic Regression is used for two classes.
- It uses the sigmoid function.
- The sigmoid function produces probabilities between
0and1. - A classification threshold converts probabilities into predicted classes.
0and1are commonly used to represent two classes.- Important evaluation metrics include Accuracy, Precision, Recall, F1-score, and ROC-AUC.
- Logistic Regression can use one or multiple input features.
- It is different from Linear Regression because it is mainly designed for classification.
Conclusion
Logistic Regression is a fundamental classification algorithm in machine learning. It estimates the probability of an observation belonging to a class and uses a classification threshold to produce a predicted class. It is relatively simple, interpretable, and useful in applications such as healthcare, banking, marketing, education, cybersecurity, and e-commerce.