Linear Regression
1. Introduction
Linear Regression is a supervised machine learning and statistical technique used to study the relationship between variables and predict a continuous numerical value.
It finds the best-fitting straight line through a set of data points.
Simple Definition
Linear Regression is a method used to predict a continuous dependent variable based on one or more independent variables by fitting a straight line to the data.
Example
Suppose we want to predict a student's exam score based on the number of hours they study.
| Study Hours | Exam Score |
|---|---|
| 1 | 40 |
| 2 | 45 |
| 3 | 55 |
| 4 | 65 |
| 5 | 75 |
Linear Regression can learn the relationship between study hours and exam score and predict the score for a new number of study hours.
2. Basic Terms
Linear Regression uses two important types of variables.
Independent Variable
The variable used to make the prediction.
It is usually represented by:
X
Example
Study Hours
Dependent Variable
The variable that we want to predict.
It is usually represented by:
Y
Example
Exam Score
Therefore:
X = Study Hours Y = Exam Score
3. Linear Regression Formula
The basic equation of simple linear regression is:
Y = a + bX
Where:
- Y = predicted value
- X = independent variable
- a = intercept
- b = slope/coefficient
Another common notation is:
Y = β₀ + β₁X
Where:
- β₀ = intercept
- β₁ = coefficient/slope
4. Intercept
The intercept is the predicted value of Y when X = 0.
For example:
Y = 30 + 10X
Here:
Intercept = 30
If:
X = 0
Then:
Y = 30 + 10(0) Y = 30
So, the predicted value is 30.
5. Slope
The slope represents how much the dependent variable changes when the independent variable increases by one unit.
For example:
Y = 30 + 10X
Here:
Slope = 10
This means that for every 1-unit increase in X, the predicted value of Y increases by 10 units.
6. Example of Linear Regression
Suppose the regression equation is:
Y = 20 + 5X
If:
X = 4
Then:
Y = 20 + 5(4) Y = 20 + 20 Y = 40
Therefore:
Predicted Y = 40
7. Types of Linear Regression
There are two major types of Linear Regression.
7.1 Simple Linear Regression
Simple Linear Regression uses one independent variable to predict one dependent variable.
Formula
Y = a + bX
Example
Predict salary based on years of experience.
Experience → Salary
There is only one independent variable:
X = Experience
7.2 Multiple Linear Regression
Multiple Linear Regression uses two or more independent variables to predict a dependent variable.
Formula
Y = β₀ + β₁X₁ + β₂X₂ + β₃X₃ + ... + βₙXₙ
Example
Predict house price using:
- House size
- Number of bedrooms
- Location
- Age of house
The model can be represented as:
Price = β₀ + β₁(Size) + β₂(Bedrooms) + β₃(Age)
8. Difference Between Simple and Multiple Linear Regression
| Simple Linear Regression | Multiple Linear Regression |
|---|---|
| Uses one independent variable | Uses two or more independent variables |
Y = a + bX | Y = β₀ + β₁X₁ + β₂X₂ + ... |
| Easier to understand | More complex |
| Example: Salary vs Experience | Example: Salary vs Experience + Education + Age |
| One predictor | Multiple predictors |
9. Best-Fit Line
Linear Regression attempts to find the best-fit line through the data points.
The line should represent the relationship between X and Y as closely as possible.
Y │ │ ● │ ● │ ● │ ● │ ● │________________________ X
The actual data points may not fall exactly on the line.
The difference between an actual value and its predicted value is called a residual or error.
10. Residual
A residual is the difference between the actual value and predicted value.
Formula
Residual = Actual Value - Predicted Value
Example
Suppose:
Actual value = 80 Predicted value = 75
Then:
Residual = 80 - 75 Residual = 5
11. Least Squares Method
Linear Regression commonly uses the Ordinary Least Squares (OLS) method to find the best-fitting line.
The method tries to minimize the sum of squared errors.
Sum of Squared Errors = Σ(Actual - Predicted)²
The model chooses coefficients that minimize this value.
Why square the errors?
Squaring:
- Makes negative errors positive.
- Gives larger errors more influence.
- Provides a mathematical objective that can be optimized.
12. How Linear Regression Works
The general process is:
Collect Data ↓ Clean Data ↓ Select Variables ↓ Split Data ↓ Train Model ↓ Find Best-Fit Line ↓ Make Predictions ↓ Evaluate Model
13. Steps to Build a Linear Regression Model
Step 1: Collect Data
Collect relevant data from sources such as:
- Databases
- Excel
- CSV files
- APIs
- Surveys
Step 2: Clean the Data
Check for:
- Missing values
- Duplicate records
- Incorrect values
- Outliers
- Incorrect data types
Step 3: Select Variables
Identify:
Independent variable(s) → X Dependent variable → Y
Step 4: Split the Data
The dataset is commonly divided into:
Training Data Testing Data
For example:
80% → Training 20% → Testing
The exact split can vary depending on the problem.
Step 5: Train the Model
The model learns the relationship between the input variables and the target variable.
Step 6: Make Predictions
Use the trained model to predict values for new data.
Step 7: Evaluate the Model
Common evaluation metrics include:
- MAE
- MSE
- RMSE
- R²
14. Mean Absolute Error (MAE)
MAE measures the average absolute difference between actual and predicted values.
Formula
MAE = (1/n) Σ |Actual - Predicted|
Example
If prediction errors are:
2, 4, 6
Then:
MAE = (2 + 4 + 6) / 3 = 4
A lower MAE generally indicates smaller prediction errors.
15. Mean Squared Error (MSE)
MSE calculates the average squared prediction error.
Formula
MSE = (1/n) Σ(Actual - Predicted)²
Larger errors have a greater effect because the errors are squared.
A lower MSE generally indicates better predictive accuracy.
16. Root Mean Squared Error (RMSE)
RMSE is the square root of MSE.
Formula
RMSE = √MSE
RMSE is useful because its unit is the same as the target variable.
For example, if the target is measured in rupees, RMSE is also measured in rupees.
17. R-Squared (R²)
R² measures how much of the variation in the dependent variable is explained by the regression model.
It is commonly written as:
R²
A value closer to 1 indicates that the model explains a larger proportion of the observed variation in the data.
However, R² should not be used alone to determine whether a model is appropriate.
18. Assumptions of Linear Regression
Important assumptions include:
1. Linearity
The relationship between the predictors and the target should be reasonably linear.
2. Independence
Observations should generally be independent of one another.
3. Homoscedasticity
The spread of residuals should be reasonably constant across fitted values.
4. Normality of Residuals
For many statistical inference procedures, residuals are assumed to be approximately normally distributed.
5. Limited Multicollinearity
In multiple regression, independent variables should not be excessively correlated with each other.
19. Correlation vs Linear Regression
Correlation measures the strength and direction of a linear relationship between two variables.
Linear Regression models that relationship and can be used for prediction.
| Correlation | Linear Regression |
|---|---|
| Measures relationship | Models relationship |
| Usually symmetric between variables | Distinguishes predictor and target |
| Correlation coefficient ranges from -1 to +1 | Produces an equation |
| Does not by itself define a prediction equation | Can be used for prediction |
| Does not establish causation | Does not establish causation |
20. Positive and Negative Relationships
Positive Relationship
When X increases, Y tends to increase.
X ↑ → Y ↑
Example:
Study Hours ↑ → Exam Score ↑
Negative Relationship
When X increases, Y tends to decrease.
X ↑ → Y ↓
Example:
Product Price ↑ → Demand ↓
The actual relationship depends on the data and context.
21. Linear Regression Using Python
The scikit-learn library provides tools for Linear Regression.
Installation
pip install scikit-learn
Example
import numpy as np from sklearn.linear_model import LinearRegression # Independent variable X = np.array([[1], [2], [3], [4], [5]]) # Dependent variable y = np.array([40, 50, 60, 70, 80]) # Create model model = LinearRegression() # Train model model.fit(X, y) # Predict prediction = model.predict([[6]]) print("Predicted value:", prediction[0])
The model learns the relationship between X and y and predicts the value for X = 6.
22. Finding Coefficients
We can find the intercept and slope using:
print("Intercept:", model.intercept_) print("Slope:", model.coef_[0])
The equation can then be represented as:
Y = Intercept + Slope × X
23. Linear Regression with a Dataset
Example using Pandas:
import pandas as pd from sklearn.model_selection import train_test_split from sklearn.linear_model import LinearRegression from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score # Load dataset data = pd.read_csv("data.csv") # Independent variable X = data[["Hours"]] # Dependent variable y = data["Score"] # Split data X_train, X_test, y_train, y_test = train_test_split( X, y, test_size=0.2, random_state=42 ) # Create model model = LinearRegression() # Train model model.fit(X_train, y_train) # Predict y_pred = model.predict(X_test) # Evaluate mae = mean_absolute_error(y_test, y_pred) mse = mean_squared_error(y_test, y_pred) rmse = mse ** 0.5 r2 = r2_score(y_test, y_pred) print("MAE:", mae) print("MSE:", mse) print("RMSE:", rmse) print("R²:", r2)
24. Example: House Price Prediction
Suppose we want to predict house prices.
Input:
House Size
Output:
House Price
Example data:
| Size (sq.ft.) | Price |
|---|---|
| 500 | 2000000 |
| 700 | 2800000 |
| 900 | 3600000 |
| 1100 | 4400000 |
The model can learn a relationship between house size and price.
After training, a new house size can be provided to estimate its price.
25. Applications of Linear Regression
Linear Regression is used in many areas.
Business
- Sales forecasting
- Revenue prediction
- Demand analysis
Finance
- Financial forecasting
- Risk analysis
- Investment analysis
Real Estate
- House price prediction
- Property valuation
Education
- Score prediction
- Student performance analysis
Healthcare
- Predicting numerical health-related measurements
- Analyzing relationships between variables
Marketing
- Sales and advertising analysis
- Customer behavior analysis
Economics
- Economic forecasting
- Relationship analysis
26. Advantages of Linear Regression
- Easy to understand.
- Simple to implement.
- Computationally efficient.
- Provides interpretable coefficients.
- Useful for numerical prediction.
- Works well when relationships are approximately linear.
- Can be used as a baseline model.
- Useful for statistical analysis.
27. Limitations of Linear Regression
- Assumes a linear relationship.
- Sensitive to outliers.
- Can be affected by multicollinearity.
- May perform poorly when relationships are strongly nonlinear.
- Requires appropriate data preparation.
- A high R² does not prove causation.
- Predictions outside the observed data range can be unreliable.
28. Linear Regression Example in Real Life
Suppose a business wants to predict monthly sales based on advertising expenditure.
Historical data:
| Advertising Cost | Sales |
|---|---|
| 10,000 | 100,000 |
| 20,000 | 150,000 |
| 30,000 | 210,000 |
| 40,000 | 270,000 |
The business can train a Linear Regression model using:
X = Advertising Cost Y = Sales
The model can then estimate sales for a new advertising budget.
29. Important Formulas
Linear Regression Equation
Y = a + bX
Multiple Linear Regression
Y = β₀ + β₁X₁ + β₂X₂ + ... + βₙXₙ
Residual
Residual = Actual - Predicted
MAE
MAE = (1/n) Σ|Actual - Predicted|
MSE
MSE = (1/n) Σ(Actual - Predicted)²
RMSE
RMSE = √MSE
30. Key Points to Remember
- Linear Regression is a supervised learning technique.
- It predicts a continuous numerical value.
- It models a relationship using a straight line.
Xrepresents the independent variable.Yrepresents the dependent variable.- Simple Linear Regression uses one predictor.
- Multiple Linear Regression uses multiple predictors.
- The basic equation is
Y = a + bX. - Least Squares is commonly used to find the best-fitting coefficients.
- MAE, MSE, RMSE, and R² are common evaluation measures.
- Linear Regression does not by itself prove causation.
Conclusion
Linear Regression is one of the fundamental techniques in statistics and machine learning. It is useful for understanding relationships between variables and predicting continuous numerical outcomes. Because its mathematical structure is relatively simple and its coefficients are interpretable, it is widely used as both a predictive model and a starting point for more advanced analysis.