K-Means Clustering
1. Introduction
K-Means Clustering is an unsupervised machine learning algorithm used to divide data into different groups called clusters.
It groups similar data points together based on their characteristics.
For example, a shopping company may have customer data such as:
- Age
- Income
- Spending score
K-Means can group customers into categories such as:
- Low-spending customers
- Medium-spending customers
- High-spending customers
2. What is K-Means?
K-Means is a clustering algorithm that divides a dataset into K clusters.
Here:
- K = Number of clusters
- Means = The algorithm calculates the mean/average position of points in each cluster.
Simple Definition
K-Means is an unsupervised learning algorithm that groups similar data points into K clusters based on their distance from cluster centers.
3. Why is K-Means Used?
K-Means is used when we have data but do not have predefined labels.
For example:
Customer Data ↓ K-Means ↓ ┌─────────────┐ │ Cluster 1 │ │ Cluster 2 │ │ Cluster 3 │ └─────────────┘
It can help discover hidden patterns in data.
4. Example
Suppose a company has customers with:
| Customer | Age | Income |
|---|---|---|
| A | 20 | 20,000 |
| B | 22 | 25,000 |
| C | 25 | 27,000 |
| D | 40 | 70,000 |
| E | 42 | 75,000 |
| F | 45 | 80,000 |
If we choose:
K = 2
K-Means may create:
Cluster 1
Young customers with lower income.
Cluster 2
Older customers with higher income.
The algorithm discovers these groups automatically.
5. Important Terms
5.1 Cluster
A cluster is a group of similar data points.
Example:
Cluster 1 → Young customers Cluster 2 → Middle-aged customers Cluster 3 → Older customers
5.2 K
K represents the number of clusters we want.
For example:
K = 2 → 2 clusters K = 3 → 3 clusters K = 5 → 5 clusters
Choosing an appropriate value of K is important.
5.3 Centroid
A centroid is the center point of a cluster.
For example:
● ● ● ★ Centroid
The centroid represents the average location of the data points in that cluster.
5.4 Distance
K-Means usually uses distance to determine which data points are close to each other.
The most common distance is Euclidean distance.
Euclidean Distance Formula
For two points:
A(x₁, y₁) B(x₂, y₂)
The distance is:
d = √((x₂ - x₁)² + (y₂ - y₁)²)
For multiple dimensions:
d = √Σ(xᵢ - yᵢ)²
6. How K-Means Works
K-Means follows an iterative process.
Step 1: Choose K
First, select the number of clusters.
Example:
K = 3
Step 2: Initialize Centroids
The algorithm selects initial centroid positions.
Centroid 1 Centroid 2 Centroid 3
Step 3: Assign Data Points
Each data point is assigned to the nearest centroid.
Data Point ↓ Calculate distance ↓ Find nearest centroid ↓ Assign to cluster
Step 4: Calculate New Centroids
After assigning the points, the algorithm calculates the new center of each cluster.
The new centroid is based on the mean of the points in that cluster.
Step 5: Repeat
The algorithm repeats:
Assign points ↓ Calculate centroids ↓ Assign points again ↓ Calculate centroids again
until the centroids stop changing significantly or the algorithm reaches its stopping condition.
7. K-Means Algorithm
The basic algorithm can be written as:
1. Choose K. 2. Select K initial centroids. 3. Calculate distance from each point to each centroid. 4. Assign each point to the nearest centroid. 5. Calculate new centroids. 6. Repeat steps 3–5. 7. Stop when the clusters become stable.
8. Simple Flow
Start ↓ Choose value of K ↓ Initialize centroids ↓ Calculate distances ↓ Assign data to clusters ↓ Calculate new centroids ↓ Centroids changed? ↙ ↘ Yes No ↓ ↓ Repeat Final Clusters
9. Objective of K-Means
K-Means tries to make the data points within each cluster as close as possible to their centroid.
It minimizes the Within-Cluster Sum of Squares (WCSS).
The objective function is:
WCSS = Σ Σ ||xᵢ - μⱼ||²
Where:
xᵢ= data pointμⱼ= centroid of the cluster||xᵢ - μⱼ||²= squared distance between a point and its centroid
A smaller WCSS generally means points are more compact around their centroids.
10. Choosing the Value of K
One of the most important tasks in K-Means is choosing the correct value of K.
A common method is the:
Elbow Method
The Elbow Method calculates WCSS for different values of K.
Example:
| K | WCSS |
|---|---|
| 1 | 500 |
| 2 | 300 |
| 3 | 180 |
| 4 | 160 |
| 5 | 150 |
We plot K against WCSS.
WCSS │ 500│ * │ \ 300│ * │ \ 180│ * │ \__ 160│ * │ \__ 150│ * └────────────────────── K 1 2 3 4 5
The point where the decrease starts becoming less significant is called the elbow.
That value can be considered as a candidate for K, but it should also be checked using domain knowledge and other validation methods.
11. Python Libraries
Common Python libraries for K-Means include:
- NumPy
- Pandas
- Matplotlib
- Scikit-learn
Import example:
import pandas as pd import matplotlib.pyplot as plt from sklearn.cluster import KMeans from sklearn.preprocessing import StandardScaler
12. Simple K-Means Example in Python
from sklearn.cluster import KMeans X = [ [1, 2], [1, 3], [2, 2], [8, 8], [9, 8], [8, 9] ] model = KMeans( n_clusters=2, random_state=42, n_init=10 ) model.fit(X) print("Cluster Labels:") print(model.labels_) print("Centroids:") print(model.cluster_centers_)
Explanation
n_clusters=2
means that we want two clusters.
model.fit(X)
trains the K-Means model.
model.labels_
returns the cluster assigned to each data point.
model.cluster_centers_
returns the centroid of each cluster.
13. K-Means with Pandas
Suppose we have customer data:
import pandas as pd from sklearn.cluster import KMeans data = { "Age": [20, 22, 25, 40, 42, 45], "Income": [20000, 25000, 27000, 70000, 75000, 80000] } df = pd.DataFrame(data) model = KMeans( n_clusters=2, random_state=42, n_init=10 ) df["Cluster"] = model.fit_predict(df[["Age", "Income"]]) print(df)
The output will contain a new column:
Age Income Cluster 20 20000 0 22 25000 0 25 27000 0 40 70000 1 42 75000 1 45 80000 1
The actual cluster numbers can be swapped because cluster labels are arbitrary.
14. Feature Scaling
Feature scaling is very important for K-Means.
Suppose we have:
Age → 20 to 60 Income → 20,000 to 100,000
Income has much larger numerical values than age.
Without scaling, the income feature may have a much stronger effect on distance calculations.
We can use StandardScaler.
from sklearn.preprocessing import StandardScaler scaler = StandardScaler() X_scaled = scaler.fit_transform(df[["Age", "Income"]])
Then:
model = KMeans( n_clusters=2, random_state=42, n_init=10 ) df["Cluster"] = model.fit_predict(X_scaled)
15. Complete K-Means Example
import pandas as pd import matplotlib.pyplot as plt from sklearn.cluster import KMeans from sklearn.preprocessing import StandardScaler # Create dataset data = { "Age": [20, 22, 25, 40, 42, 45], "Income": [20000, 25000, 27000, 70000, 75000, 80000] } df = pd.DataFrame(data) # Select features X = df[["Age", "Income"]] # Scale features scaler = StandardScaler() X_scaled = scaler.fit_transform(X) # Create K-Means model model = KMeans( n_clusters=2, random_state=42, n_init=10 ) # Predict clusters df["Cluster"] = model.fit_predict(X_scaled) print(df) # Display centroids in scaled space print("Centroids:") print(model.cluster_centers_)
16. Visualizing K-Means
We can visualize clusters using a scatter plot.
import matplotlib.pyplot as plt plt.scatter( df["Age"], df["Income"], c=df["Cluster"] ) plt.xlabel("Age") plt.ylabel("Income") plt.title("Customer Clusters") plt.show()
This allows us to see how customers are divided into different groups.
17. Elbow Method in Python
import matplotlib.pyplot as plt from sklearn.cluster import KMeans wcss = [] for k in range(1, 11): model = KMeans( n_clusters=k, random_state=42, n_init=10 ) model.fit(X_scaled) wcss.append(model.inertia_) plt.plot(range(1, 11), wcss, marker="o") plt.xlabel("Number of Clusters (K)") plt.ylabel("WCSS") plt.title("Elbow Method") plt.show()
Important
model.inertia_
returns the within-cluster sum of squared distances, commonly called inertia in scikit-learn.
18. K-Means Evaluation
Because K-Means is unsupervised, we normally do not have true target labels to calculate ordinary classification accuracy.
Some useful clustering evaluation methods are:
18.1 Inertia
Measures the sum of squared distances from points to their cluster centroids.
Lower values indicate more compact clusters, but inertia usually decreases as K increases, so it should not be used alone.
18.2 Silhouette Score
The Silhouette Score measures how well-separated and internally cohesive the clusters are.
Its value ranges approximately from:
-1 to +1
Generally:
- Higher positive values → better-separated clusters
- Around 0 → overlapping clusters
- Negative values → some points may be assigned to unsuitable clusters
Python:
from sklearn.metrics import silhouette_score score = silhouette_score(X_scaled, df["Cluster"]) print("Silhouette Score:", score)
19. K-Means Parameters
Important parameters in scikit-learn include:
| Parameter | Meaning |
|---|---|
n_clusters | Number of clusters |
init | Method for initializing centroids |
n_init | Number of centroid initializations tried |
max_iter | Maximum iterations for one run |
random_state | Controls reproducibility |
algorithm | Algorithm used for distance computations |
Example:
model = KMeans( n_clusters=3, init="k-means++", n_init=10, max_iter=300, random_state=42 )
20. Advantages of K-Means
- Easy to understand.
- Simple to implement.
- Relatively fast for many datasets.
- Works well when clusters are compact and reasonably separated.
- Useful for customer segmentation.
- Can process large datasets efficiently.
- Easy to visualize when there are two or three features.
21. Limitations of K-Means
- The value of K usually needs to be chosen beforehand.
- Results can depend on the initial centroid positions.
- Sensitive to outliers.
- Sensitive to feature scaling.
- May perform poorly with non-spherical or irregularly shaped clusters.
- Different runs can produce different results if initialization is not controlled.
- It is not always suitable when clusters have very different sizes or densities.
- Cluster labels do not have an inherent meaning.
22. Applications of K-Means
K-Means is widely used in different fields.
1. Customer Segmentation
Groups customers according to:
- Age
- Income
- Spending
- Purchase behavior
2. Marketing
Companies can identify groups of customers with similar characteristics.
3. Image Compression
Similar colors can be grouped together to reduce the number of colors in an image.
4. Document Clustering
Documents can be grouped according to their content.
5. Recommendation Systems
Users with similar behavior can be grouped.
6. Anomaly Detection
Very distant points may help identify unusual observations, although specialized anomaly-detection methods may be more appropriate.
7. Geographic Analysis
Locations can be grouped based on geographic or demographic characteristics.
23. Real-Life Example: Customer Segmentation
Suppose an online store has:
Customer Age Annual Income Spending Score
The company chooses:
K = 3
K-Means may produce:
Cluster 1 → Low spending customers Cluster 2 → Medium spending customers Cluster 3 → High spending customers
The company can then analyze each cluster separately.
This is called customer segmentation.
24. K-Means vs KNN
K-Means and KNN have similar names but are very different algorithms.
| K-Means | KNN |
|---|---|
| Unsupervised learning | Supervised learning |
| Used for clustering | Used for classification/regression |
| Does not require target labels | Requires labeled training data |
| Creates clusters | Predicts based on neighbors |
| Uses centroids | Uses nearest data points |
| K = number of clusters | K = number of neighbors |
25. K-Means vs Decision Tree
| K-Means | Decision Tree |
|---|---|
| Unsupervised | Supervised |
| Clustering | Classification/Regression |
| No target variable required | Target variable required |
| Uses centroids | Uses decision rules |
| Groups similar data | Predicts outcomes |
26. K-Means vs Hierarchical Clustering
| K-Means | Hierarchical Clustering |
|---|---|
| Requires K | Does not require K in the same way |
| Uses centroids | Builds a hierarchy of clusters |
| Usually faster for large datasets | Can be computationally expensive |
| Produces flat clusters | Produces a hierarchy/dendrogram |
| Easy to implement | Useful for exploring nested relationships |
27. Important Points
Remember these key concepts:
K-Means ↓ Unsupervised Learning ↓ Clustering ↓ Choose K ↓ Initialize Centroids ↓ Assign Points ↓ Calculate New Centroids ↓ Repeat ↓ Final Clusters
Key points:
- K-Means is an unsupervised learning algorithm.
- It divides data into K clusters.
- A centroid represents the center of a cluster.
- Distance is used to assign points to clusters.
- Euclidean distance is commonly used.
- Feature scaling can be important.
- The Elbow Method can help select K.
- Silhouette Score can help evaluate cluster separation.
- K-Means is useful for customer segmentation and pattern discovery.
- Cluster numbers themselves have no inherent meaning.
28. K-Means Workflow
Raw Data ↓ Data Preprocessing ↓ Select Features ↓ Scale Features ↓ Choose K ↓ Train K-Means ↓ Assign Data to Clusters ↓ Evaluate Clusters ↓ Visualize Results ↓ Interpret Clusters
29. When Should You Use K-Means?
K-Means is useful when:
- You do not have target labels.
- You want to discover groups in your data.
- Data can reasonably be represented using numerical features.
- You expect relatively compact and separated clusters.
- You need a simple and fast clustering method.
30. Conclusion
K-Means Clustering is a popular unsupervised machine learning algorithm used to divide data into groups of similar observations.
It works by:
- Choosing the number of clusters.
- Selecting initial centroids.
- Assigning points to the nearest centroid.
- Recalculating centroids.
- Repeating the process until the solution stabilizes.
It is commonly used for customer segmentation, marketing, image compression, document clustering, and pattern discovery.
Understanding K-Means is an important step toward learning more advanced unsupervised machine learning techniques.