ºÝºÝߣ

ºÝºÝߣShare a Scribd company logo
1
DTS304TC: Machine Learning
Lecture 8: Gaussian Mixture Model (GMM)
Dr Kang Dang
D-5032, Taicang Campus
Kang.Dang@xjtlu.edu.cn
Tel: 88973341
2
Acknowledges
This set of lecture notes has been adapted from
materials originally provided by Dr. Gan Hong Seng and
Christopher M. Bishop's lecture notes.
3
Course Outline
? What it is GMM?
? The concept of Mixture of Gaussians
? EM algorithm & Latent Variables l,
4
What is Gaussian Mixture Model?
? Probabilistic Model used for clustering and classification tasks.
? Assumption: data is generated by a mixture of several Gaussian
distributions, each with its own mean and variance.
? Application: by fitting a GMM to the data:
? Identify underlying clusters.
? Make predictions on new data points through probabilistic
assignments to each cluster..
? What is Gaussian Mixture Model
5
Example of Gaussian Distribution
X-Axis: Data Values
Y-Axis: Frequency or Probability of Occurrence
? Bell-Shaped Curve: illustrates that most data is clustered around the mean.
? Mean is depicted by the vertical line at the center.
? Standard Deviation measures the spread of the data
6
Example of Gaussian Distribution
7
Multivariate Gaussian Distribution
Likelihood Function
? Data set
? The probability of observing x given the Gaussian distribution:
Assume observed data points generated independently
? This probability is a function of the parameters this is known as the
likelihood function
Maximum Likelihood
? Obtaining the parameters by the given dataset, and maximizing the
likelihood function
? Equivalently maximize the log likelihood
Maximum Likelihood Solution
? Maximizing w.r.t. the mean gives the sample mean
? Maximizing w.r.t covariance gives the sample covariance
11
Mixture Models
? So estimating parameters for a single Gaussian is simple.
? How about modelling non-Gaussian data?
? Mixture models can be powerful to handle many non-gaussian data
distributions!
12
Mixture Model
Mixture Models are a collection of the weighted sum of a number of
probability density functions (PDFs) where the weights are determined by a
distribution
13
Example of Mixture Model
14
Hard Assignments (K-Means Clustering)
? Exclusive Assignment: each data point is assigned to a single
cluster.
? Cluster Membership: data points belong to one, and only
one, cluster.
15
Soft Assignments (GMM)
? Probabilistic Assignment: Assigns a probability for each data point
indicating its likelihood of belonging to each Gaussian distribution in
the mixture.
? Partial Membership: A single data point can have partial membership
in multiple Gaussian distributions.
16
Q&A
? When to use hard assignment and when to use soft assignment?
17
Hard vs Soft Assignemnts
? When to Use Hard Assignments
? Ideal for data with clearly separable, distinct clusters.
? Most effective when there is minimal overlap between clusters.
? When to Use Soft Assignments
? Suitable for data that is not easily separable into distinct clusters.
? Ideal for handling data with significant overlap between clusters.
18
Hard Assignments vs Soft Assignments
19
Mixture of Gaussian in 1D
20
Mixture of Gaussian in 2D
? Model Assumption: Data points are generated by a combination of several 2D Gaussian distributions.
? Distinct Parameters: Each distribution has its own mean (center point) and covariance matrix (shape and
orientation).
21
Parameters of PDF
22
What is inside GMM?
23
Gaussian Mixture Model as PDF
Q&A:
? How to prove a function is a PDF?
24
Gaussian Mixture Model as PDF
25
Gaussian Mixture Model as PDF
Gaussian Mixtures
? Linear super-position of Gaussians
? Normalization and positivity require
? Can interpret the mixing coefficients as prior probabilities
Sampling from the Gaussian Mixture
? To generate a data point:
? first pick one of the components with probability
? then draw a sample from that component
? Repeat these two steps for each new data point
Fitting the Gaussian Mixture
? We wish to invert this process ¨C given the data set, find the
corresponding parameters:
? mixing coefficients
? means
? covariances
? If we knew which component generated each data point, the
maximum likelihood solution would involve fitting each component to
the corresponding cluster
? Problem: the data set is unlabelled
? We shall refer to the labels as latent (= hidden) variables
Synthetic Data Set Without Labels
Posterior Probabilities
? We can think of the mixing coefficients as prior probabilities for the
components
? For a given value of we can evaluate the corresponding posterior
probabilities, called responsibilities
? These are given from Bayes¡¯ theorem by
Posterior Probabilities (colour coded)
Posterior Probability Map
Maximum Likelihood for the GMM
? The log likelihood function takes the form
? Note: sum over components appears inside the log
? There is no closed form solution for maximum likelihood
Problems and Solutions
? How to maximize the log likelihood
? solved by expectation-maximization (EM) algorithm
? This is the topic of our lecture
? How to avoid singularities in the likelihood function
? solved by a Bayesian treatment
? How to choose number K of components
? also solved by a Bayesian treatment
EM Algorithm ¨C Informal Derivation
? Let us proceed by simply differentiating the log likelihood
? Setting derivative with respect to equal to zero gives
giving
which is simply the weighted mean of the data
EM Algorithm ¨C Informal Derivation
? Similarly for the covariances
? For mixing coefficients use a Lagrange multiplier to give
37
EM Algorithm for GMM Estimation
38
EM Algorithm for GMM Estimation
39
EM Algorithm for GMM Estimation -
Summary
Evaluate the log likelihood
EM Algorithm ¨C Informal Derivation
? An iterative scheme for solving them:
? Make initial guesses for the parameters
? Alternate between the following two stages:
1. E-step: evaluate responsibilities
2. M-step: update parameters using ML results
Christopher M. Bishop
GMM Clustering Presentation ºÝºÝߣs for Machine Learning Course
GMM Clustering Presentation ºÝºÝߣs for Machine Learning Course
GMM Clustering Presentation ºÝºÝߣs for Machine Learning Course
GMM Clustering Presentation ºÝºÝߣs for Machine Learning Course
GMM Clustering Presentation ºÝºÝߣs for Machine Learning Course
47
GMM and K-Means Differences
K-means Clustering
? Assumption: Spherical clusters with equal probability.
? Cluster Assignment: Hard assignment (points belong to one cluster).
? Cluster Shape: Only identifies circular clusters.
? Algorithm: Minimizes within-cluster variance.
? Outlier Sensitivity: High, due to mean calculation.
Gaussian Mixture Models (GMM)
? Assumption: Data from multiple Gaussian distributions.
? Cluster Assignment: Soft assignment (probabilistic cluster
membership).
? Cluster Shape: Identifies elliptical clusters.
? Algorithm: Maximizes likelihood using expectation-maximization.
? Outlier Sensitivity: Lower, due to probabilistic framework.
48
GMM and K-Means Differences
Flexibility in Cluster Shapes: GMM can model elliptical and varying size clusters, not
just spherical.
Soft Clustering and Uncertainty: Provides membership probabilities, offering a
nuanced understanding of cluster belonging.
Density Estimation: GMM estimates the density distribution of each cluster, not just
central tendency.
Model Complexity: GMM captures complex cluster structures but requires more data
and computational power.
49
GMM and K-Means Differences
Use K-means When:
? You need a fast, simple, and interpretable model.
? Your data is expected to form spherical clusters.
? Computational resources are limited.
Use GMM When:
? You suspect clusters are non-spherical or have different sizes.
? You need a measure of uncertainty in cluster assignments.
? You have enough data to estimate the additional parameters reliably.
Takeaway:
? K-means is efficient for well-separated, spherical clusters.
? GMM is more flexible, capturing complex cluster shapes and providing
probabilistic cluster assignments.

More Related Content

Similar to GMM Clustering Presentation ºÝºÝߣs for Machine Learning Course (20)

DS9 - Clustering.pptx
DS9 - Clustering.pptxDS9 - Clustering.pptx
DS9 - Clustering.pptx
JK970901
?
How Machine Learning Helps Organizations to Work More Efficiently?
How Machine Learning Helps Organizations to Work More Efficiently?How Machine Learning Helps Organizations to Work More Efficiently?
How Machine Learning Helps Organizations to Work More Efficiently?
Tuan Yang
?
clustering in DataMining and differences in models/ clustering in data mining
clustering in DataMining and differences in models/ clustering in data miningclustering in DataMining and differences in models/ clustering in data mining
clustering in DataMining and differences in models/ clustering in data mining
RevathiSundar4
?
2002_Spring_CS525_Lggggggfdtfffdfgecture_2.ppt
2002_Spring_CS525_Lggggggfdtfffdfgecture_2.ppt2002_Spring_CS525_Lggggggfdtfffdfgecture_2.ppt
2002_Spring_CS525_Lggggggfdtfffdfgecture_2.ppt
fetnbadani
?
Lecture 18: Gaussian Mixture Models and Expectation Maximization
Lecture 18: Gaussian Mixture Models and Expectation MaximizationLecture 18: Gaussian Mixture Models and Expectation Maximization
Lecture 18: Gaussian Mixture Models and Expectation Maximization
butest
?
Algorithm explanations
Algorithm explanationsAlgorithm explanations
Algorithm explanations
nikita kapil
?
clustering using different methods in .pdf
clustering using different methods in .pdfclustering using different methods in .pdf
clustering using different methods in .pdf
officialnovice7
?
Mean shift and Hierarchical clustering
Mean shift and Hierarchical clustering Mean shift and Hierarchical clustering
Mean shift and Hierarchical clustering
Yan Xu
?
K MEANS CLUSTERING - UNSUPERVISED LEARNING
K MEANS CLUSTERING - UNSUPERVISED LEARNINGK MEANS CLUSTERING - UNSUPERVISED LEARNING
K MEANS CLUSTERING - UNSUPERVISED LEARNING
PalanivelG6
?
A popular clustering algorithm is known as K-means, which will follow an iter...
A popular clustering algorithm is known as K-means, which will follow an iter...A popular clustering algorithm is known as K-means, which will follow an iter...
A popular clustering algorithm is known as K-means, which will follow an iter...
KranthiKiran615171
?
Part 2: Unsupervised Learning Machine Learning Techniques
Part 2: Unsupervised Learning Machine Learning Techniques Part 2: Unsupervised Learning Machine Learning Techniques
Part 2: Unsupervised Learning Machine Learning Techniques
butest
?
GBM package in r
GBM package in rGBM package in r
GBM package in r
mark_landry
?
Optimization as a model for few shot learning
Optimization as a model for few shot learningOptimization as a model for few shot learning
Optimization as a model for few shot learning
Katy Lee
?
Advanced database and data mining & clustering concepts
Advanced database and data mining & clustering conceptsAdvanced database and data mining & clustering concepts
Advanced database and data mining & clustering concepts
NithyananthSengottai
?
UNIT_V_Cluster Analysis.pptx
UNIT_V_Cluster Analysis.pptxUNIT_V_Cluster Analysis.pptx
UNIT_V_Cluster Analysis.pptx
sandeepsandy494692
?
Data mining techniques unit v
Data mining techniques unit vData mining techniques unit v
Data mining techniques unit v
malathieswaran29
?
3b318431-df9f-4a2c-9909-61ecb6af8444.pptx
3b318431-df9f-4a2c-9909-61ecb6af8444.pptx3b318431-df9f-4a2c-9909-61ecb6af8444.pptx
3b318431-df9f-4a2c-9909-61ecb6af8444.pptx
NANDHINIS900805
?
CSA 3702 machine learning module 3
CSA 3702 machine learning module 3CSA 3702 machine learning module 3
CSA 3702 machine learning module 3
Nandhini S
?
Big data Clustering Algorithms And Strategies
Big data Clustering Algorithms And StrategiesBig data Clustering Algorithms And Strategies
Big data Clustering Algorithms And Strategies
Farzad Nozarian
?
machine learning - Clustering in R
machine learning - Clustering in Rmachine learning - Clustering in R
machine learning - Clustering in R
Sudhakar Chavan
?
DS9 - Clustering.pptx
DS9 - Clustering.pptxDS9 - Clustering.pptx
DS9 - Clustering.pptx
JK970901
?
How Machine Learning Helps Organizations to Work More Efficiently?
How Machine Learning Helps Organizations to Work More Efficiently?How Machine Learning Helps Organizations to Work More Efficiently?
How Machine Learning Helps Organizations to Work More Efficiently?
Tuan Yang
?
clustering in DataMining and differences in models/ clustering in data mining
clustering in DataMining and differences in models/ clustering in data miningclustering in DataMining and differences in models/ clustering in data mining
clustering in DataMining and differences in models/ clustering in data mining
RevathiSundar4
?
2002_Spring_CS525_Lggggggfdtfffdfgecture_2.ppt
2002_Spring_CS525_Lggggggfdtfffdfgecture_2.ppt2002_Spring_CS525_Lggggggfdtfffdfgecture_2.ppt
2002_Spring_CS525_Lggggggfdtfffdfgecture_2.ppt
fetnbadani
?
Lecture 18: Gaussian Mixture Models and Expectation Maximization
Lecture 18: Gaussian Mixture Models and Expectation MaximizationLecture 18: Gaussian Mixture Models and Expectation Maximization
Lecture 18: Gaussian Mixture Models and Expectation Maximization
butest
?
Algorithm explanations
Algorithm explanationsAlgorithm explanations
Algorithm explanations
nikita kapil
?
clustering using different methods in .pdf
clustering using different methods in .pdfclustering using different methods in .pdf
clustering using different methods in .pdf
officialnovice7
?
Mean shift and Hierarchical clustering
Mean shift and Hierarchical clustering Mean shift and Hierarchical clustering
Mean shift and Hierarchical clustering
Yan Xu
?
K MEANS CLUSTERING - UNSUPERVISED LEARNING
K MEANS CLUSTERING - UNSUPERVISED LEARNINGK MEANS CLUSTERING - UNSUPERVISED LEARNING
K MEANS CLUSTERING - UNSUPERVISED LEARNING
PalanivelG6
?
A popular clustering algorithm is known as K-means, which will follow an iter...
A popular clustering algorithm is known as K-means, which will follow an iter...A popular clustering algorithm is known as K-means, which will follow an iter...
A popular clustering algorithm is known as K-means, which will follow an iter...
KranthiKiran615171
?
Part 2: Unsupervised Learning Machine Learning Techniques
Part 2: Unsupervised Learning Machine Learning Techniques Part 2: Unsupervised Learning Machine Learning Techniques
Part 2: Unsupervised Learning Machine Learning Techniques
butest
?
Optimization as a model for few shot learning
Optimization as a model for few shot learningOptimization as a model for few shot learning
Optimization as a model for few shot learning
Katy Lee
?
Advanced database and data mining & clustering concepts
Advanced database and data mining & clustering conceptsAdvanced database and data mining & clustering concepts
Advanced database and data mining & clustering concepts
NithyananthSengottai
?
Data mining techniques unit v
Data mining techniques unit vData mining techniques unit v
Data mining techniques unit v
malathieswaran29
?
3b318431-df9f-4a2c-9909-61ecb6af8444.pptx
3b318431-df9f-4a2c-9909-61ecb6af8444.pptx3b318431-df9f-4a2c-9909-61ecb6af8444.pptx
3b318431-df9f-4a2c-9909-61ecb6af8444.pptx
NANDHINIS900805
?
CSA 3702 machine learning module 3
CSA 3702 machine learning module 3CSA 3702 machine learning module 3
CSA 3702 machine learning module 3
Nandhini S
?
Big data Clustering Algorithms And Strategies
Big data Clustering Algorithms And StrategiesBig data Clustering Algorithms And Strategies
Big data Clustering Algorithms And Strategies
Farzad Nozarian
?
machine learning - Clustering in R
machine learning - Clustering in Rmachine learning - Clustering in R
machine learning - Clustering in R
Sudhakar Chavan
?

More from ssuserfece35 (7)

Adaboost Classifier for Machine Learning Course
Adaboost Classifier for Machine Learning CourseAdaboost Classifier for Machine Learning Course
Adaboost Classifier for Machine Learning Course
ssuserfece35
?
Build_Machine_Learning_System for Machine Learning Course
Build_Machine_Learning_System for Machine Learning CourseBuild_Machine_Learning_System for Machine Learning Course
Build_Machine_Learning_System for Machine Learning Course
ssuserfece35
?
K-Means Clustering Presentation ºÝºÝߣs for Machine Learning Course
K-Means Clustering Presentation ºÝºÝߣs for Machine Learning CourseK-Means Clustering Presentation ºÝºÝߣs for Machine Learning Course
K-Means Clustering Presentation ºÝºÝߣs for Machine Learning Course
ssuserfece35
?
Introduction to Machine Learning Lectures
Introduction to Machine Learning LecturesIntroduction to Machine Learning Lectures
Introduction to Machine Learning Lectures
ssuserfece35
?
hyperparamater search netowrk technnique
hyperparamater search netowrk technniquehyperparamater search netowrk technnique
hyperparamater search netowrk technnique
ssuserfece35
?
5 ÍÆÏë¿Æ¼¼Infervision_Intro_NV_English Intro Material
5 ÍÆÏë¿Æ¼¼Infervision_Intro_NV_English Intro Material5 ÍÆÏë¿Æ¼¼Infervision_Intro_NV_English Intro Material
5 ÍÆÏë¿Æ¼¼Infervision_Intro_NV_English Intro Material
ssuserfece35
?
Transformer in Medical Imaging A brief review
Transformer in Medical Imaging A brief reviewTransformer in Medical Imaging A brief review
Transformer in Medical Imaging A brief review
ssuserfece35
?
Adaboost Classifier for Machine Learning Course
Adaboost Classifier for Machine Learning CourseAdaboost Classifier for Machine Learning Course
Adaboost Classifier for Machine Learning Course
ssuserfece35
?
Build_Machine_Learning_System for Machine Learning Course
Build_Machine_Learning_System for Machine Learning CourseBuild_Machine_Learning_System for Machine Learning Course
Build_Machine_Learning_System for Machine Learning Course
ssuserfece35
?
K-Means Clustering Presentation ºÝºÝߣs for Machine Learning Course
K-Means Clustering Presentation ºÝºÝߣs for Machine Learning CourseK-Means Clustering Presentation ºÝºÝߣs for Machine Learning Course
K-Means Clustering Presentation ºÝºÝߣs for Machine Learning Course
ssuserfece35
?
Introduction to Machine Learning Lectures
Introduction to Machine Learning LecturesIntroduction to Machine Learning Lectures
Introduction to Machine Learning Lectures
ssuserfece35
?
hyperparamater search netowrk technnique
hyperparamater search netowrk technniquehyperparamater search netowrk technnique
hyperparamater search netowrk technnique
ssuserfece35
?
5 ÍÆÏë¿Æ¼¼Infervision_Intro_NV_English Intro Material
5 ÍÆÏë¿Æ¼¼Infervision_Intro_NV_English Intro Material5 ÍÆÏë¿Æ¼¼Infervision_Intro_NV_English Intro Material
5 ÍÆÏë¿Æ¼¼Infervision_Intro_NV_English Intro Material
ssuserfece35
?
Transformer in Medical Imaging A brief review
Transformer in Medical Imaging A brief reviewTransformer in Medical Imaging A brief review
Transformer in Medical Imaging A brief review
ssuserfece35
?

Recently uploaded (20)

Viceroys of India & Their Tenure ¨C Key Events During British Rule
Viceroys of India & Their Tenure ¨C Key Events During British RuleViceroys of India & Their Tenure ¨C Key Events During British Rule
Viceroys of India & Their Tenure ¨C Key Events During British Rule
DeeptiKumari61
?
Unit 3: Combustion in Spark Ignition Engines
Unit 3: Combustion in Spark Ignition EnginesUnit 3: Combustion in Spark Ignition Engines
Unit 3: Combustion in Spark Ignition Engines
NileshKumbhar21
?
Unit1 Inroduction to Internal Combustion Engines
Unit1  Inroduction to Internal Combustion EnginesUnit1  Inroduction to Internal Combustion Engines
Unit1 Inroduction to Internal Combustion Engines
NileshKumbhar21
?
Berry_Kanisha_BAS_PB1_202503 (2) (2).pdf
Berry_Kanisha_BAS_PB1_202503 (2) (2).pdfBerry_Kanisha_BAS_PB1_202503 (2) (2).pdf
Berry_Kanisha_BAS_PB1_202503 (2) (2).pdf
KanishaBerry
?
Yale VMOC Special Report - Measles Outbreak Southwest US 3-30-2025 FINAL v2...
Yale VMOC Special Report - Measles Outbreak  Southwest US 3-30-2025  FINAL v2...Yale VMOC Special Report - Measles Outbreak  Southwest US 3-30-2025  FINAL v2...
Yale VMOC Special Report - Measles Outbreak Southwest US 3-30-2025 FINAL v2...
Yale School of Public Health - The Virtual Medical Operations Center (VMOC)
?
Anti-Fungal Agents.pptx Medicinal Chemistry III B. Pharm Sem VI
Anti-Fungal Agents.pptx Medicinal Chemistry III B. Pharm Sem VIAnti-Fungal Agents.pptx Medicinal Chemistry III B. Pharm Sem VI
Anti-Fungal Agents.pptx Medicinal Chemistry III B. Pharm Sem VI
Samruddhi Khonde
?
Knownsense 2025 prelims- U-25 General Quiz.pdf
Knownsense 2025 prelims- U-25 General Quiz.pdfKnownsense 2025 prelims- U-25 General Quiz.pdf
Knownsense 2025 prelims- U-25 General Quiz.pdf
Pragya - UEM Kolkata Quiz Club
?
Unit No 4- Chemotherapy of Malignancy.pptx
Unit No  4- Chemotherapy of Malignancy.pptxUnit No  4- Chemotherapy of Malignancy.pptx
Unit No 4- Chemotherapy of Malignancy.pptx
Ashish Umale
?
Design approaches and ethical challenges in Artificial Intelligence tools for...
Design approaches and ethical challenges in Artificial Intelligence tools for...Design approaches and ethical challenges in Artificial Intelligence tools for...
Design approaches and ethical challenges in Artificial Intelligence tools for...
Yannis
?
MIPLM subject matter expert Dr Alihan Kaya
MIPLM subject matter expert Dr Alihan KayaMIPLM subject matter expert Dr Alihan Kaya
MIPLM subject matter expert Dr Alihan Kaya
MIPLM
?
The basics of sentences session 9pptx.pptx
The basics of sentences session 9pptx.pptxThe basics of sentences session 9pptx.pptx
The basics of sentences session 9pptx.pptx
heathfieldcps1
?
NURSING PROCESS AND ITS STEPS .pptx
NURSING PROCESS AND ITS STEPS                 .pptxNURSING PROCESS AND ITS STEPS                 .pptx
NURSING PROCESS AND ITS STEPS .pptx
PoojaSen20
?
EDL 290F Week 5 - Facing Headwinds and Hairpin Turns (2025).pdf
EDL 290F Week 5  - Facing Headwinds and Hairpin Turns (2025).pdfEDL 290F Week 5  - Facing Headwinds and Hairpin Turns (2025).pdf
EDL 290F Week 5 - Facing Headwinds and Hairpin Turns (2025).pdf
Liz Walsh-Trevino
?
Week 6 - EDL 290F - No Drop Ride (2025).pdf
Week 6 - EDL 290F - No Drop Ride (2025).pdfWeek 6 - EDL 290F - No Drop Ride (2025).pdf
Week 6 - EDL 290F - No Drop Ride (2025).pdf
Liz Walsh-Trevino
?
Chapter 6. Business and Corporate Strategy Formulation.pdf
Chapter 6. Business and Corporate Strategy Formulation.pdfChapter 6. Business and Corporate Strategy Formulation.pdf
Chapter 6. Business and Corporate Strategy Formulation.pdf
Rommel Regala
?
ANORECTAL MALFORMATIONS: NURSING MANAGEMENT PPT.pptx
ANORECTAL MALFORMATIONS: NURSING MANAGEMENT PPT.pptxANORECTAL MALFORMATIONS: NURSING MANAGEMENT PPT.pptx
ANORECTAL MALFORMATIONS: NURSING MANAGEMENT PPT.pptx
PRADEEP ABOTHU
?
EDL 290F Week 4 - Group Ride (2025).pdf
EDL 290F Week 4  - Group Ride (2025).pdfEDL 290F Week 4  - Group Ride (2025).pdf
EDL 290F Week 4 - Group Ride (2025).pdf
Liz Walsh-Trevino
?
U.S. Department of Education certification
U.S. Department of Education certificationU.S. Department of Education certification
U.S. Department of Education certification
Mebane Rash
?
How to Install Odoo 18 with Pycharm - Odoo 18 ºÝºÝߣs
How to Install Odoo 18 with Pycharm - Odoo 18 ºÝºÝߣsHow to Install Odoo 18 with Pycharm - Odoo 18 ºÝºÝߣs
How to Install Odoo 18 with Pycharm - Odoo 18 ºÝºÝߣs
Celine George
?
ANTIVIRAL agent by Mrs. Manjushri Dabhade
ANTIVIRAL agent by Mrs. Manjushri DabhadeANTIVIRAL agent by Mrs. Manjushri Dabhade
ANTIVIRAL agent by Mrs. Manjushri Dabhade
Dabhade madam Dabhade
?
Viceroys of India & Their Tenure ¨C Key Events During British Rule
Viceroys of India & Their Tenure ¨C Key Events During British RuleViceroys of India & Their Tenure ¨C Key Events During British Rule
Viceroys of India & Their Tenure ¨C Key Events During British Rule
DeeptiKumari61
?
Unit 3: Combustion in Spark Ignition Engines
Unit 3: Combustion in Spark Ignition EnginesUnit 3: Combustion in Spark Ignition Engines
Unit 3: Combustion in Spark Ignition Engines
NileshKumbhar21
?
Unit1 Inroduction to Internal Combustion Engines
Unit1  Inroduction to Internal Combustion EnginesUnit1  Inroduction to Internal Combustion Engines
Unit1 Inroduction to Internal Combustion Engines
NileshKumbhar21
?
Berry_Kanisha_BAS_PB1_202503 (2) (2).pdf
Berry_Kanisha_BAS_PB1_202503 (2) (2).pdfBerry_Kanisha_BAS_PB1_202503 (2) (2).pdf
Berry_Kanisha_BAS_PB1_202503 (2) (2).pdf
KanishaBerry
?
Anti-Fungal Agents.pptx Medicinal Chemistry III B. Pharm Sem VI
Anti-Fungal Agents.pptx Medicinal Chemistry III B. Pharm Sem VIAnti-Fungal Agents.pptx Medicinal Chemistry III B. Pharm Sem VI
Anti-Fungal Agents.pptx Medicinal Chemistry III B. Pharm Sem VI
Samruddhi Khonde
?
Unit No 4- Chemotherapy of Malignancy.pptx
Unit No  4- Chemotherapy of Malignancy.pptxUnit No  4- Chemotherapy of Malignancy.pptx
Unit No 4- Chemotherapy of Malignancy.pptx
Ashish Umale
?
Design approaches and ethical challenges in Artificial Intelligence tools for...
Design approaches and ethical challenges in Artificial Intelligence tools for...Design approaches and ethical challenges in Artificial Intelligence tools for...
Design approaches and ethical challenges in Artificial Intelligence tools for...
Yannis
?
MIPLM subject matter expert Dr Alihan Kaya
MIPLM subject matter expert Dr Alihan KayaMIPLM subject matter expert Dr Alihan Kaya
MIPLM subject matter expert Dr Alihan Kaya
MIPLM
?
The basics of sentences session 9pptx.pptx
The basics of sentences session 9pptx.pptxThe basics of sentences session 9pptx.pptx
The basics of sentences session 9pptx.pptx
heathfieldcps1
?
NURSING PROCESS AND ITS STEPS .pptx
NURSING PROCESS AND ITS STEPS                 .pptxNURSING PROCESS AND ITS STEPS                 .pptx
NURSING PROCESS AND ITS STEPS .pptx
PoojaSen20
?
EDL 290F Week 5 - Facing Headwinds and Hairpin Turns (2025).pdf
EDL 290F Week 5  - Facing Headwinds and Hairpin Turns (2025).pdfEDL 290F Week 5  - Facing Headwinds and Hairpin Turns (2025).pdf
EDL 290F Week 5 - Facing Headwinds and Hairpin Turns (2025).pdf
Liz Walsh-Trevino
?
Week 6 - EDL 290F - No Drop Ride (2025).pdf
Week 6 - EDL 290F - No Drop Ride (2025).pdfWeek 6 - EDL 290F - No Drop Ride (2025).pdf
Week 6 - EDL 290F - No Drop Ride (2025).pdf
Liz Walsh-Trevino
?
Chapter 6. Business and Corporate Strategy Formulation.pdf
Chapter 6. Business and Corporate Strategy Formulation.pdfChapter 6. Business and Corporate Strategy Formulation.pdf
Chapter 6. Business and Corporate Strategy Formulation.pdf
Rommel Regala
?
ANORECTAL MALFORMATIONS: NURSING MANAGEMENT PPT.pptx
ANORECTAL MALFORMATIONS: NURSING MANAGEMENT PPT.pptxANORECTAL MALFORMATIONS: NURSING MANAGEMENT PPT.pptx
ANORECTAL MALFORMATIONS: NURSING MANAGEMENT PPT.pptx
PRADEEP ABOTHU
?
EDL 290F Week 4 - Group Ride (2025).pdf
EDL 290F Week 4  - Group Ride (2025).pdfEDL 290F Week 4  - Group Ride (2025).pdf
EDL 290F Week 4 - Group Ride (2025).pdf
Liz Walsh-Trevino
?
U.S. Department of Education certification
U.S. Department of Education certificationU.S. Department of Education certification
U.S. Department of Education certification
Mebane Rash
?
How to Install Odoo 18 with Pycharm - Odoo 18 ºÝºÝߣs
How to Install Odoo 18 with Pycharm - Odoo 18 ºÝºÝߣsHow to Install Odoo 18 with Pycharm - Odoo 18 ºÝºÝߣs
How to Install Odoo 18 with Pycharm - Odoo 18 ºÝºÝߣs
Celine George
?
ANTIVIRAL agent by Mrs. Manjushri Dabhade
ANTIVIRAL agent by Mrs. Manjushri DabhadeANTIVIRAL agent by Mrs. Manjushri Dabhade
ANTIVIRAL agent by Mrs. Manjushri Dabhade
Dabhade madam Dabhade
?

GMM Clustering Presentation ºÝºÝߣs for Machine Learning Course

  • 1. 1 DTS304TC: Machine Learning Lecture 8: Gaussian Mixture Model (GMM) Dr Kang Dang D-5032, Taicang Campus Kang.Dang@xjtlu.edu.cn Tel: 88973341
  • 2. 2 Acknowledges This set of lecture notes has been adapted from materials originally provided by Dr. Gan Hong Seng and Christopher M. Bishop's lecture notes.
  • 3. 3 Course Outline ? What it is GMM? ? The concept of Mixture of Gaussians ? EM algorithm & Latent Variables l,
  • 4. 4 What is Gaussian Mixture Model? ? Probabilistic Model used for clustering and classification tasks. ? Assumption: data is generated by a mixture of several Gaussian distributions, each with its own mean and variance. ? Application: by fitting a GMM to the data: ? Identify underlying clusters. ? Make predictions on new data points through probabilistic assignments to each cluster.. ? What is Gaussian Mixture Model
  • 5. 5 Example of Gaussian Distribution X-Axis: Data Values Y-Axis: Frequency or Probability of Occurrence ? Bell-Shaped Curve: illustrates that most data is clustered around the mean. ? Mean is depicted by the vertical line at the center. ? Standard Deviation measures the spread of the data
  • 6. 6 Example of Gaussian Distribution
  • 8. Likelihood Function ? Data set ? The probability of observing x given the Gaussian distribution: Assume observed data points generated independently ? This probability is a function of the parameters this is known as the likelihood function
  • 9. Maximum Likelihood ? Obtaining the parameters by the given dataset, and maximizing the likelihood function ? Equivalently maximize the log likelihood
  • 10. Maximum Likelihood Solution ? Maximizing w.r.t. the mean gives the sample mean ? Maximizing w.r.t covariance gives the sample covariance
  • 11. 11 Mixture Models ? So estimating parameters for a single Gaussian is simple. ? How about modelling non-Gaussian data? ? Mixture models can be powerful to handle many non-gaussian data distributions!
  • 12. 12 Mixture Model Mixture Models are a collection of the weighted sum of a number of probability density functions (PDFs) where the weights are determined by a distribution
  • 14. 14 Hard Assignments (K-Means Clustering) ? Exclusive Assignment: each data point is assigned to a single cluster. ? Cluster Membership: data points belong to one, and only one, cluster.
  • 15. 15 Soft Assignments (GMM) ? Probabilistic Assignment: Assigns a probability for each data point indicating its likelihood of belonging to each Gaussian distribution in the mixture. ? Partial Membership: A single data point can have partial membership in multiple Gaussian distributions.
  • 16. 16 Q&A ? When to use hard assignment and when to use soft assignment?
  • 17. 17 Hard vs Soft Assignemnts ? When to Use Hard Assignments ? Ideal for data with clearly separable, distinct clusters. ? Most effective when there is minimal overlap between clusters. ? When to Use Soft Assignments ? Suitable for data that is not easily separable into distinct clusters. ? Ideal for handling data with significant overlap between clusters.
  • 18. 18 Hard Assignments vs Soft Assignments
  • 20. 20 Mixture of Gaussian in 2D ? Model Assumption: Data points are generated by a combination of several 2D Gaussian distributions. ? Distinct Parameters: Each distribution has its own mean (center point) and covariance matrix (shape and orientation).
  • 23. 23 Gaussian Mixture Model as PDF Q&A: ? How to prove a function is a PDF?
  • 26. Gaussian Mixtures ? Linear super-position of Gaussians ? Normalization and positivity require ? Can interpret the mixing coefficients as prior probabilities
  • 27. Sampling from the Gaussian Mixture ? To generate a data point: ? first pick one of the components with probability ? then draw a sample from that component ? Repeat these two steps for each new data point
  • 28. Fitting the Gaussian Mixture ? We wish to invert this process ¨C given the data set, find the corresponding parameters: ? mixing coefficients ? means ? covariances ? If we knew which component generated each data point, the maximum likelihood solution would involve fitting each component to the corresponding cluster ? Problem: the data set is unlabelled ? We shall refer to the labels as latent (= hidden) variables
  • 29. Synthetic Data Set Without Labels
  • 30. Posterior Probabilities ? We can think of the mixing coefficients as prior probabilities for the components ? For a given value of we can evaluate the corresponding posterior probabilities, called responsibilities ? These are given from Bayes¡¯ theorem by
  • 33. Maximum Likelihood for the GMM ? The log likelihood function takes the form ? Note: sum over components appears inside the log ? There is no closed form solution for maximum likelihood
  • 34. Problems and Solutions ? How to maximize the log likelihood ? solved by expectation-maximization (EM) algorithm ? This is the topic of our lecture ? How to avoid singularities in the likelihood function ? solved by a Bayesian treatment ? How to choose number K of components ? also solved by a Bayesian treatment
  • 35. EM Algorithm ¨C Informal Derivation ? Let us proceed by simply differentiating the log likelihood ? Setting derivative with respect to equal to zero gives giving which is simply the weighted mean of the data
  • 36. EM Algorithm ¨C Informal Derivation ? Similarly for the covariances ? For mixing coefficients use a Lagrange multiplier to give
  • 37. 37 EM Algorithm for GMM Estimation
  • 38. 38 EM Algorithm for GMM Estimation
  • 39. 39 EM Algorithm for GMM Estimation - Summary Evaluate the log likelihood
  • 40. EM Algorithm ¨C Informal Derivation ? An iterative scheme for solving them: ? Make initial guesses for the parameters ? Alternate between the following two stages: 1. E-step: evaluate responsibilities 2. M-step: update parameters using ML results
  • 47. 47 GMM and K-Means Differences K-means Clustering ? Assumption: Spherical clusters with equal probability. ? Cluster Assignment: Hard assignment (points belong to one cluster). ? Cluster Shape: Only identifies circular clusters. ? Algorithm: Minimizes within-cluster variance. ? Outlier Sensitivity: High, due to mean calculation. Gaussian Mixture Models (GMM) ? Assumption: Data from multiple Gaussian distributions. ? Cluster Assignment: Soft assignment (probabilistic cluster membership). ? Cluster Shape: Identifies elliptical clusters. ? Algorithm: Maximizes likelihood using expectation-maximization. ? Outlier Sensitivity: Lower, due to probabilistic framework.
  • 48. 48 GMM and K-Means Differences Flexibility in Cluster Shapes: GMM can model elliptical and varying size clusters, not just spherical. Soft Clustering and Uncertainty: Provides membership probabilities, offering a nuanced understanding of cluster belonging. Density Estimation: GMM estimates the density distribution of each cluster, not just central tendency. Model Complexity: GMM captures complex cluster structures but requires more data and computational power.
  • 49. 49 GMM and K-Means Differences Use K-means When: ? You need a fast, simple, and interpretable model. ? Your data is expected to form spherical clusters. ? Computational resources are limited. Use GMM When: ? You suspect clusters are non-spherical or have different sizes. ? You need a measure of uncertainty in cluster assignments. ? You have enough data to estimate the additional parameters reliably. Takeaway: ? K-means is efficient for well-separated, spherical clusters. ? GMM is more flexible, capturing complex cluster shapes and providing probabilistic cluster assignments.

Editor's Notes

  • #4: In the realm of statistical analysis, the Gaussian Mixture Model (GMM) is a versatile probabilistic tool that serves both for clustering and classification tasks. It operates under the assumption that the data points are produced by a blend of multiple Gaussian distributions, each characterized by distinct parameters¡ªmean and variance that define their centers and spreads, respectively. By applying a GMM to a dataset, we can uncover latent groupings inherent in the data, revealing the underlying structure. Furthermore, the model empowers us to make informed predictions about where new data points might belong within these clusters, not through rigid assignment but by calculating the likelihood of membership in each cluster, thereby yielding a more nuanced, probabilistic classification.
  • #47: K-means operates on the assumption that each cluster is spherical and all clusters are equally likely, assigning each data point to a single cluster in a 'hard' manner, meaning points are fully in one cluster or another. This algorithm seeks to make the variation within each cluster as small as possible, but it tends to be sensitive to outliers because it uses the mean of the points to determine cluster centers and can only identify circular-shaped clusters. On the other hand, GMM assumes that data points are drawn from several Gaussian distributions, which allows for 'soft' cluster assignment. This means that it assigns points to clusters based on the probability of membership, making it more flexible in accommodating elliptical cluster shapes. The GMM algorithm uses an expectation-maximization process to maximize the likelihood of the data points given the model, and it is generally less sensitive to outliers due to its probabilistic nature.
  • #48: In academic discourse, the Gaussian Mixture Model (GMM) is prized for its flexibility in capturing a wide variety of cluster shapes, including elliptical forms and clusters of different sizes, rather than being confined to identifying only spherical clusters as some other methods are. GMM extends beyond simple cluster assignment by providing membership probabilities for each data point, thereby offering a more sophisticated and nuanced view of how data points relate to potential clusters. This model excels in estimating the density distribution within each cluster, which provides a richer understanding than merely pinpointing the central tendency. However, the intricacy of GMM in modeling complex cluster configurations comes at a cost; it necessitates a larger dataset and more computational resources to perform effectively.
  • #49: Choose K-means if you're looking for a quick, straightforward method that's easy to explain and when you think your data naturally splits into neat, round groups. It's also a good pick when you don't have a lot of computing power. On the other hand, go for the Gaussian Mixture Model (GMM) when you have a hunch that your clusters aren't just simple spheres or when they come in different sizes. GMM is also helpful when you want to know how sure the model is about which group each piece of data belongs to, but remember, it needs a good amount of data to work properly. To sum it up, K-means is your go-to for quick and clean clustering of round groups, while GMM is the choice for more complex situations and gives you insights into the probability of each data point's membership in a cluster.