際際滷

際際滷Share a Scribd company logo

Lecture 3: Unsupervised Learning

Sang Jun Lee
Sang Jun Lee

- POSTECH EECE695J, "ル 蠍一 覦 豌螳螻旧 ", Lecture 3 - Contents: unsupervised learning, k-means algorithm, spectral clustering, anomaly detection

1 of 36
Downloaded 26 times
Unsupervised Learning (+ regularization) Sang Jun Lee Ph.D. candidate, POSTECH Email: lsj4u0208@postech.ac.kr EECE695J 蠍郁概豪J(ル蠍一覦鶮螳螻旧) LECTURE 3 (2017. 9. 14)
2 Lecture 2: supervised learning Machine learning : supervised learning, unsupervised learning, etc. Supervised learning: linear regression, logistic regression (classification), etc. 1-page Review Supervised learning Given a set of labeled example, 倹 = (ヰヰ＄, ＄) ＄=1 , learn a mapping : which minimizes L(鐃署 = , ) Unsupervised learning Given a set of unlabeled example, 倹 = (ヰヰ＄) ＄=1 , learn a meaningful representation of the data Linear regression 署 = =0 ヰヰ = 署署 署 署 = 1 2 裡=1 署 2 腫 署 ヰヰ () 署 = 1 1+署 署 0.5 = 1 署 < 0.5 = 0 署 = =1 () log 署( ) + (1 ) log(1 署( )) + 腫 署 ヰヰ Logistic regression
Overfitting? Training data 讌豺蟆(over) fit 朱 豢碁ゼ 讌 覈詩 覓語 朱朱 給一危一 讌豺蟆 ｸル 覲旧″ 覈(覿 curve)襦 誤 覦 Overfitting 譴願鍵 覦覯 Reduce the number of parameters (一危磯ゼ feature vector襯 螻殊朱 蟲燕朱 data dimension 豢) Regularization (cost function) 3 The problem of overfitting hypothesis function 蟆企朱 螳
Regularization Parameter 螳襯 讌 蠍磯ゼ 螳伎朱 overfitting 狩 覦覯 Housing price prediction 覓語 hypothesis function: 署 = 0 + 1 ヰ1 + 2 ヰ2 覈語 磯 under-fitting (high bias) overfitting (high variance) 覓語螳 覦 ! 4 The problem of overfitting
Regularization Linear regression cost function: 署 署 = 1 2 鐃 =1 署 2 parameter 署 蠍磯ゼ 譴企 regularization parameter Gradient descent update rule: 5 The problem of overfitting 署 署 = 1 2 鐃 =1 署 2 + 鐃 =1 2 0 0 腫 署 ヰ0 () 1 腫 腫 1 鐃 =1 署 ヰヰ 0 regularize 讌 Update parameter 蠍磯ゼ 螳る 螻!
Normal equation 覲 Linear regression normal equation : , where = , 署 = 0 , and = (1) () Regularization parameter襯 覃 谿瑚: + 腫殊 譟伎 ( > 0) 6 Regularization Positive semi-definiter (if , is not invertiable) Positive definite
Validation set 蟲! 覲旧° 覈語 蟲燕 validation 焔レ 蟯谿 署 = 0 + 1 ヰ1 + + ヰヰ 蠍一 regularization parameter ()襯 伎朱 蟆 譴! (螳 覓 覃 under-fitting 覓語螳 覦) 7 Bias vs. Variance 蠏碁 豢豌: https://www.coursera.org/learn/machine-learning/resources/LIZza
8 Clustering 觜訣 轟煙 一危磯れ 覓矩 螻襴讀 K-means algorithm Spectral clustering Anomaly detection Density estimation Unsupervised learning EECE695J (2017) Sang Jun Lee (POSTECH) K-means Cluster center襦覿一 蟇磯Μ襯 蠍一朱 clustering Spectral clustering 咋(, 乞): vertices and edges 朱 觜朱 蠏碁 襯 partitioning 蟆瑚
9 譯殊伎 螳 一危( 倹 ＄=1 )襯 螳 center ( = 1, , 情)襯 谿城 覓語 一危磯れ 螳 蠏碁９朱 partitioning Initialize ( = 1, , 情) Repeat until convergence! K-means algorithm EECE695J (2017) Sang Jun Lee (POSTECH) Assignment step Assign all data points to the cluster for which 2 is smallest Update step Compute new means for every cluster = 1 裡＄≠駒駒 谿場狩蟆 optimal cluster optimal mean碁.. mean 螻 optimal cluster襯 襾殊 谿剰 谿場 optimal cluster 襦 optimal mean 螻 Not optimal solution!
10 K-means algorithm EECE695J (2017) Sang Jun Lee (POSTECH) Initialize
11 K-means algorithm EECE695J (2017) Sang Jun Lee (POSTECH) Initialize Assignment step
12 K-means algorithm EECE695J (2017) Sang Jun Lee (POSTECH) Initialize Assignment step Update step
13 K-means algorithm EECE695J (2017) Sang Jun Lee (POSTECH) Initialize Assignment step Update step Assignment step
14 K-means algorithm EECE695J (2017) Sang Jun Lee (POSTECH) Initialize Assignment step Update step Assignment step Update step
15 K-means algorithm EECE695J (2017) Sang Jun Lee (POSTECH) Initialize Assignment step Update step Assignment step Update step Assignment step
16 K-means algorithm in TensorFlow EECE695J (2017) Sang Jun Lee (POSTECH) ipython 轟 蠏碁殊 蠏碁Μ蠍 覈轟 螳 normal distribution朱 sample data 蟲 vectors_set : 2000x2
17 K-means algorithm in TensorFlow EECE695J (2017) Sang Jun Lee (POSTECH)
18 K-means algorithm in TensorFlow EECE695J (2017) Sang Jun Lee (POSTECH) 2000螳 譴 k螳襯 襦 initial mean 螳朱 expanded_vectors: (tensor) 1x2000x2 expanded_centroids: (tensor) 4x1x2
19 K-means algorithm in TensorFlow EECE695J (2017) Sang Jun Lee (POSTECH) Assignment 螻殊 螳 れ願 蟆 殊 operation 企麹 graph
20 K-means algorithm in TensorFlow EECE695J (2017) Sang Jun Lee (POSTECH) れ 糾骸 (100 覦覲) K=4 K=2
21 K-means algorithm 譴 螳讌 伎 Mean initialization 企至 蟆瑚? Random initialization: N螳 譴 k螳襯 覓伎襦 觸 豐蠍郁朱 豐蠍郁 磯 clustering 蟆郁骸螳 譬 Random initialization 螻殊 覯 Cluster 螳 k 蟆一 朱朱 cluster 螳襯 一危一 覿 磯 蟆一.. K-means algorithm EECE695J (2017) Sang Jun Lee (POSTECH) 蠏碁 豢豌: https://wikidocs.net/4693
22 K-means algorithm 譴 螳讌 伎 Cluster 螳 k 蟆一 Elbow method: k螳 磯ジ cost function 襯 蠏碁語 , 轟 k 危 cost螳 覲螳 elbow point襯 k螳朱 蟆一 k螳 磯ジ cost function 覲螳 smooth 蟆曙 elbow point襯 谿剰鍵 . K-means algorithm EECE695J (2017) Sang Jun Lee (POSTECH) 蠏碁 豢豌: https://wikidocs.net/4693
23 Clustering criteria: The affinities of data within the same cluster should be high The affinities of data between different clusters should be low Spectral clustering EECE695J (2017) 蠏碁 谿語＾: POSTECH CSED441 lecture13 一危 伎 蟯
24 Optimization objective: Affinity 豐 豕螳 cluster襯 谿城 蟆 覈 ( : 一危一 cluster 覲企ゼ 企 vector) Convert the discrete optimization problem to continuous domain (+ ヰレ length襦 normalize) Spectral clustering EECE695J (2017) Discrete optimization problem Maximum eigenvalue problem
25 Spectral clustering るジ 危 Minimum-cut algorithm Spectral clustering EECE695J (2017)
26 Minimum-cut algorithm Gaussian similarity function Spectral clustering EECE695J (2017) ゐゐ = exp{ ｐｐ ｐｐ 2 22 } Diagonal = 1 Off-diagonal 1 Affinity matrix matrix
27 Objective function: Spectral clustering EECE695J (2017) Reference: POSTECH CSED441 lecture13
28 豕 覓語 蟆郁記 螳 eigenvalue problem朱 覦.. (Unnormalized) graph Laplacian: For every vector , 瑞瑞 = 1 2 =1 =1 ゐゐ ヰヰ ヰヰ 2 0 (positive semi-definite) 瑞酔 1螳 = 0 襷譟燕 eigenvalue襯 螳讌覃, 蠏 eigenvector = 企. 覈 一危郁 cluster襦 覓苦企 蟆 覩瑚 朱襦.. Minimum-cut algorithm solution second minimum eigenvalue 企麹 eigenvector Spectral clustering EECE695J (2017) 瑞 = 倹
29 Minimum-cut algorithm: Find the second minimum eigenvector of 瑞 = 倹 Partition the second minimum eigenvector 谿瑚: Two moon data second minimum eigenvector 谿瑚: Minimum-cut algorithm optimal solution 谿城 蟆 optimal solution approximation 谿城 Spectral clustering EECE695J (2017) Second smallest eigenvector
30 K-means 螻襴讀 螳! 豐蠍郁 磯ジ 螻襴讀 蟆郁骸 覲 Spectral clustering 螻磯 襷 一危一 cluster螳 highly non-convex 螻殊! 一危一 觜訣 蟆郁骸 豢 K-means vs. spectral clustering EECE695J (2017) Reference: POSTECH Machine Learning lecture8 K-means Spectral clustering
31 朱 一危郁 螻 螳 覿襯 螳讌 , (觜螳 り骸 螳) 伎 一危磯れ 蟆豢企企 覦覯? Anomaly detection EECE695J (2017)
32 朱 一危郁 螻 螳 覿襯 螳讌 , (觜螳 り骸 螳) 伎 一危磯れ 蟆豢企企 覦覯? Density estimation using multivariate Gaussian distribution Anomaly detection EECE695J (2017)
33 Density estimation using multivariate Gaussian distribution Parameter fitting: Given training set {ヰ : = 1, , } = 1 鐃 =1 ヰ() 裡 = 1 鐃 =1 ヰ ヰ Anomaly if ヰ; , 裡 < for given Anomaly detection EECE695J (2017)
34 Density estimation using multivariate Gaussian distribution Anomaly detection EECE695J (2017)
35 Regularization for preventing overfitting Unsupervised learning Clustering: k-means, spectral clustering Anomaly detection (density estimation) Summary
36 Date: 2017. 9. 21 (Thur) Time: 14:00-15:15 Introduction to neural networks Basic structure of neural networks: linear transformation, activation functions Implementation of neural network in Python and TensorFlow Preview (Lecture 4)

Recommended

Lecture 4: Neural Networks I
Lecture 4: Neural Networks ILecture 4: Neural Networks I
Lecture 4: Neural Networks I
Sang Jun Lee
油
Lecture 2: Supervised Learning
Lecture 2: Supervised LearningLecture 2: Supervised Learning
Lecture 2: Supervised Learning
Sang Jun Lee
油
Recurrent Neural Net 企螻 る
Recurrent Neural Net 企螻 るRecurrent Neural Net 企螻 る
Recurrent Neural Net 企螻 る
覦 蟾
油
Generative adversarial network
Generative adversarial networkGenerative adversarial network
Generative adversarial network
螳覩手記 螳覩手記
油
03.12 cnn backpropagation
03.12 cnn backpropagation03.12 cnn backpropagation
03.12 cnn backpropagation
Dea-hwan Ki
油
[蠍一螳] Graph Convolutional Network (GCN)
[蠍一螳] Graph Convolutional Network (GCN)[蠍一螳] Graph Convolutional Network (GCN)
[蠍一螳] Graph Convolutional Network (GCN)
Donghyeon Kim
油
Variational AutoEncoder(VAE)
Variational AutoEncoder(VAE)Variational AutoEncoder(VAE)
Variational AutoEncoder(VAE)
螳覩手記 螳覩手記
油
Dsh data sensitive hashing for high dimensional k-nn search
Dsh data sensitive hashing for high dimensional k-nn searchDsh data sensitive hashing for high dimensional k-nn search
Dsh data sensitive hashing for high dimensional k-nn search
WooSung Choi
油
neural network 蠍一
neural network 蠍一neural network 蠍一
neural network 蠍一
Dea-hwan Ki
油
Focal loss (Detection & Classification)
Focal loss (Detection & Classification)Focal loss (Detection & Classification)
Focal loss (Detection & Classification)
覦 蟾
油
InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...
InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...
InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...
覦 蟾
油
Anomaly Detection with GANs
Anomaly Detection with GANsAnomaly Detection with GANs
Anomaly Detection with GANs
覦 蟾
油
Vs^3 net for machine reading comprehension question answering
Vs^3 net for machine reading comprehension question answeringVs^3 net for machine reading comprehension question answering
Vs^3 net for machine reading comprehension question answering
NAVER Engineering
油
蠍一 Variational autoencoder
蠍一 Variational autoencoder蠍一 Variational autoencoder
蠍一 Variational autoencoder
覦 蟾
油
A neural image caption generator
A neural image caption generatorA neural image caption generator
A neural image caption generator
覦 蟾
油
檎稼稼螳襴
檎稼稼螳襴檎稼稼螳襴
檎稼稼螳襴
譬 豕
油
Deep Learning & Convolutional Neural Network
Deep Learning & Convolutional Neural NetworkDeep Learning & Convolutional Neural Network
Deep Learning & Convolutional Neural Network
agdatalab
油
Attention is all you need る
Attention is all you need るAttention is all you need る
Attention is all you need る
Junho Lee
油
[蠍] Tutorial: Sparse variational dropout
[蠍] Tutorial: Sparse variational dropout[蠍] Tutorial: Sparse variational dropout
[蠍] Tutorial: Sparse variational dropout
Wuhyun Rico Shin
油
Deep learning seminar_snu_161031
Deep learning seminar_snu_161031Deep learning seminar_snu_161031
Deep learning seminar_snu_161031
Jinwon Lee
油
Q Learning螻 CNN 伎 Object Localization
Q Learning螻 CNN 伎 Object LocalizationQ Learning螻 CNN 伎 Object Localization
Q Learning螻 CNN 伎 Object Localization
覦 蟾
油
Graph Neural Network (蟲)
Graph Neural Network (蟲)Graph Neural Network (蟲)
Graph Neural Network (蟲)
Jungwon Kim
油
Learning to remember rare events
Learning to remember rare eventsLearning to remember rare events
Learning to remember rare events
覦 蟾
油
Murpy's Machine Learing: 10. Directed Graphical Model
Murpy's Machine Learing: 10. Directed Graphical ModelMurpy's Machine Learing: 10. Directed Graphical Model
Murpy's Machine Learing: 10. Directed Graphical Model
Jungkyu Lee
油
7. Linear Regression
7. Linear Regression7. Linear Regression
7. Linear Regression
Jungkyu Lee
油
8. Logistic Regression
8. Logistic Regression8. Logistic Regression
8. Logistic Regression
Jungkyu Lee
油
RUCK 2017 觜一危 覿 覈
RUCK 2017 觜一危 覿 覈 RUCK 2017 觜一危 覿 覈
RUCK 2017 觜一危 覿 覈
r-kor
油
襾誤殊 襾語 : Gaussian Processes
襾誤殊 襾語 : Gaussian Processes襾誤殊 襾語 : Gaussian Processes
襾誤殊 襾語 : Gaussian Processes
Jungkyu Lee
油
Deep Learning from scratch 4 : neural network learning
Deep Learning from scratch 4 : neural network learningDeep Learning from scratch 4 : neural network learning
Deep Learning from scratch 4 : neural network learning
JinSooKim80
油
Guided policy search
Guided policy searchGuided policy search
Guided policy search
Jaehyeon Park
油

More Related Content

What's hot (20)

neural network 蠍一
neural network 蠍一neural network 蠍一
neural network 蠍一
Dea-hwan Ki
油
Focal loss (Detection & Classification)
Focal loss (Detection & Classification)Focal loss (Detection & Classification)
Focal loss (Detection & Classification)
覦 蟾
油
InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...
InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...
InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...
覦 蟾
油
Anomaly Detection with GANs
Anomaly Detection with GANsAnomaly Detection with GANs
Anomaly Detection with GANs
覦 蟾
油
Vs^3 net for machine reading comprehension question answering
Vs^3 net for machine reading comprehension question answeringVs^3 net for machine reading comprehension question answering
Vs^3 net for machine reading comprehension question answering
NAVER Engineering
油
蠍一 Variational autoencoder
蠍一 Variational autoencoder蠍一 Variational autoencoder
蠍一 Variational autoencoder
覦 蟾
油
A neural image caption generator
A neural image caption generatorA neural image caption generator
A neural image caption generator
覦 蟾
油
檎稼稼螳襴
檎稼稼螳襴檎稼稼螳襴
檎稼稼螳襴
譬 豕
油
Deep Learning & Convolutional Neural Network
Deep Learning & Convolutional Neural NetworkDeep Learning & Convolutional Neural Network
Deep Learning & Convolutional Neural Network
agdatalab
油
Attention is all you need る
Attention is all you need るAttention is all you need る
Attention is all you need る
Junho Lee
油
[蠍] Tutorial: Sparse variational dropout
[蠍] Tutorial: Sparse variational dropout[蠍] Tutorial: Sparse variational dropout
[蠍] Tutorial: Sparse variational dropout
Wuhyun Rico Shin
油
Deep learning seminar_snu_161031
Deep learning seminar_snu_161031Deep learning seminar_snu_161031
Deep learning seminar_snu_161031
Jinwon Lee
油
Q Learning螻 CNN 伎 Object Localization
Q Learning螻 CNN 伎 Object LocalizationQ Learning螻 CNN 伎 Object Localization
Q Learning螻 CNN 伎 Object Localization
覦 蟾
油
Graph Neural Network (蟲)
Graph Neural Network (蟲)Graph Neural Network (蟲)
Graph Neural Network (蟲)
Jungwon Kim
油
Learning to remember rare events
Learning to remember rare eventsLearning to remember rare events
Learning to remember rare events
覦 蟾
油
Murpy's Machine Learing: 10. Directed Graphical Model
Murpy's Machine Learing: 10. Directed Graphical ModelMurpy's Machine Learing: 10. Directed Graphical Model
Murpy's Machine Learing: 10. Directed Graphical Model
Jungkyu Lee
油
7. Linear Regression
7. Linear Regression7. Linear Regression
7. Linear Regression
Jungkyu Lee
油
8. Logistic Regression
8. Logistic Regression8. Logistic Regression
8. Logistic Regression
Jungkyu Lee
油
RUCK 2017 觜一危 覿 覈
RUCK 2017 觜一危 覿 覈 RUCK 2017 觜一危 覿 覈
RUCK 2017 觜一危 覿 覈
r-kor
油
襾誤殊 襾語 : Gaussian Processes
襾誤殊 襾語 : Gaussian Processes襾誤殊 襾語 : Gaussian Processes
襾誤殊 襾語 : Gaussian Processes
Jungkyu Lee
油
neural network 蠍一
neural network 蠍一neural network 蠍一
neural network 蠍一
Dea-hwan Ki
油
Focal loss (Detection & Classification)
Focal loss (Detection & Classification)Focal loss (Detection & Classification)
Focal loss (Detection & Classification)
覦 蟾
油
InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...
InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...
InfoGAN: Interpretable Representation Learning by Information Maximizing Gene...
覦 蟾
油
Anomaly Detection with GANs
Anomaly Detection with GANsAnomaly Detection with GANs
Anomaly Detection with GANs
覦 蟾
油
Vs^3 net for machine reading comprehension question answering
Vs^3 net for machine reading comprehension question answeringVs^3 net for machine reading comprehension question answering
Vs^3 net for machine reading comprehension question answering
NAVER Engineering
油
蠍一 Variational autoencoder
蠍一 Variational autoencoder蠍一 Variational autoencoder
蠍一 Variational autoencoder
覦 蟾
油
A neural image caption generator
A neural image caption generatorA neural image caption generator
A neural image caption generator
覦 蟾
油
檎稼稼螳襴
檎稼稼螳襴檎稼稼螳襴
檎稼稼螳襴
譬 豕
油
Deep Learning & Convolutional Neural Network
Deep Learning & Convolutional Neural NetworkDeep Learning & Convolutional Neural Network
Deep Learning & Convolutional Neural Network
agdatalab
油
Attention is all you need る
Attention is all you need るAttention is all you need る
Attention is all you need る
Junho Lee
油
[蠍] Tutorial: Sparse variational dropout
[蠍] Tutorial: Sparse variational dropout[蠍] Tutorial: Sparse variational dropout
[蠍] Tutorial: Sparse variational dropout
Wuhyun Rico Shin
油
Deep learning seminar_snu_161031
Deep learning seminar_snu_161031Deep learning seminar_snu_161031
Deep learning seminar_snu_161031
Jinwon Lee
油
Q Learning螻 CNN 伎 Object Localization
Q Learning螻 CNN 伎 Object LocalizationQ Learning螻 CNN 伎 Object Localization
Q Learning螻 CNN 伎 Object Localization
覦 蟾
油
Graph Neural Network (蟲)
Graph Neural Network (蟲)Graph Neural Network (蟲)
Graph Neural Network (蟲)
Jungwon Kim
油
Learning to remember rare events
Learning to remember rare eventsLearning to remember rare events
Learning to remember rare events
覦 蟾
油
Murpy's Machine Learing: 10. Directed Graphical Model
Murpy's Machine Learing: 10. Directed Graphical ModelMurpy's Machine Learing: 10. Directed Graphical Model
Murpy's Machine Learing: 10. Directed Graphical Model
Jungkyu Lee
油
7. Linear Regression
7. Linear Regression7. Linear Regression
7. Linear Regression
Jungkyu Lee
油
8. Logistic Regression
8. Logistic Regression8. Logistic Regression
8. Logistic Regression
Jungkyu Lee
油
RUCK 2017 觜一危 覿 覈
RUCK 2017 觜一危 覿 覈 RUCK 2017 觜一危 覿 覈
RUCK 2017 觜一危 覿 覈
r-kor
油
襾誤殊 襾語 : Gaussian Processes
襾誤殊 襾語 : Gaussian Processes襾誤殊 襾語 : Gaussian Processes
襾誤殊 襾語 : Gaussian Processes
Jungkyu Lee
油

Similar to Lecture 3: Unsupervised Learning (20)

Deep Learning from scratch 4 : neural network learning
Deep Learning from scratch 4 : neural network learningDeep Learning from scratch 4 : neural network learning
Deep Learning from scratch 4 : neural network learning
JinSooKim80
油
Guided policy search
Guided policy searchGuided policy search
Guided policy search
Jaehyeon Park
油
Coursera Machine Learning (by Andrew Ng)_螳襴
Coursera Machine Learning (by Andrew Ng)_螳襴Coursera Machine Learning (by Andrew Ng)_螳襴
Coursera Machine Learning (by Andrew Ng)_螳襴
SANG WON PARK
油
Neural network (perceptron)
Neural network (perceptron)Neural network (perceptron)
Neural network (perceptron)
Jeonghun Yoon
油
Deep Learning from scratch 5 : backpropagation
Deep Learning from scratch 5 : backpropagation Deep Learning from scratch 5 : backpropagation
Deep Learning from scratch 5 : backpropagation
JinSooKim80
油
[SOPT] 一危 蟲譟 覦 螻襴讀 ろ磯 - #01 : 螳, 蠏殊 覲旧°, 覦一, 郁屋襴ろ
[SOPT] 一危 蟲譟 覦 螻襴讀 ろ磯 - #01 : 螳, 蠏殊 覲旧°, 覦一, 郁屋襴ろ[SOPT] 一危 蟲譟 覦 螻襴讀 ろ磯 - #01 : 螳, 蠏殊 覲旧°, 覦一, 郁屋襴ろ
[SOPT] 一危 蟲譟 覦 螻襴讀 ろ磯 - #01 : 螳, 蠏殊 覲旧°, 覦一, 郁屋襴ろ
S.O.P.T - Shout Our Passion Together
油
Eigendecomposition and pca
Eigendecomposition and pcaEigendecomposition and pca
Eigendecomposition and pca
Jinhwan Suk
油
2016 UCPC
2016 UCPC 2016 UCPC
2016 UCPC
JeonDaePeuYeon
油
Control as Inference.pptx
Control as Inference.pptxControl as Inference.pptx
Control as Inference.pptx
ssuserbd1647
油
Gaussian Mixture Model
Gaussian Mixture ModelGaussian Mixture Model
Gaussian Mixture Model
KyeongUkJang
油
Multinomial classification and application of ML
Multinomial classification and application of MLMultinomial classification and application of ML
Multinomial classification and application of ML
覦
油
螳蟾螻 襾 Trpo
螳蟾螻 襾 Trpo螳蟾螻 襾 Trpo
螳蟾螻 襾 Trpo
Woong won Lee
油
Auto-Encoders and Variational Auto-Encoders
Auto-Encoders and Variational Auto-EncodersAuto-Encoders and Variational Auto-Encoders
Auto-Encoders and Variational Auto-Encoders
Jinho Lee
油
RLCode A3C 所 蟾蟆 危危蠍
RLCode A3C 所 蟾蟆 危危蠍RLCode A3C 所 蟾蟆 危危蠍
RLCode A3C 所 蟾蟆 危危蠍
Woong won Lee
油
伎螻 貅殊る 覦一磯 螳 濠
伎螻 貅殊る 覦一磯 螳 濠伎螻 貅殊る 覦一磯 螳 濠
伎螻 貅殊る 覦一磯 螳 濠
Woong won Lee
油
[Probability for machine learning]
[Probability for machine learning][Probability for machine learning]
[Probability for machine learning]
螳覩手記 螳覩手記
油
07. PCA
07. PCA07. PCA
07. PCA
Jeonghun Yoon
油
3.neural networks
3.neural networks3.neural networks
3.neural networks
Haesun Park
油
08. spectal clustering
08. spectal clustering08. spectal clustering
08. spectal clustering
Jeonghun Yoon
油
蟲瑚概讌レ郁規 螳給 蟆郁骸覲願
蟲瑚概讌レ郁規 螳給 蟆郁骸覲願蟲瑚概讌レ郁規 螳給 蟆郁骸覲願
蟲瑚概讌レ郁規 螳給 蟆郁骸覲願
Euijin Jeong
油
Deep Learning from scratch 4 : neural network learning
Deep Learning from scratch 4 : neural network learningDeep Learning from scratch 4 : neural network learning
Deep Learning from scratch 4 : neural network learning
JinSooKim80
油
Guided policy search
Guided policy searchGuided policy search
Guided policy search
Jaehyeon Park
油
Coursera Machine Learning (by Andrew Ng)_螳襴
Coursera Machine Learning (by Andrew Ng)_螳襴Coursera Machine Learning (by Andrew Ng)_螳襴
Coursera Machine Learning (by Andrew Ng)_螳襴
SANG WON PARK
油
Neural network (perceptron)
Neural network (perceptron)Neural network (perceptron)
Neural network (perceptron)
Jeonghun Yoon
油
Deep Learning from scratch 5 : backpropagation
Deep Learning from scratch 5 : backpropagation Deep Learning from scratch 5 : backpropagation
Deep Learning from scratch 5 : backpropagation
JinSooKim80
油
[SOPT] 一危 蟲譟 覦 螻襴讀 ろ磯 - #01 : 螳, 蠏殊 覲旧°, 覦一, 郁屋襴ろ
[SOPT] 一危 蟲譟 覦 螻襴讀 ろ磯 - #01 : 螳, 蠏殊 覲旧°, 覦一, 郁屋襴ろ[SOPT] 一危 蟲譟 覦 螻襴讀 ろ磯 - #01 : 螳, 蠏殊 覲旧°, 覦一, 郁屋襴ろ
[SOPT] 一危 蟲譟 覦 螻襴讀 ろ磯 - #01 : 螳, 蠏殊 覲旧°, 覦一, 郁屋襴ろ
S.O.P.T - Shout Our Passion Together
油
Eigendecomposition and pca
Eigendecomposition and pcaEigendecomposition and pca
Eigendecomposition and pca
Jinhwan Suk
油
2016 UCPC
2016 UCPC 2016 UCPC
2016 UCPC
JeonDaePeuYeon
油
Control as Inference.pptx
Control as Inference.pptxControl as Inference.pptx
Control as Inference.pptx
ssuserbd1647
油
Gaussian Mixture Model
Gaussian Mixture ModelGaussian Mixture Model
Gaussian Mixture Model
KyeongUkJang
油
Multinomial classification and application of ML
Multinomial classification and application of MLMultinomial classification and application of ML
Multinomial classification and application of ML
覦
油
螳蟾螻 襾 Trpo
螳蟾螻 襾 Trpo螳蟾螻 襾 Trpo
螳蟾螻 襾 Trpo
Woong won Lee
油
Auto-Encoders and Variational Auto-Encoders
Auto-Encoders and Variational Auto-EncodersAuto-Encoders and Variational Auto-Encoders
Auto-Encoders and Variational Auto-Encoders
Jinho Lee
油
RLCode A3C 所 蟾蟆 危危蠍
RLCode A3C 所 蟾蟆 危危蠍RLCode A3C 所 蟾蟆 危危蠍
RLCode A3C 所 蟾蟆 危危蠍
Woong won Lee
油
伎螻 貅殊る 覦一磯 螳 濠
伎螻 貅殊る 覦一磯 螳 濠伎螻 貅殊る 覦一磯 螳 濠
伎螻 貅殊る 覦一磯 螳 濠
Woong won Lee
油
[Probability for machine learning]
[Probability for machine learning][Probability for machine learning]
[Probability for machine learning]
螳覩手記 螳覩手記
油
07. PCA
07. PCA07. PCA
07. PCA
Jeonghun Yoon
油
3.neural networks
3.neural networks3.neural networks
3.neural networks
Haesun Park
油
08. spectal clustering
08. spectal clustering08. spectal clustering
08. spectal clustering
Jeonghun Yoon
油
蟲瑚概讌レ郁規 螳給 蟆郁骸覲願
蟲瑚概讌レ郁規 螳給 蟆郁骸覲願蟲瑚概讌レ郁規 螳給 蟆郁骸覲願
蟲瑚概讌レ郁規 螳給 蟆郁骸覲願
Euijin Jeong
油

Lecture 3: Unsupervised Learning

  • 1. Unsupervised Learning (+ regularization) Sang Jun Lee Ph.D. candidate, POSTECH Email: lsj4u0208@postech.ac.kr EECE695J 蠍郁概豪J(ル蠍一覦鶮螳螻旧) LECTURE 3 (2017. 9. 14)
  • 2. 2 Lecture 2: supervised learning Machine learning : supervised learning, unsupervised learning, etc. Supervised learning: linear regression, logistic regression (classification), etc. 1-page Review Supervised learning Given a set of labeled example, 倹 = (ヰヰ＄, ＄) ＄=1 , learn a mapping : which minimizes L(鐃署 = , ) Unsupervised learning Given a set of unlabeled example, 倹 = (ヰヰ＄) ＄=1 , learn a meaningful representation of the data Linear regression 署 = =0 ヰヰ = 署署 署 署 = 1 2 裡=1 署 2 腫 署 ヰヰ () 署 = 1 1+署 署 0.5 = 1 署 < 0.5 = 0 署 = =1 () log 署( ) + (1 ) log(1 署( )) + 腫 署 ヰヰ Logistic regression
  • 3. Overfitting? Training data 讌豺蟆(over) fit 朱 豢碁ゼ 讌 覈詩 覓語 朱朱 給一危一 讌豺蟆 ｸル 覲旧″ 覈(覿 curve)襦 誤 覦 Overfitting 譴願鍵 覦覯 Reduce the number of parameters (一危磯ゼ feature vector襯 螻殊朱 蟲燕朱 data dimension 豢) Regularization (cost function) 3 The problem of overfitting hypothesis function 蟆企朱 螳
  • 4. Regularization Parameter 螳襯 讌 蠍磯ゼ 螳伎朱 overfitting 狩 覦覯 Housing price prediction 覓語 hypothesis function: 署 = 0 + 1 ヰ1 + 2 ヰ2 覈語 磯 under-fitting (high bias) overfitting (high variance) 覓語螳 覦 ! 4 The problem of overfitting
  • 5. Regularization Linear regression cost function: 署 署 = 1 2 鐃 =1 署 2 parameter 署 蠍磯ゼ 譴企 regularization parameter Gradient descent update rule: 5 The problem of overfitting 署 署 = 1 2 鐃 =1 署 2 + 鐃 =1 2 0 0 腫 署 ヰ0 () 1 腫 腫 1 鐃 =1 署 ヰヰ 0 regularize 讌 Update parameter 蠍磯ゼ 螳る 螻!
  • 6. Normal equation 覲 Linear regression normal equation : , where = , 署 = 0 , and = (1) () Regularization parameter襯 覃 谿瑚: + 腫殊 譟伎 ( > 0) 6 Regularization Positive semi-definiter (if , is not invertiable) Positive definite
  • 7. Validation set 蟲! 覲旧° 覈語 蟲燕 validation 焔レ 蟯谿 署 = 0 + 1 ヰ1 + + ヰヰ 蠍一 regularization parameter ()襯 伎朱 蟆 譴! (螳 覓 覃 under-fitting 覓語螳 覦) 7 Bias vs. Variance 蠏碁 豢豌: https://www.coursera.org/learn/machine-learning/resources/LIZza
  • 8. 8 Clustering 觜訣 轟煙 一危磯れ 覓矩 螻襴讀 K-means algorithm Spectral clustering Anomaly detection Density estimation Unsupervised learning EECE695J (2017) Sang Jun Lee (POSTECH) K-means Cluster center襦覿一 蟇磯Μ襯 蠍一朱 clustering Spectral clustering 咋(, 乞): vertices and edges 朱 觜朱 蠏碁 襯 partitioning 蟆瑚
  • 9. 9 譯殊伎 螳 一危( 倹 ＄=1 )襯 螳 center ( = 1, , 情)襯 谿城 覓語 一危磯れ 螳 蠏碁９朱 partitioning Initialize ( = 1, , 情) Repeat until convergence! K-means algorithm EECE695J (2017) Sang Jun Lee (POSTECH) Assignment step Assign all data points to the cluster for which 2 is smallest Update step Compute new means for every cluster = 1 裡＄≠駒駒 谿場狩蟆 optimal cluster optimal mean碁.. mean 螻 optimal cluster襯 襾殊 谿剰 谿場 optimal cluster 襦 optimal mean 螻 Not optimal solution!
  • 10. 10 K-means algorithm EECE695J (2017) Sang Jun Lee (POSTECH) Initialize
  • 11. 11 K-means algorithm EECE695J (2017) Sang Jun Lee (POSTECH) Initialize Assignment step
  • 12. 12 K-means algorithm EECE695J (2017) Sang Jun Lee (POSTECH) Initialize Assignment step Update step
  • 13. 13 K-means algorithm EECE695J (2017) Sang Jun Lee (POSTECH) Initialize Assignment step Update step Assignment step
  • 14. 14 K-means algorithm EECE695J (2017) Sang Jun Lee (POSTECH) Initialize Assignment step Update step Assignment step Update step
  • 15. 15 K-means algorithm EECE695J (2017) Sang Jun Lee (POSTECH) Initialize Assignment step Update step Assignment step Update step Assignment step
  • 16. 16 K-means algorithm in TensorFlow EECE695J (2017) Sang Jun Lee (POSTECH) ipython 轟 蠏碁殊 蠏碁Μ蠍 覈轟 螳 normal distribution朱 sample data 蟲 vectors_set : 2000x2
  • 17. 17 K-means algorithm in TensorFlow EECE695J (2017) Sang Jun Lee (POSTECH)
  • 18. 18 K-means algorithm in TensorFlow EECE695J (2017) Sang Jun Lee (POSTECH) 2000螳 譴 k螳襯 襦 initial mean 螳朱 expanded_vectors: (tensor) 1x2000x2 expanded_centroids: (tensor) 4x1x2
  • 19. 19 K-means algorithm in TensorFlow EECE695J (2017) Sang Jun Lee (POSTECH) Assignment 螻殊 螳 れ願 蟆 殊 operation 企麹 graph
  • 20. 20 K-means algorithm in TensorFlow EECE695J (2017) Sang Jun Lee (POSTECH) れ 糾骸 (100 覦覲) K=4 K=2
  • 21. 21 K-means algorithm 譴 螳讌 伎 Mean initialization 企至 蟆瑚? Random initialization: N螳 譴 k螳襯 覓伎襦 觸 豐蠍郁朱 豐蠍郁 磯 clustering 蟆郁骸螳 譬 Random initialization 螻殊 覯 Cluster 螳 k 蟆一 朱朱 cluster 螳襯 一危一 覿 磯 蟆一.. K-means algorithm EECE695J (2017) Sang Jun Lee (POSTECH) 蠏碁 豢豌: https://wikidocs.net/4693
  • 22. 22 K-means algorithm 譴 螳讌 伎 Cluster 螳 k 蟆一 Elbow method: k螳 磯ジ cost function 襯 蠏碁語 , 轟 k 危 cost螳 覲螳 elbow point襯 k螳朱 蟆一 k螳 磯ジ cost function 覲螳 smooth 蟆曙 elbow point襯 谿剰鍵 . K-means algorithm EECE695J (2017) Sang Jun Lee (POSTECH) 蠏碁 豢豌: https://wikidocs.net/4693
  • 23. 23 Clustering criteria: The affinities of data within the same cluster should be high The affinities of data between different clusters should be low Spectral clustering EECE695J (2017) 蠏碁 谿語＾: POSTECH CSED441 lecture13 一危 伎 蟯
  • 24. 24 Optimization objective: Affinity 豐 豕螳 cluster襯 谿城 蟆 覈 ( : 一危一 cluster 覲企ゼ 企 vector) Convert the discrete optimization problem to continuous domain (+ ヰレ length襦 normalize) Spectral clustering EECE695J (2017) Discrete optimization problem Maximum eigenvalue problem
  • 25. 25 Spectral clustering るジ 危 Minimum-cut algorithm Spectral clustering EECE695J (2017)
  • 26. 26 Minimum-cut algorithm Gaussian similarity function Spectral clustering EECE695J (2017) ゐゐ = exp{ ｐｐ ｐｐ 2 22 } Diagonal = 1 Off-diagonal 1 Affinity matrix matrix
  • 27. 27 Objective function: Spectral clustering EECE695J (2017) Reference: POSTECH CSED441 lecture13
  • 28. 28 豕 覓語 蟆郁記 螳 eigenvalue problem朱 覦.. (Unnormalized) graph Laplacian: For every vector , 瑞瑞 = 1 2 =1 =1 ゐゐ ヰヰ ヰヰ 2 0 (positive semi-definite) 瑞酔 1螳 = 0 襷譟燕 eigenvalue襯 螳讌覃, 蠏 eigenvector = 企. 覈 一危郁 cluster襦 覓苦企 蟆 覩瑚 朱襦.. Minimum-cut algorithm solution second minimum eigenvalue 企麹 eigenvector Spectral clustering EECE695J (2017) 瑞 = 倹
  • 29. 29 Minimum-cut algorithm: Find the second minimum eigenvector of 瑞 = 倹 Partition the second minimum eigenvector 谿瑚: Two moon data second minimum eigenvector 谿瑚: Minimum-cut algorithm optimal solution 谿城 蟆 optimal solution approximation 谿城 Spectral clustering EECE695J (2017) Second smallest eigenvector
  • 30. 30 K-means 螻襴讀 螳! 豐蠍郁 磯ジ 螻襴讀 蟆郁骸 覲 Spectral clustering 螻磯 襷 一危一 cluster螳 highly non-convex 螻殊! 一危一 觜訣 蟆郁骸 豢 K-means vs. spectral clustering EECE695J (2017) Reference: POSTECH Machine Learning lecture8 K-means Spectral clustering
  • 31. 31 朱 一危郁 螻 螳 覿襯 螳讌 , (觜螳 り骸 螳) 伎 一危磯れ 蟆豢企企 覦覯? Anomaly detection EECE695J (2017)
  • 32. 32 朱 一危郁 螻 螳 覿襯 螳讌 , (觜螳 り骸 螳) 伎 一危磯れ 蟆豢企企 覦覯? Density estimation using multivariate Gaussian distribution Anomaly detection EECE695J (2017)
  • 33. 33 Density estimation using multivariate Gaussian distribution Parameter fitting: Given training set {ヰ : = 1, , } = 1 鐃 =1 ヰ() 裡 = 1 鐃 =1 ヰ ヰ Anomaly if ヰ; , 裡 < for given Anomaly detection EECE695J (2017)
  • 34. 34 Density estimation using multivariate Gaussian distribution Anomaly detection EECE695J (2017)
  • 35. 35 Regularization for preventing overfitting Unsupervised learning Clustering: k-means, spectral clustering Anomaly detection (density estimation) Summary
  • 36. 36 Date: 2017. 9. 21 (Thur) Time: 14:00-15:15 Introduction to neural networks Basic structure of neural networks: linear transformation, activation functions Implementation of neural network in Python and TensorFlow Preview (Lecture 4)
AboutCopyright
息 2025 際際滷