際際滷

際際滷Share a Scribd company logo

[????] ??? ??? EEG ?? ??

D
Donghyeon Kim

* ???? ??? EEG ??? ?? ??? ???? ?????. * EEG? ??? ???? 4?? ????? ?? ?????. * ??????? ???? ??? A-GIST ???? ??????. * ?? ?? (???, ???): https://youtu.be/gcP2vN41HZw

Technology
1 of 50
Downloaded 142 times
1
2
3
4
5
6
7
8
Most read
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
Most read
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
Most read
50
Introduction to EEG-based Systems with Deep Learning 2018.04.04. ??? (dhkim518@gist.ac.kr)
? Background (??? ?? ??) ? ??? ?? & ?-??? ????? ? ?-??? ?????: ???? 4?? ???? ?? & 7? ?? ?? ?? I. Resting State EEG ? [I-1] Seizure Detection ? [I-2] Gender Prediction II. Mental-task Imagery ? [II-1] Motor Imagery (Left/ Right) III. Steady-state Evoked Potential (SSEP) ? [III-1] SSVEP Classification IV. Event Related Potential (ERP) ? [IV-1] P300 Speller ? [IV-2] Transferring Human Visual Capabilities to Machine ? [IV-3] EEG-in/ Image-out ?? ?? 2
? ??? ?? ?? ???? ???? ????? ????? ? ?? ??? ???? ??? ?? ????? ???? ?? ? ??? ? ｀??? ??? ?? ??? ???/ ?? ????? ??? ?????¨ ? ??? ?? ?? ?? ??? ? ???? ??? ??? ????? ? ?? ??? ???? ?? ??? ?? ????? ?????? ????? ? ?????? ??? ??? ??? ｀?? ????? ???? ??? ???? ?? ?? ?? ???¨ ??? ? ?? ??, ????, BCI, ??? ?? ?? ??? ?? ? ?? ? CNN, LSTM, GAN ?? ??? ??? ?? ?? ? Temporal, spectral, spatial feature? ??? ?? ? ?? ? ????? 3
? ??? (Electroencephalogram, EEG) ? ? ?? ??? ??? ??? ? ??? ??? ??, ???? ??? ?? ??? ?? ? 1929? Hans Berger? ?? ???? ?? ???? ????? [Berger, ｀29] ? ?-??? ?????(Brain-computer interface, BCI)? ?? ???? ???? ?? ? BCI: ??? ???? ???? ??, ?? ??? ???? ??? ????? ? ??? ??? ?? 3??; ??, ???, ?? Background 4 ?? ??? ??? ??? ??, ?? ?? ??? ?? ??? ??? ?? ?? ??? δ (0.1-4 Hz), θ (4-8 Hz), α (8-13 Hz), β (13-30 Hz), γ (30-50 Hz) ?? ?? ?? ?? ???? ????? ? ?? ?? [Berger, ｀29] Berger, H. (1929). Ueber das Elektroenkephalogramm des Menschen. Archiv f┨r Psychiatrie und Nervenkrankheiten, 87(1), 527-570. International 10-20 System
? ???? ?? ?? ??? ?? ? ???? ??? ?? ?? ? I. Resting state EEG ? II. Mental-task Imagery ? III. Steady-state Evoked Potential (SSEP) ? IV. Event Related Potential (ERP) Background 5 Signal ProcessingSignal Acquisition Applications ? Feature extraction/ pattern recognition ? Machine learning based approaches ? Brainwave ? Entertainments (Game) ? Therapy/ healthcare ? Prostheses ? Authentication
? Resting EEG ? ??? ?? ????? ??? ?? ? ?? EO (Eye-open)?? EC (Eye-closed) ??? ??? ???? ?? ?? ? ???? ?? ? [I-1] Seizure Detection ? [I-2] Gender Prediction I. Resting State EEG 6 . . . .
? ?? ?? ? Seizure: ??, ?? ????? ???? ??? ????? ?? ?? ?? ? Epilepsy: ??, ??? ??? ??? ??? ?? ???? ?? [I-1] Seizure Detection 7 Acharya, U. R., Oh, S. L., Hagiwara, Y., Tan, J. H., & Adeli, H. (2017). Deep convolutional neural network for the automated detection and diagnosis of seizure using EEG signals. Computers in biology and medicine.
? Motivation & Objective ? EEG? ?? ??? ?? ??? ??? ?? ?? ??? ???? ???? ?? ? ???? ?? (unprovoked seizure)? 2? ?? ?? ? ??? ???? ? ??? ?? ??: ｀direct visual inspection¨ ? ?? ??? ??, ??? ?? ｀time-consuming¨, ｀technical artifact¨, ｀variable results¨, ｀limitation to identifying abnormality¨ ? ??? Computer-aided diagnosis (CAD) ???? ?? ? 13-layer Convolutional Neural Network ? Normal, Pre-ictal (?? ?? ??), Seizure 3?? ??? ?? ??? ???? [I-1] Seizure Detection 8
? Dataset ? Public dataset [Andrzejak, ｀01] ? 3 states: Normal, Pre-ictal, and Seizure ? 100 segments for each class ? ??? EEG segment ?? ? Single channel EEG signals ? 23.6 sec, Ns = 4097 (Ns: ?? ?) ? Z-score normalization (mean 0, std. 1) ? Architecture ? 13-layer CNN [I-1] Seizure Detection 9 [Andrzejak ¨01] R.G. Andrzejak, K. Lehnertz, C. Rieke, F. Mormann, P. David, C.E. Elger, Indications of nonlinear deterministic and finite dimensional structures in time series of brain electrical activity: dependence on recording region and brain state, Phys. Rev. E 64 (2001) 061907. Color information - Kernel size - Max-pooling - Fully connected layer
? Experimental Validation ? 10-fold cross-validation ? EEG signals? 10 ???? ???, 9 ?? train ? ?? 1 ?? test ?? ??? 10? ?? ? Train ???? 70 % ???? ???? 150 runs* ???? (batch size: 3), 30 % ???? validation? ?? (overfitting ?? ??) ? Results & Discussion ? Accuracy: 88.67%, Sensitivity: 95.00%, and Specificity: 90.00% ? ?? ??? ?? ???? ??? ???? ??? ? ? ??? ｀Novelty¨? EEG ??? Seizure detection? ???? ???? ??? ? ? ? ?? ??? ?? ???? ? ?? ??? ?? ??? ?? [I-1] Seizure Detection 10 *Run: one iteration of the full training set
? Motivation & Objective ? ????? ???? ????? ??? ????. ? ??? ??? ?? ?????, ?????? ???? ??????? ???? ?? ? ?? ??? ?? ?? ??? ??? ?? ? ??? ｀sex-specific information¨ ? ??? ???? ?????, ?????? ??? ?? ??? visual inspection/ quantitative inspection ??? ?? ?? ? ??: ??? ??? ?? ??? ?? ???? ??? ????? ???? ?? ?? [I-2] Gender Prediction 11 Putten, M. J., Olbrich, S., & Arns, M. (2018). Predicting sex from brain rhythms with deep learning. Scientific reports, 8(1), 3069.
? Dataset ? EEG data from 6 different Labs* ? 1308 subjects (mean age 43.38 (18.42 SD) y; range 18C98 y; 47% males) ? Eye-closed (EC) Resting EEG 2 min. ? ??? EEG segment ?? ? 24 channels/ 2 sec, Fs = 128 Hz, Ns = 256 (?? ?)/ 1 ???? 40 EEG segments ?? ? Band-pass filtering [0.5, 25] ? Architecture ? 6-layer Convolutional Neural Network [I-2] Gender Prediction 12 *Brain Resource International Database (New York, Rhode Island, Nijmegen, London, Adelaide and Sydney).
? Experimental Validation ? ??? ?? ??? ?? ???? cross-validation? ?? ?? ? 1000? ???? ??? (40,000 EEG segments, 47% male)? train??, 308? ???? ??? (12,320 EEG segments, 49% male)? test ? ?? ?? 150 runs* ?? (batch size: 70) ? Results & Discussion ? Accuracy: 81 % (p < 10-5) ? ???? ?? ?, ? ?? ??? ??? (?? layer, ? ? filter size, ?/?? ?? ?) ? Visualization of six convolutional layers ? Deep dream algorithm ?? [Mahendran, ¨15][Mahendran, ¨16] ? ??? CNN ???? ??? (12-25 Hz) ?? ?? ? ??? ??? logistic regression model ?? 70 % accuracy [I-2] Gender Prediction 13 [Mahendran, ¨15] Mahendran, A. & Vedaldi, A. Understanding deep image representations by inverting them. in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015). [Mahendran, ｀16] Mahendran, A. & Vedaldi, A. Visualizing Deep Convolutional Neural Networks Using Natural Pre-images. Int. J. Comput. Vis. 120, 233C255 (2016).
? Mental-task Imagery ? ???? ???? ???? ?? ?? ? ?? ?? ?? (motor imagery), ? ?? (speech imagery) ?? ?? ?? ? ???? ?? ? [II-1] Motor imagery: Imagination of moving a body part without actual movement II. Mental-task Imagery 14
? Motivation & Objective ? BCI ???? ?? ?? ?? ??? ????? ????? ??? ??? ? ????? ????? ｀Feature Extraction¨ ??? ｀Classification¨ ??? ??? ??? ?? ?? ?? ? ????? ? Common Spatial Pattern (CSP), Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), Support Vector Machine (SVM) ?? ?????? ? ?? training data ? ?? ???? ???? ?? ??? ?? ?? ?????, ?? ???? ???? BCI ???? ?????? ????? ???? ??? ??? ? ??? ? ??? ???? ???? ??? ??? ? ? ????? ??? ??? ?? ??? ?? ??? ?? ??? ?? ?? [II-1] Motor Imagery 15 Tabar, Y. R., & Halici, U. (2016). A novel deep learning approach for classification of EEG motor imagery signals. Journal of neural engineering, 14(1), 016003.
? Dataset ? Public dataset (BCI Competition IV dataset 2b) ? 9? ???? ???? ?? ?? ??? ?? ?? ? ??? EEG segment ?? ? 3 channels (C3, Cz, C4) ? 2 sec (cue? ??? ? 0.5~2.5? ?? ??? ??), Fs = 250 Hz ? 1 ???? 400 EEG segments ?? ? Input image form ? ?? (time), ??? (frequency), ?? (channel location) ??? ??? ??? ?? ? ? Short-time Fourier transform (STFT) ?? (window size: 64, time lapses = 14) ? 1 channel EEG segment (Ns = 500) ? STFT ? frequency-time image (257 x 32) ? ??? ??? ????? ???? mu ??(8-13 Hz) ? beta ??(17-30 Hz) ? ?? ? 3?? (C3, Cz, C4) ??? ?? Frequency-Time images (size of 31 x 32) ? ?? input ? 3??? ?? ??? ? ???? ?? (size of 93 x 32) [II-1] Motor Imagery 16
? Architecture ? Convolutional Neural Network (CNN) ? ?? ???? Convolution ?? ? ?? 30? ??, ? ?? ???? (93 x 3) ? ?? ? Max-pooling? (10 x 1) ? ?? ?? ? Batch size = 50?? 300 runs ?? ?? [II-1] Motor Imagery 17
? Architecture ? Convolutional Neural Network (CNN) ? Stacked Auto Encoder (SAE) ? 6 ?? ??? AE? ??? ???? ? SAE? ???? ??? AE ?? unsupervised ??? ?? ????? ???? ? ??? AE? Batch size = 20? 200 runs ?? ??? ?, Fine tuning ???? batch size = 40? 200 runs ?? ???? [II-1] Motor Imagery 18
? Architecture ? Convolutional Neural Network (CNN) ? Stacked Auto Encoder (SAE) ? Combined CNN-SAE ? CNN?? ??? convolutional layer ??? SAE? ??? ?? [II-1] Motor Imagery 19
? Architecture ? Convolutional Neural Network (CNN) ? ?? ???? Convolution ?? ? ?? 30? ??, ? ?? ???? (93 x 3) ? ?? ? Max-pooling? (10 x 1) ? ?? ?? ? Batch size = 50?? 300 runs ?? ?? ? Stacked Auto Encoder (SAE) ? 6 ?? ??? AE? ??? ???? ? SAE? ???? ??? AE ?? unsupervised ??? ?? ????? ???? ? ??? AE? Batch size = 20? 200 runs ?? ??? ?, Fine tuning ???? batch size = 40? 200 runs ?? ???? ? Combined CNN-SAE ? CNN?? ??? convolutional layer ??? SAE? ??? ?? [II-1] Motor Imagery 20
? Experimental Validation ? 10 x 10 Fold Cross-validation ? ? ???? 400 Trials? Random?? 10 ???? ???, 9 ?? train ? ?? 1 ?? test ?? ??? ?? ??? ??? 10? ?? ? Random ?? 10 ???? ??? ???? ??? ??? 10? ?? ? Results & Discussion ? CNN-SAE ? ?? ?? ??? ????, ??? ????? ?? ??? ???. ? CNN?? ??? ?? 30? ?? ? ??? Feature? ?? ??, ?? ??? ?? ? ??? ???? ?? ?? ?? ???? [II-1] Motor Imagery 21
? Steady-state Evoked Potential (SSEP) ? ??/ ??/ ?? ??? ????? ??? ? ???? ?? ? ?? ??? ???? ??? ?? ?? ? ???? ?? ? [III-1] SSVEP Classification III. Steady-state Evoked Potential 22 Flickering Light (20 Hz) Oscillatory Response (20 Hz)
[III-1] SSVEP Classification 23 Kwak, N. S., M┨ller, K. R., & Lee, S. W. (2017). A convolutional neural network for steady state visual evoked potential classification under ambulatory environment. PloS one, 12(2), e0172578.
? Dataset ? 7? ???? ???? ?? ?? ? ???? ? Random auditory cue? ???? 3? ?? start beep? ??? ???? 5?? ?? LED ?? ? Cue: ?? 9 Hz, ??? 11 Hz, ???? 13 Hz, ??? 15 Hz, ?? 17 Hz (5 classes) ? Task 1 (Static SSVEP): ??? ??? ???? cue? ?? (? class ? 10??, ? 50?) ? Task 2 (Ambulatory SSVEP): ??? ???? ??? cue ? ?? (? class ? 50??, ? 250 ?) ? ??? EEG segment ?? ? 8 channels (?? ?? ???) ? BPF [4 - 40 Hz] ??? EEG ??? 2 sec (Fs = 1000 Hz) ??? segment? ?? ? ??? ?? ? ??? FFT ? [5 - 35 Hz]? ???? 120 samples ?? ? 0-1 Normalization ? ? EEG segment? size? (120 x 8) ? ?? ?? ??? ???? ?? ??? ?? ? 2 sec sliding window with different shift sizes (60, 30, 20, 15, 12, 10 msec) ? ??? ? (50, 100, 150, 200, 250, 300 data samples) [III-1] SSVEP Classification 24 ´ Shift size
? Architecture ? CNN-1 ? [C1] Convolution? ????? 8? channels ???? ?? ? [C2] Convolution? ?????? 11 frequency samples ???? ?? [III-1] SSVEP Classification 25
? Architecture ? CNN-1 ? CNN-2 ? CNN-1? ?? ?? ?????, F3 layer? 3? ??? ???? Visualization ??? ?? [III-1] SSVEP Classification 26
? Architecture ? CNN-1 ? CNN-2 ? NN [III-1] SSVEP Classification 27
? Architecture ? CNN-1/ CNN-2/ NN ? Experimental Validation ? ? ??? ?? 2?? static/ ambulatory SSVEP ??? ??? 10-fold CV ?? ? ?? ?? ??? ???? ?? ??? ?? ? Learning rate = 0.1 / Learning iterations = 50? (10? ?? ??? error rate? 0.5 % ??? ???? ?? ??) [III-1] SSVEP Classification 28
? Results & Discussion ? static/ ambulatory ???? ??? ???? ?? ?, ??? ???? ??? ?? ?? ? Feature Representation (CNN-2 ?? ?? ??) [IV-1] SSVEP Classification 29
? Event Related Potential (ERP) ? ??/ ??/ ?? ?? ??? ???? ? ?? ? ??????, P300 ??? ??? ??? ??? ?? ?? ? ???? ?? ? [IV-1] P300 Speller ? [IV-2] Transferring Human Visual Capabilities to Machine ? [IV-3] EEG-in/ Image-out IV. Event Related Potential 30
? P300 Speller? ?? ? ??? ?? (a) P300 detection: ??? ?? ?? P300 ??? ?? ??? (b) Character Recognition: ???? ?? ??? ???? ??? [IV-1] P300 Speller 31 Cecotti, H., & Graser, A. (2011). Convolutional neural networks for P300 detection with application to brain-computer interfaces. IEEE transactions on pattern analysis and machine intelligence, 33(3), 433-445.
? Dataset ? Public dataset (Data set II from the third BCI competition) ? 2? ???? ???? P300 Speller ?? ?? ? Training database: ? 85 characters / Test database: ? 100 characters ? 12? (6 rows, 6 columns) ?? ?????, ? 12? ???? 15? ?? ? ??? EEG segment ?? ? 64 channels ? 650 msec (cue? ??? ? 0 ~ 650 msec ??? ??), Fs = 120 Hz ? ? ?? ? ?? Standardization ? Architecture ? [L0] Input size: 64 Ch. x 78 Ns ? [L1] Convolution ? (1x64) ??? ?? 10? ? 10@78x1 ? [L2] Convolution + Subsampling ? (13x1) ??? ?? 5? ? 50@6x1 ? [L3, L4] Fully connected ? 300 nodes C 100 nodes C 2 output [IV-1] P300 Speller 32
? Experimental Validation ? ? ??? ?? ??? training dataset ?? ????, test dataset?? ??? ?? ? Training dataset ? 95 % training, 5 %? validation ? Validation ???? ?? least mean square error ? ???? ??? ?? ?? (CNN ??? ??? variation ? ? ???? ??? ??? ??? ?????) ? Results & Discussion ? ??? ???? ???? ??? ?? ?? ? (A) P300 Detection (B) Character Recognition [IV-1] P300 Speller 33
? CNN? ? ??? ??? ???, ??? ??? visual object classifier ??? [IV-2] Transferring Human Visual Capabilities to Machine 34 Spampinato, C., Palazzo, S., Kavasidis, I., Giordano, D., Souly, N., & Shah, M. (2017, July). Deep learning human mind for automated visual classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 6809-6817).
? Objective & Contribution ? ?? ??? ?? ??? ??? ?? ??? ERP ??? ??? ? ?? ??? ?? ? ERP ?? ????? ?? visual descriptor? ??? computer vision ??? ?? ? ? ??? EEG ????? ?? ???? ? trained model ? publicly release ?? [IV-2] Transferring Human Visual Capabilities to Machine 35
Part 1. ?? ??? ??? ?? ERP ??? ? Dataset ? 6? ???? ???? ?? ?? ? ???? ? ImageNet ???? 40 classes 2,000 images (50 from each class) ?? ? 10 ? ???? ??? ???? 0.5 ?? ???? ???? EEG Recording ? ??? EEG segment ?? ? 128 channels ? BPF [14 - 71 Hz] ??? EEG ??? 440 msec (Fs = 1000 Hz) ?? ? Beta (15-31 Hz), Gamma (32-70 Hz) ??? ?? ?? ? cognitive process? ?? ? ??? ??? ?? 0.5 ? ??? ?? ?? 40 msec ??? 440 msec ??? ?? ? ? EEG segment? size? (128 x 440) [IV-2] Transferring Human Visual Capabilities to Machine 36
Part 1. ?? ??? ??? ?? ERP ??? ? Architecture ? a) Common LSTM ? Common? ??: ?? EEG ??(128?)? ???? ?? LSTM layer? ???? ?? [IV-2] Transferring Human Visual Capabilities to Machine 37
Part 1. ?? ??? ??? ?? ERP ??? ? Architecture ? a) Common LSTM ? b) Channel LSTM + Common LSTM ? ??? single EEG ???? LSTM ??? ????, ??? inter-channel analysis ? ???? ?? common LSTM ? ??? ?? [IV-2] Transferring Human Visual Capabilities to Machine 38
Part 1. ?? ??? ??? ?? ERP ??? ? Architecture ? a) Common LSTM ? b) Channel LSTM + Common LSTM ? c) Common LSTM + output layer ? ?? ??? a) common LSTM? ?????, ? ??? output layer? LSTM? ?? input?? ?? ?? (ReLU) ?? ???? ?? [IV-2] Transferring Human Visual Capabilities to Machine 39
Part 1. ?? ??? ??? ?? ERP ??? ? Experimental Validation ? Training 80 % (1,600 images), validation 10 % (200), test 10 % (200) ? Results & Discussion [IV-2] Transferring Human Visual Capabilities to Machine 40
Part 2. ???? EEG ?? Feature? ???? CNN-based Regression ? ?? ?? 2?? ? Approach 1. End-to-end training ? Pre-trained AlexNet CNN? weights? initialization? ?? ? GoogleNet, VGG? ?? ??? ??; ?? ????? ????? ???? ?? ? ? ? ??? layer? ?? softmax ???? regression layer? ?? ? Regression layer ?? ?? ?? EEG feature vector dimension? ?? ?? ? ?? ??? Euclidean loss? ?? ?? ?? [IV-2] Transferring Human Visual Capabilities to Machine 41
Part 2. ???? EEG ?? Feature? ???? CNN-based Regression ? ?? ?? 2?? ? Approach 2. Deep feature extraction followed by regressor training ? Pre-trained AlexNet, GoogleNet, VGG? weights? initialization? ?? ? ? ??? CNN ??? ???? ??? ??? ???? regression methods ? ?? ? k-NN regression, ridge regression, random forest regression ? CNN ???? ??? ??? ??? EEG feature vector? ?? [IV-2] Transferring Human Visual Capabilities to Machine 42
Part 2. ???? EEG ?? Feature? ???? CNN-based Regression ? Experimental Validation ? ?? Part 1?? ?? ??? ?? 128 Common + 128 Output ? feature vector ?? ? ? 2000 ?? ???? ??, CNN? regression ?? ?? ｀confusing¨ ? ???? ??? ??? ????? ??? EEG feature vector? ???? ?? ? ? ?? ???? ??? ?? ? Average: 6 ? ???? ? ???? ?? ??? feature vector? average ?? ? Best: ??? ???? ???, 6 ? ???? EEG feature vectors ??? ?? ?? classification loss? ?? ?? feature vector ??? ?? ? Result & Discussion ? Mean-square error? ?? ?? ? Approach 2? [GoogleNet + k-NN regression + Average EEG feature vector]? ?? ?? [IV-2] Transferring Human Visual Capabilities to Machine 43
Part 3. Automated Visual Classification ? Experimental Validation ? ?? Part 1, Part 2?? ?? ??? ?? ???? ?? ?? ? GoogleNet + k-NN + Average EEG feature vector + 128 Common, 128 Output ? ??? ??? ? ?? Test data (10 %, 200? ???)?? 89.7 % ?? ??? ? ???? ?? Caltech-101? 30 class? ?? ?? ?? ?? ? GoogleNet/ VGG/ ??? ??? feature extractor? ?? ? Multiclass SVM classifier? ?? ???? ?? ??? ?? ? Result & Discussion ? Human Brain-driven Automated Visual Classification ? GoogleNet? ????? ??? ????, VGG?? ??? ?? ?? ? EEG encoder? regressor? ???? ?? ?? ????? ??? ???? ???? ｀impressive¨ ?? ?? [IV-2] Transferring Human Visual Capabilities to Machine 44
? Read the Mind, Generate the Image ???? ???? ??? ? ????? ? ??? ??? ???? ????? [IV-3] EEG-in/ Image-out 45 Palazzo, S., Spampinato, C., Kavasidis, I., Giordano, D., & Shah, M. (2017, October). Generative Adversarial Networks Conditioned by Brain Signals. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3410-3418).
? Objective & Contribution ? Generative model? ??? ??? ??? ? ?? ???? ?? ? ??? ???? ? ??? ???? ?? ??, ??? ??? ????? ? ?? ?? ? Condition Generative Adversarial Network (cGAN) [IV-3] EEG-in/ Image-out 46
? Architecture ? A. Learning EEG Manifold ? B. Image Generation ? B-1. Generator ? B-2. Discriminator [IV-3] EEG-in/ Image-out 47 Input G1 G2 G3 G4 G5 1 x 100 (Random Noise) + 1 x 128 (EEG Feature) 4 x 4 x 512 8 x 8 x 256 16 x 16 x 128 32 x 32 x 64 64 x 64 x 3 Input DC1 DC2 DC3 DC4 DFC5 DFC6 64 x 64 x 3 32 x 32 x 64 16 x 16 x 128 8 x 8 x 256 4 x 4 x 512 + 4 x 4 x 8 (EEG Feature) 1024 1
? Experimental Validation ? Condition GAN ????? ?????, ?? ??? ?? ?? (40 class, 50 images) ? ???, ? ??? ??? ?? ????? ? 1??: EEG feature? zero vector? ??, non-conditional GAN ??? 100 epochs ?? ?? ? 2??: EEG feature vector? ?? ???? conditional GAN ??? 50 epochs ???? ? ??? ???? ??: inception score ? inception classification accuracy ? Results & Discussion ? ??? ???? ?? inception network classification ?? 43 % ??? ?? ? ?? ???? ????, 40 class ? ?? 43 % ?? ????? ?? (1/40 = 2.5 %) [IV-3] EEG-in/ Image-out 48 Good Results: Jack-o¨-Lantern & Panda Bad results: Banana & Bolete
? ??? ??? ??? ???? ???? 4?? ????? 7?? ?? ?? Conclusive Remarks 49 ?? ???? ?? ??? ??? ?? EEG domain I. Resting State EEG 1. Seizure Detection 1-D 13-L CNN ?? 2. Gender Prediction 6-L CNN ??-?? II. Mental-task Imagery 3. Motor Imagery (L/ R) CNN + SAE ??-??-??? III. Steady-state Evoked Potential 4. SSVEP Classification CNN & Visualization using 3 nodes ???-?? IV. Event Related Potential 5. P300 Speller CNN ??-?? 6. Human Visual Capabilities to Machine LSTM & CNN-regression ??-?? 7. EEG-in/ Image-out LSTM & (conditional) GAN ??-??
Thank you 50

More Related Content

What's hot (20)

PDF
フ`リエ篁擦隼R抹の碧怏み
yuichi takeda
?
PDF
Wasserstein GAN ?? ???? I
Sungbin Lim
?
PDF
???? ???? ????????, ???? ??? ??????.
Yongho Ha
?
PPTX
NLPにおけるAttention゛Seq2Seq から BERTまで゛
Takuya Ono
?
PPTX
Attention Is All You Need
Illia Polosukhin
?
PPTX
Deep Learning - RNN and CNN
Pradnya Saval
?
PDF
????? ???????? ?????? 1??? (2022??? ??????)
ssusercdf17c
?
PDF
＾An Introduction to Data Augmentation Techniques in ML Frameworks,￣ a Present...
Edge AI and Vision Alliance
?
PPTX
CNN Machine learning DeepLearning
Abhishek Sharma
?
PDF
[DLi氏]Hindsight Experience Replay
Deep Learning JP
?
PPTX
Transformer in Vision
Sangmin Woo
?
PDF
Attention is All You Need (Transformer)
Jeong-Gwan Lee
?
PPTX
Transformers in Vision: From Zero to Hero
Bill Liu
?
PDF
?????? ?? ?
NAVER Engineering
?
PDF
Tutorial on Deep Learning
inside-BigData.com
?
PDF
1???? GAN(Generative Adversarial Network) ?? ????
NAVER Engineering
?
PPTX
Understanding RNN and LSTM
宗殻 剋
?
PPTX
???? / ????? ????
Yongha Kim
?
PDF
Imagen: Photorealistic Text-to-Image Diffusion Models with Deep Language Unde...
Vitaly Bondar
?
PPTX
TensorFlow Tutorial | Deep Learning With TensorFlow | TensorFlow Tutorial For...
Simplilearn
?
フ`リエ篁擦隼R抹の碧怏み
yuichi takeda
?
Wasserstein GAN ?? ???? I
Sungbin Lim
?
???? ???? ????????, ???? ??? ??????.
Yongho Ha
?
NLPにおけるAttention゛Seq2Seq から BERTまで゛
Takuya Ono
?
Attention Is All You Need
Illia Polosukhin
?
Deep Learning - RNN and CNN
Pradnya Saval
?
????? ???????? ?????? 1??? (2022??? ??????)
ssusercdf17c
?
＾An Introduction to Data Augmentation Techniques in ML Frameworks,￣ a Present...
Edge AI and Vision Alliance
?
CNN Machine learning DeepLearning
Abhishek Sharma
?
[DLi氏]Hindsight Experience Replay
Deep Learning JP
?
Transformer in Vision
Sangmin Woo
?
Attention is All You Need (Transformer)
Jeong-Gwan Lee
?
Transformers in Vision: From Zero to Hero
Bill Liu
?
?????? ?? ?
NAVER Engineering
?
Tutorial on Deep Learning
inside-BigData.com
?
1???? GAN(Generative Adversarial Network) ?? ????
NAVER Engineering
?
Understanding RNN and LSTM
宗殻 剋
?
???? / ????? ????
Yongha Kim
?
Imagen: Photorealistic Text-to-Image Diffusion Models with Deep Language Unde...
Vitaly Bondar
?
TensorFlow Tutorial | Deep Learning With TensorFlow | TensorFlow Tutorial For...
Simplilearn
?

Similar to [????] ??? ??? EEG ?? ?? (20)

PDF
[GDG DevFest Gwangju 2018] ?????? ???: ??/ IoT/ ????/ ???/ ?????? ????
Donghyeon Kim
?
PPTX
iT Cafe - Deep Learning
Dongmin Kim
?
PDF
Data Augmentation of Wearable Sensor Data for Parkinson¨s
Woodam Lim
?
PPTX
Anomaly detection practive_using_deep_learning
?? ?
?
PDF
Technical Report on "Neuromarketing: Graph Transformer-based Consumer Prefere...
Network Science Lab, The Catholic University of Korea
?
PDF
Ai study - 2 layer feed-forward network, Backpropagation
DaeHeeKim31
?
PDF
4???????? ?????
Namkug Kim
?
PDF
Neural network? ??? ???
Tae Young Lee
?
PDF
????? ?? ???
SK(?) C&C - ???
?
PDF
Neural stream
Mk Kim
?
PDF
4?????? ????
Namkug Kim
?
PDF
?????(Recurrent neural networks) ??
Byoung-Hee Kim
?
PPTX
Deep Learning for AI (1)
Dongheon Lee
?
PDF
???????????????
jaypi Ko
?
PPTX
Cnn ??
?? ?
?
PPTX
????, ???? ??? ???
Jinwon Lee
?
PDF
Pycon Korea 2018 ?????? ?? ????
Park Sejin
?
PDF
?? ?? ????_?_????_??
?? ?
?
PDF
Lecture 4: Neural Networks I
Sang Jun Lee
?
PDF
?? ????? ???? ????
Seonghyun Kim
?
[GDG DevFest Gwangju 2018] ?????? ???: ??/ IoT/ ????/ ???/ ?????? ????
Donghyeon Kim
?
iT Cafe - Deep Learning
Dongmin Kim
?
Data Augmentation of Wearable Sensor Data for Parkinson¨s
Woodam Lim
?
Anomaly detection practive_using_deep_learning
?? ?
?
Technical Report on "Neuromarketing: Graph Transformer-based Consumer Prefere...
Network Science Lab, The Catholic University of Korea
?
Ai study - 2 layer feed-forward network, Backpropagation
DaeHeeKim31
?
4???????? ?????
Namkug Kim
?
Neural network? ??? ???
Tae Young Lee
?
????? ?? ???
SK(?) C&C - ???
?
Neural stream
Mk Kim
?
4?????? ????
Namkug Kim
?
?????(Recurrent neural networks) ??
Byoung-Hee Kim
?
Deep Learning for AI (1)
Dongheon Lee
?
???????????????
jaypi Ko
?
Cnn ??
?? ?
?
????, ???? ??? ???
Jinwon Lee
?
Pycon Korea 2018 ?????? ?? ????
Park Sejin
?
?? ?? ????_?_????_??
?? ?
?
Lecture 4: Neural Networks I
Sang Jun Lee
?
?? ????? ???? ????
Seonghyun Kim
?
Ad

[????] ??? ??? EEG ?? ??

  • 1. Introduction to EEG-based Systems with Deep Learning 2018.04.04. ??? (dhkim518@gist.ac.kr)
  • 2. ? Background (??? ?? ??) ? ??? ?? & ?-??? ????? ? ?-??? ?????: ???? 4?? ???? ?? & 7? ?? ?? ?? I. Resting State EEG ? [I-1] Seizure Detection ? [I-2] Gender Prediction II. Mental-task Imagery ? [II-1] Motor Imagery (Left/ Right) III. Steady-state Evoked Potential (SSEP) ? [III-1] SSVEP Classification IV. Event Related Potential (ERP) ? [IV-1] P300 Speller ? [IV-2] Transferring Human Visual Capabilities to Machine ? [IV-3] EEG-in/ Image-out ?? ?? 2
  • 3. ? ??? ?? ?? ???? ???? ????? ????? ? ?? ??? ???? ??? ?? ????? ???? ?? ? ??? ? ｀??? ??? ?? ??? ???/ ?? ????? ??? ?????¨ ? ??? ?? ?? ?? ??? ? ???? ??? ??? ????? ? ?? ??? ???? ?? ??? ?? ????? ?????? ????? ? ?????? ??? ??? ??? ｀?? ????? ???? ??? ???? ?? ?? ?? ???¨ ??? ? ?? ??, ????, BCI, ??? ?? ?? ??? ?? ? ?? ? CNN, LSTM, GAN ?? ??? ??? ?? ?? ? Temporal, spectral, spatial feature? ??? ?? ? ?? ? ????? 3
  • 4. ? ??? (Electroencephalogram, EEG) ? ? ?? ??? ??? ??? ? ??? ??? ??, ???? ??? ?? ??? ?? ? 1929? Hans Berger? ?? ???? ?? ???? ????? [Berger, ｀29] ? ?-??? ?????(Brain-computer interface, BCI)? ?? ???? ???? ?? ? BCI: ??? ???? ???? ??, ?? ??? ???? ??? ????? ? ??? ??? ?? 3??; ??, ???, ?? Background 4 ?? ??? ??? ??? ??, ?? ?? ??? ?? ??? ??? ?? ?? ??? δ (0.1-4 Hz), θ (4-8 Hz), α (8-13 Hz), β (13-30 Hz), γ (30-50 Hz) ?? ?? ?? ?? ???? ????? ? ?? ?? [Berger, ｀29] Berger, H. (1929). Ueber das Elektroenkephalogramm des Menschen. Archiv f┨r Psychiatrie und Nervenkrankheiten, 87(1), 527-570. International 10-20 System
  • 5. ? ???? ?? ?? ??? ?? ? ???? ??? ?? ?? ? I. Resting state EEG ? II. Mental-task Imagery ? III. Steady-state Evoked Potential (SSEP) ? IV. Event Related Potential (ERP) Background 5 Signal ProcessingSignal Acquisition Applications ? Feature extraction/ pattern recognition ? Machine learning based approaches ? Brainwave ? Entertainments (Game) ? Therapy/ healthcare ? Prostheses ? Authentication
  • 6. ? Resting EEG ? ??? ?? ????? ??? ?? ? ?? EO (Eye-open)?? EC (Eye-closed) ??? ??? ???? ?? ?? ? ???? ?? ? [I-1] Seizure Detection ? [I-2] Gender Prediction I. Resting State EEG 6 . . . .
  • 7. ? ?? ?? ? Seizure: ??, ?? ????? ???? ??? ????? ?? ?? ?? ? Epilepsy: ??, ??? ??? ??? ??? ?? ???? ?? [I-1] Seizure Detection 7 Acharya, U. R., Oh, S. L., Hagiwara, Y., Tan, J. H., & Adeli, H. (2017). Deep convolutional neural network for the automated detection and diagnosis of seizure using EEG signals. Computers in biology and medicine.
  • 8. ? Motivation & Objective ? EEG? ?? ??? ?? ??? ??? ?? ?? ??? ???? ???? ?? ? ???? ?? (unprovoked seizure)? 2? ?? ?? ? ??? ???? ? ??? ?? ??: ｀direct visual inspection¨ ? ?? ??? ??, ??? ?? ｀time-consuming¨, ｀technical artifact¨, ｀variable results¨, ｀limitation to identifying abnormality¨ ? ??? Computer-aided diagnosis (CAD) ???? ?? ? 13-layer Convolutional Neural Network ? Normal, Pre-ictal (?? ?? ??), Seizure 3?? ??? ?? ??? ???? [I-1] Seizure Detection 8
  • 9. ? Dataset ? Public dataset [Andrzejak, ｀01] ? 3 states: Normal, Pre-ictal, and Seizure ? 100 segments for each class ? ??? EEG segment ?? ? Single channel EEG signals ? 23.6 sec, Ns = 4097 (Ns: ?? ?) ? Z-score normalization (mean 0, std. 1) ? Architecture ? 13-layer CNN [I-1] Seizure Detection 9 [Andrzejak ¨01] R.G. Andrzejak, K. Lehnertz, C. Rieke, F. Mormann, P. David, C.E. Elger, Indications of nonlinear deterministic and finite dimensional structures in time series of brain electrical activity: dependence on recording region and brain state, Phys. Rev. E 64 (2001) 061907. Color information - Kernel size - Max-pooling - Fully connected layer
  • 10. ? Experimental Validation ? 10-fold cross-validation ? EEG signals? 10 ???? ???, 9 ?? train ? ?? 1 ?? test ?? ??? 10? ?? ? Train ???? 70 % ???? ???? 150 runs* ???? (batch size: 3), 30 % ???? validation? ?? (overfitting ?? ??) ? Results & Discussion ? Accuracy: 88.67%, Sensitivity: 95.00%, and Specificity: 90.00% ? ?? ??? ?? ???? ??? ???? ??? ? ? ??? ｀Novelty¨? EEG ??? Seizure detection? ???? ???? ??? ? ? ? ?? ??? ?? ???? ? ?? ??? ?? ??? ?? [I-1] Seizure Detection 10 *Run: one iteration of the full training set
  • 11. ? Motivation & Objective ? ????? ???? ????? ??? ????. ? ??? ??? ?? ?????, ?????? ???? ??????? ???? ?? ? ?? ??? ?? ?? ??? ??? ?? ? ??? ｀sex-specific information¨ ? ??? ???? ?????, ?????? ??? ?? ??? visual inspection/ quantitative inspection ??? ?? ?? ? ??: ??? ??? ?? ??? ?? ???? ??? ????? ???? ?? ?? [I-2] Gender Prediction 11 Putten, M. J., Olbrich, S., & Arns, M. (2018). Predicting sex from brain rhythms with deep learning. Scientific reports, 8(1), 3069.
  • 12. ? Dataset ? EEG data from 6 different Labs* ? 1308 subjects (mean age 43.38 (18.42 SD) y; range 18C98 y; 47% males) ? Eye-closed (EC) Resting EEG 2 min. ? ??? EEG segment ?? ? 24 channels/ 2 sec, Fs = 128 Hz, Ns = 256 (?? ?)/ 1 ???? 40 EEG segments ?? ? Band-pass filtering [0.5, 25] ? Architecture ? 6-layer Convolutional Neural Network [I-2] Gender Prediction 12 *Brain Resource International Database (New York, Rhode Island, Nijmegen, London, Adelaide and Sydney).
  • 13. ? Experimental Validation ? ??? ?? ??? ?? ???? cross-validation? ?? ?? ? 1000? ???? ??? (40,000 EEG segments, 47% male)? train??, 308? ???? ??? (12,320 EEG segments, 49% male)? test ? ?? ?? 150 runs* ?? (batch size: 70) ? Results & Discussion ? Accuracy: 81 % (p < 10-5) ? ???? ?? ?, ? ?? ??? ??? (?? layer, ? ? filter size, ?/?? ?? ?) ? Visualization of six convolutional layers ? Deep dream algorithm ?? [Mahendran, ¨15][Mahendran, ¨16] ? ??? CNN ???? ??? (12-25 Hz) ?? ?? ? ??? ??? logistic regression model ?? 70 % accuracy [I-2] Gender Prediction 13 [Mahendran, ¨15] Mahendran, A. & Vedaldi, A. Understanding deep image representations by inverting them. in IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015). [Mahendran, ｀16] Mahendran, A. & Vedaldi, A. Visualizing Deep Convolutional Neural Networks Using Natural Pre-images. Int. J. Comput. Vis. 120, 233C255 (2016).
  • 14. ? Mental-task Imagery ? ???? ???? ???? ?? ?? ? ?? ?? ?? (motor imagery), ? ?? (speech imagery) ?? ?? ?? ? ???? ?? ? [II-1] Motor imagery: Imagination of moving a body part without actual movement II. Mental-task Imagery 14
  • 15. ? Motivation & Objective ? BCI ???? ?? ?? ?? ??? ????? ????? ??? ??? ? ????? ????? ｀Feature Extraction¨ ??? ｀Classification¨ ??? ??? ??? ?? ?? ?? ? ????? ? Common Spatial Pattern (CSP), Principal Component Analysis (PCA), Linear Discriminant Analysis (LDA), Support Vector Machine (SVM) ?? ?????? ? ?? training data ? ?? ???? ???? ?? ??? ?? ?? ?????, ?? ???? ???? BCI ???? ?????? ????? ???? ??? ??? ? ??? ? ??? ???? ???? ??? ??? ? ? ????? ??? ??? ?? ??? ?? ??? ?? ??? ?? ?? [II-1] Motor Imagery 15 Tabar, Y. R., & Halici, U. (2016). A novel deep learning approach for classification of EEG motor imagery signals. Journal of neural engineering, 14(1), 016003.
  • 16. ? Dataset ? Public dataset (BCI Competition IV dataset 2b) ? 9? ???? ???? ?? ?? ??? ?? ?? ? ??? EEG segment ?? ? 3 channels (C3, Cz, C4) ? 2 sec (cue? ??? ? 0.5~2.5? ?? ??? ??), Fs = 250 Hz ? 1 ???? 400 EEG segments ?? ? Input image form ? ?? (time), ??? (frequency), ?? (channel location) ??? ??? ??? ?? ? ? Short-time Fourier transform (STFT) ?? (window size: 64, time lapses = 14) ? 1 channel EEG segment (Ns = 500) ? STFT ? frequency-time image (257 x 32) ? ??? ??? ????? ???? mu ??(8-13 Hz) ? beta ??(17-30 Hz) ? ?? ? 3?? (C3, Cz, C4) ??? ?? Frequency-Time images (size of 31 x 32) ? ?? input ? 3??? ?? ??? ? ???? ?? (size of 93 x 32) [II-1] Motor Imagery 16
  • 17. ? Architecture ? Convolutional Neural Network (CNN) ? ?? ???? Convolution ?? ? ?? 30? ??, ? ?? ???? (93 x 3) ? ?? ? Max-pooling? (10 x 1) ? ?? ?? ? Batch size = 50?? 300 runs ?? ?? [II-1] Motor Imagery 17
  • 18. ? Architecture ? Convolutional Neural Network (CNN) ? Stacked Auto Encoder (SAE) ? 6 ?? ??? AE? ??? ???? ? SAE? ???? ??? AE ?? unsupervised ??? ?? ????? ???? ? ??? AE? Batch size = 20? 200 runs ?? ??? ?, Fine tuning ???? batch size = 40? 200 runs ?? ???? [II-1] Motor Imagery 18
  • 19. ? Architecture ? Convolutional Neural Network (CNN) ? Stacked Auto Encoder (SAE) ? Combined CNN-SAE ? CNN?? ??? convolutional layer ??? SAE? ??? ?? [II-1] Motor Imagery 19
  • 20. ? Architecture ? Convolutional Neural Network (CNN) ? ?? ???? Convolution ?? ? ?? 30? ??, ? ?? ???? (93 x 3) ? ?? ? Max-pooling? (10 x 1) ? ?? ?? ? Batch size = 50?? 300 runs ?? ?? ? Stacked Auto Encoder (SAE) ? 6 ?? ??? AE? ??? ???? ? SAE? ???? ??? AE ?? unsupervised ??? ?? ????? ???? ? ??? AE? Batch size = 20? 200 runs ?? ??? ?, Fine tuning ???? batch size = 40? 200 runs ?? ???? ? Combined CNN-SAE ? CNN?? ??? convolutional layer ??? SAE? ??? ?? [II-1] Motor Imagery 20
  • 21. ? Experimental Validation ? 10 x 10 Fold Cross-validation ? ? ???? 400 Trials? Random?? 10 ???? ???, 9 ?? train ? ?? 1 ?? test ?? ??? ?? ??? ??? 10? ?? ? Random ?? 10 ???? ??? ???? ??? ??? 10? ?? ? Results & Discussion ? CNN-SAE ? ?? ?? ??? ????, ??? ????? ?? ??? ???. ? CNN?? ??? ?? 30? ?? ? ??? Feature? ?? ??, ?? ??? ?? ? ??? ???? ?? ?? ?? ???? [II-1] Motor Imagery 21
  • 22. ? Steady-state Evoked Potential (SSEP) ? ??/ ??/ ?? ??? ????? ??? ? ???? ?? ? ?? ??? ???? ??? ?? ?? ? ???? ?? ? [III-1] SSVEP Classification III. Steady-state Evoked Potential 22 Flickering Light (20 Hz) Oscillatory Response (20 Hz)
  • 23. [III-1] SSVEP Classification 23 Kwak, N. S., M┨ller, K. R., & Lee, S. W. (2017). A convolutional neural network for steady state visual evoked potential classification under ambulatory environment. PloS one, 12(2), e0172578.
  • 24. ? Dataset ? 7? ???? ???? ?? ?? ? ???? ? Random auditory cue? ???? 3? ?? start beep? ??? ???? 5?? ?? LED ?? ? Cue: ?? 9 Hz, ??? 11 Hz, ???? 13 Hz, ??? 15 Hz, ?? 17 Hz (5 classes) ? Task 1 (Static SSVEP): ??? ??? ???? cue? ?? (? class ? 10??, ? 50?) ? Task 2 (Ambulatory SSVEP): ??? ???? ??? cue ? ?? (? class ? 50??, ? 250 ?) ? ??? EEG segment ?? ? 8 channels (?? ?? ???) ? BPF [4 - 40 Hz] ??? EEG ??? 2 sec (Fs = 1000 Hz) ??? segment? ?? ? ??? ?? ? ??? FFT ? [5 - 35 Hz]? ???? 120 samples ?? ? 0-1 Normalization ? ? EEG segment? size? (120 x 8) ? ?? ?? ??? ???? ?? ??? ?? ? 2 sec sliding window with different shift sizes (60, 30, 20, 15, 12, 10 msec) ? ??? ? (50, 100, 150, 200, 250, 300 data samples) [III-1] SSVEP Classification 24 ´ Shift size
  • 25. ? Architecture ? CNN-1 ? [C1] Convolution? ????? 8? channels ???? ?? ? [C2] Convolution? ?????? 11 frequency samples ???? ?? [III-1] SSVEP Classification 25
  • 26. ? Architecture ? CNN-1 ? CNN-2 ? CNN-1? ?? ?? ?????, F3 layer? 3? ??? ???? Visualization ??? ?? [III-1] SSVEP Classification 26
  • 27. ? Architecture ? CNN-1 ? CNN-2 ? NN [III-1] SSVEP Classification 27
  • 28. ? Architecture ? CNN-1/ CNN-2/ NN ? Experimental Validation ? ? ??? ?? 2?? static/ ambulatory SSVEP ??? ??? 10-fold CV ?? ? ?? ?? ??? ???? ?? ??? ?? ? Learning rate = 0.1 / Learning iterations = 50? (10? ?? ??? error rate? 0.5 % ??? ???? ?? ??) [III-1] SSVEP Classification 28
  • 29. ? Results & Discussion ? static/ ambulatory ???? ??? ???? ?? ?, ??? ???? ??? ?? ?? ? Feature Representation (CNN-2 ?? ?? ??) [IV-1] SSVEP Classification 29
  • 30. ? Event Related Potential (ERP) ? ??/ ??/ ?? ?? ??? ???? ? ?? ? ??????, P300 ??? ??? ??? ??? ?? ?? ? ???? ?? ? [IV-1] P300 Speller ? [IV-2] Transferring Human Visual Capabilities to Machine ? [IV-3] EEG-in/ Image-out IV. Event Related Potential 30
  • 31. ? P300 Speller? ?? ? ??? ?? (a) P300 detection: ??? ?? ?? P300 ??? ?? ??? (b) Character Recognition: ???? ?? ??? ???? ??? [IV-1] P300 Speller 31 Cecotti, H., & Graser, A. (2011). Convolutional neural networks for P300 detection with application to brain-computer interfaces. IEEE transactions on pattern analysis and machine intelligence, 33(3), 433-445.
  • 32. ? Dataset ? Public dataset (Data set II from the third BCI competition) ? 2? ???? ???? P300 Speller ?? ?? ? Training database: ? 85 characters / Test database: ? 100 characters ? 12? (6 rows, 6 columns) ?? ?????, ? 12? ???? 15? ?? ? ??? EEG segment ?? ? 64 channels ? 650 msec (cue? ??? ? 0 ~ 650 msec ??? ??), Fs = 120 Hz ? ? ?? ? ?? Standardization ? Architecture ? [L0] Input size: 64 Ch. x 78 Ns ? [L1] Convolution ? (1x64) ??? ?? 10? ? 10@78x1 ? [L2] Convolution + Subsampling ? (13x1) ??? ?? 5? ? 50@6x1 ? [L3, L4] Fully connected ? 300 nodes C 100 nodes C 2 output [IV-1] P300 Speller 32
  • 33. ? Experimental Validation ? ? ??? ?? ??? training dataset ?? ????, test dataset?? ??? ?? ? Training dataset ? 95 % training, 5 %? validation ? Validation ???? ?? least mean square error ? ???? ??? ?? ?? (CNN ??? ??? variation ? ? ???? ??? ??? ??? ?????) ? Results & Discussion ? ??? ???? ???? ??? ?? ?? ? (A) P300 Detection (B) Character Recognition [IV-1] P300 Speller 33
  • 34. ? CNN? ? ??? ??? ???, ??? ??? visual object classifier ??? [IV-2] Transferring Human Visual Capabilities to Machine 34 Spampinato, C., Palazzo, S., Kavasidis, I., Giordano, D., Souly, N., & Shah, M. (2017, July). Deep learning human mind for automated visual classification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 6809-6817).
  • 35. ? Objective & Contribution ? ?? ??? ?? ??? ??? ?? ??? ERP ??? ??? ? ?? ??? ?? ? ERP ?? ????? ?? visual descriptor? ??? computer vision ??? ?? ? ? ??? EEG ????? ?? ???? ? trained model ? publicly release ?? [IV-2] Transferring Human Visual Capabilities to Machine 35
  • 36. Part 1. ?? ??? ??? ?? ERP ??? ? Dataset ? 6? ???? ???? ?? ?? ? ???? ? ImageNet ???? 40 classes 2,000 images (50 from each class) ?? ? 10 ? ???? ??? ???? 0.5 ?? ???? ???? EEG Recording ? ??? EEG segment ?? ? 128 channels ? BPF [14 - 71 Hz] ??? EEG ??? 440 msec (Fs = 1000 Hz) ?? ? Beta (15-31 Hz), Gamma (32-70 Hz) ??? ?? ?? ? cognitive process? ?? ? ??? ??? ?? 0.5 ? ??? ?? ?? 40 msec ??? 440 msec ??? ?? ? ? EEG segment? size? (128 x 440) [IV-2] Transferring Human Visual Capabilities to Machine 36
  • 37. Part 1. ?? ??? ??? ?? ERP ??? ? Architecture ? a) Common LSTM ? Common? ??: ?? EEG ??(128?)? ???? ?? LSTM layer? ???? ?? [IV-2] Transferring Human Visual Capabilities to Machine 37
  • 38. Part 1. ?? ??? ??? ?? ERP ??? ? Architecture ? a) Common LSTM ? b) Channel LSTM + Common LSTM ? ??? single EEG ???? LSTM ??? ????, ??? inter-channel analysis ? ???? ?? common LSTM ? ??? ?? [IV-2] Transferring Human Visual Capabilities to Machine 38
  • 39. Part 1. ?? ??? ??? ?? ERP ??? ? Architecture ? a) Common LSTM ? b) Channel LSTM + Common LSTM ? c) Common LSTM + output layer ? ?? ??? a) common LSTM? ?????, ? ??? output layer? LSTM? ?? input?? ?? ?? (ReLU) ?? ???? ?? [IV-2] Transferring Human Visual Capabilities to Machine 39
  • 40. Part 1. ?? ??? ??? ?? ERP ??? ? Experimental Validation ? Training 80 % (1,600 images), validation 10 % (200), test 10 % (200) ? Results & Discussion [IV-2] Transferring Human Visual Capabilities to Machine 40
  • 41. Part 2. ???? EEG ?? Feature? ???? CNN-based Regression ? ?? ?? 2?? ? Approach 1. End-to-end training ? Pre-trained AlexNet CNN? weights? initialization? ?? ? GoogleNet, VGG? ?? ??? ??; ?? ????? ????? ???? ?? ? ? ? ??? layer? ?? softmax ???? regression layer? ?? ? Regression layer ?? ?? ?? EEG feature vector dimension? ?? ?? ? ?? ??? Euclidean loss? ?? ?? ?? [IV-2] Transferring Human Visual Capabilities to Machine 41
  • 42. Part 2. ???? EEG ?? Feature? ???? CNN-based Regression ? ?? ?? 2?? ? Approach 2. Deep feature extraction followed by regressor training ? Pre-trained AlexNet, GoogleNet, VGG? weights? initialization? ?? ? ? ??? CNN ??? ???? ??? ??? ???? regression methods ? ?? ? k-NN regression, ridge regression, random forest regression ? CNN ???? ??? ??? ??? EEG feature vector? ?? [IV-2] Transferring Human Visual Capabilities to Machine 42
  • 43. Part 2. ???? EEG ?? Feature? ???? CNN-based Regression ? Experimental Validation ? ?? Part 1?? ?? ??? ?? 128 Common + 128 Output ? feature vector ?? ? ? 2000 ?? ???? ??, CNN? regression ?? ?? ｀confusing¨ ? ???? ??? ??? ????? ??? EEG feature vector? ???? ?? ? ? ?? ???? ??? ?? ? Average: 6 ? ???? ? ???? ?? ??? feature vector? average ?? ? Best: ??? ???? ???, 6 ? ???? EEG feature vectors ??? ?? ?? classification loss? ?? ?? feature vector ??? ?? ? Result & Discussion ? Mean-square error? ?? ?? ? Approach 2? [GoogleNet + k-NN regression + Average EEG feature vector]? ?? ?? [IV-2] Transferring Human Visual Capabilities to Machine 43
  • 44. Part 3. Automated Visual Classification ? Experimental Validation ? ?? Part 1, Part 2?? ?? ??? ?? ???? ?? ?? ? GoogleNet + k-NN + Average EEG feature vector + 128 Common, 128 Output ? ??? ??? ? ?? Test data (10 %, 200? ???)?? 89.7 % ?? ??? ? ???? ?? Caltech-101? 30 class? ?? ?? ?? ?? ? GoogleNet/ VGG/ ??? ??? feature extractor? ?? ? Multiclass SVM classifier? ?? ???? ?? ??? ?? ? Result & Discussion ? Human Brain-driven Automated Visual Classification ? GoogleNet? ????? ??? ????, VGG?? ??? ?? ?? ? EEG encoder? regressor? ???? ?? ?? ????? ??? ???? ???? ｀impressive¨ ?? ?? [IV-2] Transferring Human Visual Capabilities to Machine 44
  • 45. ? Read the Mind, Generate the Image ???? ???? ??? ? ????? ? ??? ??? ???? ????? [IV-3] EEG-in/ Image-out 45 Palazzo, S., Spampinato, C., Kavasidis, I., Giordano, D., & Shah, M. (2017, October). Generative Adversarial Networks Conditioned by Brain Signals. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 3410-3418).
  • 46. ? Objective & Contribution ? Generative model? ??? ??? ??? ? ?? ???? ?? ? ??? ???? ? ??? ???? ?? ??, ??? ??? ????? ? ?? ?? ? Condition Generative Adversarial Network (cGAN) [IV-3] EEG-in/ Image-out 46
  • 47. ? Architecture ? A. Learning EEG Manifold ? B. Image Generation ? B-1. Generator ? B-2. Discriminator [IV-3] EEG-in/ Image-out 47 Input G1 G2 G3 G4 G5 1 x 100 (Random Noise) + 1 x 128 (EEG Feature) 4 x 4 x 512 8 x 8 x 256 16 x 16 x 128 32 x 32 x 64 64 x 64 x 3 Input DC1 DC2 DC3 DC4 DFC5 DFC6 64 x 64 x 3 32 x 32 x 64 16 x 16 x 128 8 x 8 x 256 4 x 4 x 512 + 4 x 4 x 8 (EEG Feature) 1024 1
  • 48. ? Experimental Validation ? Condition GAN ????? ?????, ?? ??? ?? ?? (40 class, 50 images) ? ???, ? ??? ??? ?? ????? ? 1??: EEG feature? zero vector? ??, non-conditional GAN ??? 100 epochs ?? ?? ? 2??: EEG feature vector? ?? ???? conditional GAN ??? 50 epochs ???? ? ??? ???? ??: inception score ? inception classification accuracy ? Results & Discussion ? ??? ???? ?? inception network classification ?? 43 % ??? ?? ? ?? ???? ????, 40 class ? ?? 43 % ?? ????? ?? (1/40 = 2.5 %) [IV-3] EEG-in/ Image-out 48 Good Results: Jack-o¨-Lantern & Panda Bad results: Banana & Bolete
  • 49. ? ??? ??? ??? ???? ???? 4?? ????? 7?? ?? ?? Conclusive Remarks 49 ?? ???? ?? ??? ??? ?? EEG domain I. Resting State EEG 1. Seizure Detection 1-D 13-L CNN ?? 2. Gender Prediction 6-L CNN ??-?? II. Mental-task Imagery 3. Motor Imagery (L/ R) CNN + SAE ??-??-??? III. Steady-state Evoked Potential 4. SSVEP Classification CNN & Visualization using 3 nodes ???-?? IV. Event Related Potential 5. P300 Speller CNN ??-?? 6. Human Visual Capabilities to Machine LSTM & CNN-regression ??-?? 7. EEG-in/ Image-out LSTM & (conditional) GAN ??-??
About
? 2025 際際滷