�ݺ�ߣ

UNIVERSITY OF PALERMO
POLYTECHNIC SCHOOL
Departmentof Industrial and DigitalInnovation (DIID)
Computer ScienceEngineeringfor Intelligent Systems
Design and Implementation of Modules
for the Extraction of Biometric Parameters
in an Augmented BCI Framework
Master Degree Thesis of:
Salvatore La Bua
WWW.SLBLABS.COMMarch, 2017

Introduction
▪What
◦ Investigate the effects of the interaction with a robotic agent
on the mental status of the human player
through brain signal analysis
◦ Acceptance of a robotic agent by the user
◦ Performance improvements over a classical BCI system
▪How
◦ Rock-Paper-Scissors game integration
◦ UniPA BCI Framework based on the P300 paradigm
◦ Augmented by
◦ Eye gaze coordinate acquisition
◦ Biometric feature extraction
DESIGN AND IMPLEMENTATION OF MODULES FOR THE EXTRACTION OF
BIOMETRIC PARAMETERS IN AN AUGMENTED BCI FRAMEWORK
S. La Bua 2

Introduction
Human-Robot Interaction (HRI)
▪HRI as a multidisciplinary research topic
◦ Artificial Intelligence
◦ Human-Computer Interaction
◦ Natural Language Processing
◦ Social Sciences
◦ Design
▪Model of the user’s expectation towards a robotic agent
in a human-robot interaction
S. La Bua 3

Introduction
Brain-Computer Interfaces (BCI)
▪Direct communication between
brain and external devices
◦ Non-Invasive
◦ Partially-Invasive
◦ Invasive
▪Brain Lobes
◦ Frontal: emotions, social behaviour
◦ Temporal: speech, hearing recognition
◦ Parietal: sensory recognition
◦ Occipital: visual processing
▪Extraction of biometric features from brain signals
S. La Bua 4

Introduction
Visual Focus
▪Importance of eye gaze for direct interaction in a social
environment
▪Interfaces dedicated to people affected by degenerative
pathologies
▪Entertainment applications, such as games
▪Better advertisement placement
S. La Bua 5

Methodology
Background Information
▪Problem
◦ Effects of the behaviour of a robotic agent on the brain signals
◦ Trust context in Human-Robot Interaction
▪Feature Extraction
◦ Entropy: as a stress indicator
◦ Energy: as a concentration indicator
◦ Mental Workload: as an index of engagement in the task
▪Brain waves types
◦ δ Delta: Hz 0.5÷3 related to instinct, deep sleep
◦ θ Theta: Hz 3÷8 related to emotions
◦ α Alpha: Hz 8÷12 related to consciousness
◦ β Beta: Hz 12÷38 related to concentration, stress
◦ γ Gamma: Hz 38÷42 related to information processing
S. La Bua 6

Methodology
The math behind
Entropy:
𝐸 𝑠𝑖 = 𝑠𝑖
2
log (𝑠𝑖
2
); 𝐸 𝑠 = − σ𝑖 𝐸 𝑠𝑖
Energy:
𝐸𝑠 = ෍
𝑛=−∞
∞
𝑥(𝑛) 2
Mental Workload:
𝛽 𝑝𝑠𝑑
𝛼 𝑝𝑠𝑑 + 𝜃 𝑝𝑠𝑑
S. La Bua 7

The Proposed Solution
Architecture Structure
▪Action Selection
◦ Direct interface with the user
◦ Acquisition of bio-signals
◦ Acquisition of eye gaze coordinates
◦ Selection of the Base action
▪Feature Extraction and Analysis
◦ Bio-signals analysis
◦ Features extraction
◦ Features analysis
◦ Computation of Intention, Attention,
Stress indices
▪Response Modulation
◦ Threshold of the Base action by means of the Intention index
◦ Modulation of the resulting action by means of Attention and Stress indices
S. La Bua 8

Class Diagram
S. La Bua 9

Functional Blocks
Action Selection
◦ Eye-Tracking module
◦ Screen coordinates acquisition
◦ Weighing module
◦ Weighing of the BCI classifier response
precision and the Eye-Tracking module
response precision, by means of the
user’s skill level
◦ ID Selection module
◦ Action selection by means of the weighted BCI classifier and Eye-Tracking module
precisions
S. La Bua 10

Functional Blocks
Feature Extraction
and Analysis
◦ It makes use of external calls
to the MATLAB engine
◦ Features extracted and analysed
◦ Correlation Factor: related to the Intention index
◦ Energy: related to the Attention index
◦ Entropy: related to the Stress index
S. La Bua 11

Functional Blocks
Response Modulation
◦ Threshold module
◦ ID Selection validation by
means of Intention index
thresholding
◦ Modulation module
◦ In the case the selected ID has passed the validation step,
the resulting action is modulated by means of the Attention and Stress indices
S. La Bua 12

Robotic Controller
S. La Bua 13

Utilisation Modes
Basic Mode
◦ Simplest mode
◦ Minimal number of
modules involved
◦ Classical BCI approach
◦ P300 paradigm
classification
◦ Direct Behaviour
S. La Bua 14

Utilisation Modes
Hybrid Mode
◦ Advanced mode
◦ Eye-Tracking module
◦ Combination of brain
signals and eye gaze
◦ User skill level as
weighting parameter
◦ Composite Behaviour
S. La Bua 15

Utilisation Modes
Bio-Hybrid Mode
◦ Complete mode
◦ Feature Extraction
and Analysis
functional block
◦ Response Modulation
functional block
◦ Intention, Attention and
Stress indices computation
◦ Modulated Behaviour
S. La Bua 16

Architecture
Eye-Tracking module
P300 6x6 spelling matrix 3x3 spelling window areas
S. La Bua 17

Architecture
Eye-Tracking module
Preliminary tests results
SUBCATEGORIES FOR SINGLE
ELEMENT
FOCUS % CENTRAL FOCUS % LATERAL FOCUS % EXTERNAL FOCUS %
3-BY-3, 700X700PX 100 99.9000 0.1000 0
3-BY-3, 300X300PX 98.4562 93.2697 6.7303 1.5438
6-BY-6, 700X700PX 100 84.7408 2.7592 0
6-BY-6, 300X300PX 99.5997 75.9943 24.0057 0.4003
SUBCATEGORIES FOR ROW SPAN
SELECTION
FOCUS % CENTRAL FOCUS % LATERAL FOCUS % EXTERNAL FOCUS %
3-BY-3, 700X700PX 74.2632 93.9192 6.0808 25.7368
3-BY-3, 300X300PX 77.1340 89.9075 10.0925 22.8660
6-BY-6, 700X700PX 69.5037 96.3287 3.6713 30.4963
6-BY-6, 300X300PX 75.0674 71.7202 28.2798 24.9326
AVERAGE BY PARAMETER FOCUS % CENTRAL FOCUS %
700X700PX 85.9417 93.7222
300X300PX 87.5643 82.7229
GAIN WITH LARGER WINDOW -1.8530% +13.2966%
AVERAGE BY PARAMETER FOCUS % CENTRAL FOCUS %
3-BY-3 87.4634 94.2491
6-BY-6 86.0427 82.1960
GAIN WITH LESS DENSE MATRIX +1.6512% +14.6639%
S. La Bua 18

Architecture
Data Structures
Generic signal data structure fields
N fields dedicated to the brain signals acquisition
◦ Ch 1 – Ch 16
3 auxiliary fields to carry peculiar information
◦ A, B, C
CH 1 CH 2 · · · CH N A B C
S. La Bua 19

Architecture
Data Structures
Baseline Calibration signal
A (RED) B (CYAN) C (MAGENTA)
BASELINE CALIBRATION -2 EYES STATUS 0
S. La Bua 20

Architecture
Data Structures
Game Session signal
A (RED) B (CYAN) C (MAGENTA)
GAME SESSION TRIAL STATUS TRIAL SUB-PHASE GAZE TRACKING
S. La Bua 21

Architecture
Data Structures
P300 Calibration signal
P300 Spelling signal
A B C
P300 Spelling -1 Flashing tag 0
A B C
P300 Calibration Calibration target Flashing tag 0
S. La Bua 22

The Framework
Main Interface
1. Basic settings
◦ P300-related settings
◦ Preset modes
2. Main functionalities
◦ Signal quality check
◦ P300 Calibration and
Recognition
◦ Game session control
3. Interface modality
◦ Alphabetic or Symbolic
4. Devices
◦ Eye-Tracker settings
5. Plots and Indicators
◦ Signals and Indices
visualisation
6. Output panel
◦ Feedback for the operator
1
2
3
4
5
6
S. La Bua 24

The Framework
Baseline Acquisition Interface
Control dialog window User dialog window
S. La Bua 25

The Framework
Game Session Interface
1. Game modality
◦ Fair
◦ Cheat-to-Win/Lose
2. Trials number per session
◦ Initial Fair sub-session
◦ Middle Cheating sub-session
◦ Terminal Fair sub-session
3. Devices
◦ BCI signal acquisition
◦ Kinect gesture recognition
◦ Play against a robotic agent
4. Session panel
◦ Moves selection
◦ Trial temporal progress
1 2
3
4
S. La Bua 27

Experiments
Introduction
▪Purpose
◦ Investigate the effects of the interaction with a cheating robotic
agent on the mental status of the human player
◦ Rock-Paper-Scissors game session
▪Scenarios
◦ The robot behaves according to the game’s rules
◦ The robot exhibits a cheat-to-win behaviour
◦ The robot exhibits a cheat-to-lose behaviour
▪Game Session
Initial Fair
sub-session
Cheating
sub-session
Terminal Fair
sub-session
S. La Bua 28

Experiments
Set-up
Subjects
◦ 16 Subjects
◦ Aged 18-51
Hardware
◦ g.tec g.USBamp
◦ g.tec g.GAMMAbox
◦ g.tec g.GAMMAcap2
◦ Secondary standard PC screen
◦ Tobii EyeX eye tracker
◦ Kinect for Xbox One
◦ Telenoid
◦ Camera(s)
S. La Bua 29

Experiments
EEG Electrodes configuration
Channels-Electrodes
correspondence
L R
Ch 01 F7
Ch 02 F3
Ch 03 FZ
Ch 04 T3
Ch 05 C3
Ch 06 T5
Ch 07 P3
Ch 08 O1
Ch 09 F8
Ch 10 F4
Ch 11 T4
Ch 12 C4
Ch 13 T6
Ch 14 P4
Ch 15 PZ
Ch 16 O2
S. La Bua 30

Experiments
Protocol
S. La Bua 31

Experiments
Protocol
S. La Bua 32

Experiments
Subcategories
▪Sub-Session Analysis
◦ Analysis of the Baseline signal, Fair and Cheating sub-sessions
▪Trials Analysis
◦ Single trial analysis for each subject
▪Intra-Class Comparison
◦ Comparison between Cheat-to-Win and Cheat-to-Lose classes
▪Average Analysis
◦ Average over all subjects, by class and by sub-sessions
S. La Bua 33

Experiments
Sub-Session Analysis
Entropy
Energy
S. La Bua 34

Experiments
Sub-Session Analysis
Mental Workload Visual Focus %
S. La Bua 35

Experiments
Trials Analysis
Summary:
Entropy
Energy
Mental Workload
Visual Focus %
S. La Bua 36

Experiments
Trials Analysis
Entropy: Cheat-to-Win Cheat-to-Lose
S. La Bua 37

Experiments
Trials Analysis
Energy: Cheat-to-Win Cheat-to-Lose
S. La Bua 38

Experiments
Trials Analysis
Workload: Cheat-to-Win Cheat-to-Lose
S. La Bua 39

Experiments
Trials Analysis
Focus %: Cheat-to-Win Cheat-to-Lose
S. La Bua 40

Experiments
Intra-Class Comparison
Entropy:
Cheat-to-Win
Cheat-to-Lose
S. La Bua 41

Experiments
Energy:
Cheat-to-Win
Cheat-to-Lose
S. La Bua 42

Experiments
Mental
Workload:
Cheat-to-Win
Cheat-to-Lose
S. La Bua 43

Experiments
Visual Focus
percentage:
Cheat-to-Win
Cheat-to-Lose
S. La Bua 44

Experiments
Average Analysis
Entropy
The entropy values do not show any particular evidence of stress
ENTROPY FAIR 1 CHEAT FAIR 2
MEAN STD DEV MEAN STD DEV MEAN STD DEV
CHEAT WIN 3.8584 0.2191 3.8998 0.2540 3.8742 0.1891
CHEAT LOSE 3.7420 0.0850 3.7632 0.1177 3.7304 0.1074
S. La Bua 45

Experiments
Average Analysis
Energy
The energy values show higher concentration level for the Cheat-to-Win class
ENERGY FAIR 1 CHEAT FAIR 2
CHEAT WIN 0.2572 0.2141 0.3032 0.2267 0.2254 0.1951
CHEAT LOSE 0.1498 0.0596 0.1720 0.0948 0.1143 0.0447
S. La Bua 46

Experiments
Average Analysis
Mental Workload
The mental workload values show a slightly lower engagement level for the
Cheat-to-Win class
MENTAL WL FAIR 1 CHEAT FAIR 2
CHEAT WIN 1.3798 1.1625 0.8988 0.4215 0.9437 0.4570
CHEAT LOSE 1.0923 0.2716 1.0382 0.3229 1.0777 0.3936
S. La Bua 47

Experiments
Average Analysis
Visual Focus
The visual focus values show higher visual attention level for the Cheat-to-Win
class
FOCUS % FAIR 1 CHEAT FAIR 2
CHEAT WIN 7.89100 8.93670 9.13020 11.3344 12.1404 20.1567
CHEAT LOSE 4.59710 9.91690 3.24540 7.09430 2.20110 4.79480
S. La Bua 48

Experiments
Demo
S. La Bua 49

Conclusions and Future Works
▪A robotic agent that cheats to win is perceived as more
agentic and human-like than a robot that cheats to lose
▪Some of the Questionnaire results
▪Trust related improvement
◦ Biometric features to mitigate or amplify the effects of the
robotic agent behaviour on the subject’s emotional response
Unusual Behaviour Fair Play Intelligence
Strongly
Disagree
Strongly
Agree
S. La Bua 50

Future Works
Framework Extension
Sensor Aggregation functional block
◦ Galvanic Skin Response (GSR) sensor
◦ Heart Rate (HR) sensor
◦ Other physiological sensors
S. La Bua 51

Future Works
Extended Framework
S. La Bua 52

Thank you for
your attention
Salvatore La Bua
slabua@gmail.com
WWW.SLBLABS.COM

�ݺ�ߣ

Design and Implementation of Modules for the Extraction of Biometric Parameters in an Augmented BCI Framework

More Related Content

Design and Implementation of Modules for the Extraction of Biometric Parameters in an Augmented BCI Framework