�ݺ�ߣ

www.maynoothuniversity.ie
Robotics
Augmenting Human Performance
– robots, sensors and learning theory
Prof. Gerry Lacey,
Dept. of Electronic Engineering
28 Oct, 2021
Robotics

Robotics
Technology + Humans

Robotics
Case Studies in Human Machine Systems
1. Smart Walking frame for the Frail Blind
2. A Mixed Reality Minimally Invasive Surgical Simulator
3. Colonoscopy Quality Measurement
4. Hand Hygiene training in hospitals

Robotics
Smart Walking frame for the Frail Blind

Robotics
Personal Mobility – PamAid 2000’s

Robotics
PamAid – passive co-bot
Mode Selector, Volume
Control and speaker
Hand Brakes
Force sensing
handlebars Downward facing Lidar
Steered front wheels
Fixed rear wheels
with odometry Upward facing sonar
Horizontal Sonar
around base

Robotics
PamAid – active cobot

Robotics
Mapping and Interfaces
Actionable Map Elements Mapping with EKF SLAM with
action selection points

Robotics
Lessons Learned
• Keep what is useful, automate only when needed
– Passive co-bots easy for carers to understand
– Sense danger for active safety – runaway, etc
• Human-Machine interface is Critical
– 15min learning time
– Build on existing interfaces / metaphors
– Shared control but human has ultimate control
– Performance should be context dependant:,
• Warnings of context switching & limited number of contexts
• Performance must be consistent in a context
• Personalise settings to user needs and preferences

Robotics
References
• Gerard Lacey, Kenneth M. Dawson-Howe, The application of robotics to a
mobility aid for the elderly blind, Robotics and Autonomous Systems,
Volume 23, Issue 4,1998
• Rodriguez-Losada, D., Matia, F., Jimenez, A. & Lacey, G., Guido, the
Robotic SmartWalker for the frail visually impaired, First International
Conference on Domotics, Robotics and Remote Assistance for All-DRT4all,
2005
• G. J. Lacey and D. Rodriguez-Losada, "The Evolution of Guido," in IEEE
Robotics & Automation Magazine, vol. 15, no. 4, Dec. 2008

Robotics
Case Study 2: Surgical Skills Training
https://screenrant.com/tag/surgeon-simulator-2/

Robotics
Mixed Reality (XR) Surgical simulation

Robotics
Research Question
Can we add high fidelity haptics & ”reality” to surgical simulation?

Robotics
Stereo vision tracking
Cameras
Surgical Instruments
With fiducial markers

Robotics
Mixed Reality Simulation
• Anatomically correct plastic models inside bodyform & can have a pulse!
• Graphical overlay steps of the surgical procedure steps
• Can simulate more steps of procedure than VR:
– trocar insertion
– hand assisted
– removal of tissue
- bleeds (distractors)
- closing the wound
- team coordination

Robotics
Mixed Reality Advantages

Robotics
Measuring Surgical Skills?
• Learning has a high “Cognitive Load”
– Perceptual Blindness & Change Blindness
– Poor Situational Awareness
• Speed of surgery and number of Procedures logged not a measure of skill
– Psychomotor skills need “Deliberate practice”1
– Distributed not massed training
• 3D path smoothness is highly correlated with Proficiency
– Hand-eye coordination and path planning
– Response to distractors
1. Ericsson, K. A. (2008). Deliberate practice and acquisition of expert performance: A general over-view. Academic Emergency Medicine, 15(11)

Robotics
References
• G. Lacey, D. Ryan, D. Cassidy and D. Young, "Mixed-Reality Simulation of
Minimally Invasive Surgeries," in IEEE MultiMedia, vol. 14, no. 4, pp. 76-87,
Oct.-Dec. 2007, doi: 10.1109/MMUL.2007.79.
• Van Sickle, K.R., III, D.A.M., Gallagher, A.G. et al. Construct validation of
the ProMIS simulator using a novel laparoscopic suturing task. Surg
Endosc 19, 1227–1231 (2005). https://doi.org/10.1007/s00464-004-8274-6
• Broe, D., Ridgway, P.F., Johnson, S. et al. Construct validation of a novel
hybrid surgical simulator. Surg Endosc 20, 900–904 (2006).
https://doi.org/10.1007/s00464-005-0530-x

Robotics
Case Study 3: Colonoscopy
• Research Questions
– 30% of polyps missed in a colonoscopy
• Why?
• Can we reduce the % missed using technology ?
– Hypothesis: Poor coverage due to inexperience and moving too fast
• Research Challenges
– Understand expert behaviour
– Image processing in the colon
– 3D tracking of scope
– Realtime feedback to Clinician

Robotics
Colonoscopy Imaging

Robotics
Concept

Robotics
Colonoscopy Quality Measurement
Eye Tracker
External “Torque” Sensor
Visual Odometry
Dummy Patient

Robotics
Findings
• Experts eyes scan colons differently to novices
• Even experts don’t visualise the entire colon
– Situational awareness is hard to maintain
– Few landmarks, intestinal contents & specular surfaces
– Camera orientation is hard to control
• Solution
– combine camera visual odometry with measurements of hand
motions of operator to generate 3D map of colon
– Give live feedback to clinician if section missed
– Could be used on live patients to improve quality
• Colonoscopy is a difficult market

Robotics
References
• M. Arnold, A. Ghosh, G. Lacey, S. Patchett and H. Mulcahy, "Indistinct
Frame Detection in Colonoscopy Videos," 2009 13th International Machine
Vision and Image Processing Conference, 2009, pp. 47-52, doi:
10.1109/IMVIP.2009.16.
• Vilariño F., Lacey G., Zhou J., Mulcahy H., Patchett S. (2007) Automatic
Labelling of Colonoscopy Video for Cancer Detection. In: Martí J., Benedí
J.M., Mendonça A.M., Serrat J. (eds) Pattern Recognition and Image
Analysis. IbPRIA 2007. Lecture Notes in Computer Science, vol 4477.
Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-72847-4_38
• G Lacey, F Vilarino, Endoscopy system with motion sensors - US Patent
App. 12/736,536, 2011

Robotics
Case Study 4: Hand Hygiene Training
Target Users: Healthcare Workers Need: Reduce Hospital Infections

Robotics
SureWash – Measure Hand Hygiene

Robotics
Why is hand Hygiene hard?
• Gesture Recognition
– Fast
– Self occluding
– Precise
• Histogram of Orientation
Gradients
• Train multi-class classifier
– Use non ambiguous poses
• Performance Benchmark
– Inter-Rater Reliability (IRR)
– Construct Validity testing

Robotics
SureWash Cart

Robotics
SureWash Go

Robotics
Gamification – having fun
2 Person Games Console On Mobile Robot

Robotics
SureWash App

Robotics
Learning Science

Robotics
Lessons Learned
• Realtime feedback is key to behaviour change
• Learning physical tasks is “different”
– It takes time - “practice – REST – repeat”
– Deliberate practice – over learning – mastery learning
– Testing has a strong impact on the retention of learning
• Implementation Science
– Social structures key to group behaviour change
– Incentives, positive and negative important
– Cognitive offloading results in poor retention

Robotics
References
• Llorca, D., Vilarino, F., Zhou, Z., & Lacey, G. (2007). A multi-class SVM classifier
ensemble for automatic hand washing quality assessment. In BMVC
Proceeding of the British Machine Vision Con- ference, Warwick
• Lacey, G., Showstark, M., & Van Rhee, J. (2019). Training to Proficiency in the
WHO hand hygiene tech- nique. Journal of Medical Education and Curricular
Development.
• Lacey, G., Zhou, J., Li, X., Craven, C., & Gush, C. (2020). The impact of
automatic video auditing with real-time feedback on the quality and quantity of
handwash events in a hospital setting. American Journal of Infection Control,
48(2), 162–166.
• Gerard Lacey, Lucyna Gozdzielewska, Kareena McAloney‐Kocaman, Jonathan
Ruttle, Sean Cronin, Lesley Price (2021). Psychomotor learning theory informing
the design and evaluation of an interactive augmented reality hand hygiene
training app for healthcare workers. Education and Information Technologies,
May 2021.

Robotics
Synthesis of Key lessons
• Permanent Augmentation for sensory or
cognitive loss
– Technology: High Availability, Accuracy, Reliability, Repeatability
– HMI: Ease of Adoption, actionable & customised to current and
future needs
– Only automate what is necessary to maintain personal agency
– Regular personalisation to reflect Recovery or Decline in
capabilities

Robotics
• Temporary or non-universal Augmentation
– Technology: Mixed Availability, Accuracy, Reliability,
Repeatability
– HMI: don’t change existing workflows! Advisory role for tech
– Understand the learning curve for the task & progressively
withdraw the learning scaffold (personalisation)
– Train beyond initial competence to promote retention
– Build in regular formative assessment with real time feedback

Robotics
Human performance changes,
and our technology must
change with us

Robotics
Thanks to collaborators & Funding Agencies
Shane McNamara, Blaithin Gallagher, Derek Cassidy, Fernando Vilarino, Anarta Ghosh, David
Fernandes Llorca, Stefan Ameling, Fernando Viliarino, Pete Redmond, Mirko Arnold, Xichun Lee,
Jiang Zhu, Sofiane Yous, Jonathan Ruttle, Baichun Xia, Joan Cahill, Sean Cronin, Darren Caulfield,
Lucyna Gozdzielewska, Andrew Stewardson, Kareena McAloney-Kocaman, Prof Rozenn Dayhot, Prof
Helen Petrie, Prof Hillary Humphries, Prof Lesley Price, Prof Fidelma Fitzpatrick, Prof Steve Pachett,
Prof Hugh Mulcahy, Prof Didier Pittet…
EU Commission Enterprise Ireland UK Dept of Health

Robotics
Thank You
Gerry.Lacey@mu.ie
QR Code for slides

�ݺ�ߣ

Robots, sensors & people

More Related Content

Robots, sensors & people