�ݺ�ߣ

Topic
An Intuitive Interaction System for Fire
Safety Using
A Speech Recognition Technology
In context of Cooperative Fire Security
System using HARMS (CFS2H)
HCI

Presented by:
• Amna Iqbal
• Asma Iqbal
• Faiza Zahid
• Shabana Kausar
• Tahseen Fatima

Introduction
• An intuitive interaction system
Part of “Cooperative Fire Security System using HARMS”
(CFS2H)
Readily deal with fire in a high-rise building.
The interaction system is a bridge connecting human, as an
operator, to the whole system.
Utilizing a natural language processing (NLP) technology
using Microsoft kinect makes the interaction
Human-agent-robot-machine-sensor (HARMS) provides a
distributed network so that the systems are able to
communicate with a high level communication protocol.

Why Need For Human robot
interaction?
• Fire in a high-rise building is challenging for
firefighters to handle because many factors e.g.
Transporting firefighting equipment to the fire spot
located at a high floor, seeking the fire floor,
evacuation of people, etc. Are against firefighters.
• Firefighter ladders cannot reach high enough to
fight fires at the top of the building. Going into the
building itself, in order to extinguish the blazing fire,
is typically too dangerous and puts firefighters at
risk.
• The challenges faced by firefighters in a high-rise
building may result in a late reaction and possible
failure in the initial stages of fire suppression.

HRI For Fire Safety
• HRI (human robot interaction) system
involve
• Various types of robots
• Agents
• Humans
• HRI operation
• Fire detection
• Situation report to the operator
• Rescuing survivors guided by the operator

System configuration of CFS2H
Cooperative Fire Security System using HARMS

System Configuration of
CFS2H
• UGV robot
Search the area for the exact fire location and report to
interaction system, transmit fire location to firefighter robot
and human rescue robot.
• Big data
Analyzes the sensor data to determine the optimal
perception model to detect any potential fire.
• Human interface
Kiosk-type computer system that utilizes a Microsoft kinect
for speech recognition, a system to communicate with
cooperative system by natural language.

System Configuration of
CFS2H
• Wireless sensor network
Wireless sensor network gathers condition information
of certain place, also generate a fire alert
• Firefighting robot
extinguishes fire using fire extinguisher
• Human rescue robot
Rescue the survivor with the help of commands.

UGV RobotRescue robot Kiosk-type system
Firefighter robot

Objectives of the Interaction
System In CFS2H
• A computer system acts as a bridge to connect the
operator with the whole system (CFS2H).
• The user uses human voice to communicate with the
interaction system.
• Three main objectives are defined in order to achieve
the goal of the interaction system.
• Surveillance for fire safety
• Remote control of human rescue robot
• Survivor detection

Communication using
HARMS
• HARMS is a communication provider that allows systems to communicate with
each other with a high-level message protocol.
• HARMS supports multicast of a message to enable 1:n communication, where n
is the number of agents/systems in a system.
• All the systems are connected through TCP/IP protocol using either Ethernet or
Wi-Fi.
• All the systems of the whole system are fully connected
• The interaction system communicates with some of the systems;
 wireless sensor network
 Big data system
 Human rescue robot for rescuing survivor(s)
 UGV robot

System design and
implementation
• The intuitive interaction system that is proposed
comprised of:
Speech Module,
Face Detection Module,
HARMS Handler,
System Handler, including GUI

Cont….
• Speech Module
• Converts operator’s speech into a command
• Utilize a speech device called Microsoft kinect to be
able to recognize human speech.
• The speech recognizer uses a confidence value to
estimate individual words from speech
 Setting proper threshold for the confidence value is
important to accurately extract any words from human
voice
 Use threshold (0.2) that is achieved by trial and error
approach, a well-known method to find a solution by
reducing a previous trial error.

Cont…
Word Corresponding Command
NAVIS Wait for the following instruction.
Go straight Make rescuing robot move forward for 3 steps.
Turn left Make rescuing robot turn 90 degree to the left.
Turn right Make rescuing robot turn 90 degree to the right.
Search survivor Make rescuing robot look around to search survivor.
Situation Collect information from sensory and Big data
system to show user current condition of the building.
Command Definitions :
Matches between predefine words and commands. When operator says a
sentence with any of the defined words, the corresponding command or
function is executed.

Cont…
• Face detection module:
• Uses a face detection technique, based on object-detection
libraries provided by openCV .
• A series of frames transmitted from the human rescue robot, is
used to detect any human faces.
• While the interaction system is looking for any survivors, the
operator transmits control commands to the human rescue
robot directing it to move in order to search the surrounding
area in which some people might be waiting to be rescued.
• Once any survivor is detected by the face detection module, the
interaction system immediately sends a notification message to
let the human rescue robot know that a survivor is now in front
of the robot.

Cont…
• HARMS Module :
• All the communications or interactions
between the interaction system and other
systems are performed through HARMS
Module. It generates a message according to
HARMS messaging protocol.

Cont..
HARMS module provides 3ways to send
message:
• Unicast
• Multicast
• Broadcast.
The three message types distinguish ways of
how to react to a message.

The overview intuitive
interaction system

�ݺ�ߣ

Human computer interaction

More Related Content

Human computer interaction

Editor's Notes