際際滷

際際滷Share a Scribd company logo

Augmented Reality Web Applications with Mobile Agents in the Internet of Things

Jukka Riekki
Jukka Riekki

Augmented reality (AR) is a promising technology for building applications in an Internet of Things (IoT) environment, utilized for visualizing information provided by IoT devices. In this paper, we enable Web-based mobile AR applications with mobile agents in a resource-oriented IoT system architecture. We present an adaptable mobile agent composition that contains the data representation logic and mappings between AR applications and system resources. Thus, mobile agents and AR application-specific data structures are exposed as global system resources. System resource linkages are considered between real-world objects and their virtual representations for mobile agent-based AR applications. The agent composition also complies with the REST principles for resource access and control system-wide. This allows dynamic runtime adaptation and addressing the device and resource heterogeneity, thus eliminating the need for application-specific communication protocols. Moreover, we utilize a Web-based mobile AR application framework, running completely in a Web browser, which facilitates straightforward AR application development. Lastly, a proof of concept mobile AR application is implemented, where a coffee maker with a visual tag is connected to a low-power resource-constrained wireless sensor network node as an IoT device. A mobile agent is injected into the IoT environment to expose the state changes of the coffee maker. Through the visual tag, AR applications are able to visualize the state changes of the coffee maker in their user interface.

1 of 9
Downloaded 11 times
Augmented Reality Web Applications with Mobile Agents in the Internet of Things Lepp辰nen, T., Heikkinen, A., Karhu, A., Harjula, E., Riekki, J. and Koskela, T. Augmented Reality Web Applications with Mobile Agents in the Internet of Things, In: 8th International Conference on Next Generation Mobile Applications, Services, and Technologies (NGMAST2014), pp. 54-59, September 10-12, Oxford, UK, 2014. http://dx.doi.org/10.1109/NGMAST.2014.24 Teemu Lepp辰nen1, Arto Heikkinen2, Antti Karhu2, Erkki Harjula2, Jukka Riekki1 and Timo Koskela2 1) Department of Computer Science and Engineering, University of Oulu, Finland 2) Center for Internet Excellence, University of Oulu, Finland
Why mobile agents in IoT for AR? Augmented reality technologies provide an enhanced user interface to monitor and control smart environments Mobile agents enable injecting application-specific computations into the system Create agent-based services Re-usable in different applications and services Platform and programming language independent Adapts to fluctuating IoT system environment Transfer code between AR application and system devices Application-specific data processing and UI code for different platforms Mobile agents live in IoT system without explicit coupling to AR applications Facilitates loose coupling between AR application and system resources AR tag mapped to a POI, which is in turn mapped to a system resource, i.e. agent- based service or computation
Integration to the Web Resource-oriented architecture Resouce is the main abstraction, exposed with unique URI and state Enables various representations of the resource state for human-machine interactions Resources include: System devices, their physical components (sensors and actuators) and data Mobile agents as representation of the application-specific data processing In AR: POI entities Unified and standardized communication for resource access and control Promotes simplicity and re-usability in the application design HTTP methods: GET, POST, DELETE Create new resources and functionality, retrieve application-specific representations and control the resources Facilitates seamless information retrieval and control for both physical and virtual objects with unified interface for AR applications
Web-based mobile AR Framework Enables implementing mobile AR (MAR) applications running plugin-free in a Web browser Relies on common Web technologies HTML5, WebRTC and XML3D Modular design Compatible with all mobile platforms and operating systems Components Sensor-based registration and tracking component detects the devices position and orientation Vision-based registration and tracking component detects AR tags from camera video stream 3D scene manipulation component converts above data for XML3D XML3D provides representations of the content for HTML Web service-interface provides communication
IoT system architecture Based on IETF CoRE WGs framework Mobile device hosts the AR application in browser WSN based on 8-bit MCU nodes System components Resource directory stores the system resource descriptions Name, URI, current address, content-type, semantic description, Proxy component translates protocols over disparate networks HTTP (Wi-Fi atop 2.4Ghz) <-> CoAP (6LoWPAN atop 868Mhz) Repositories POI Database hosts POI entity descriptions with corresponding AR tags and linked resources Other repositories host more system resources: e.g. mobile agent code
Proof-of-concept application (1/2) MAR application running in a Web browser in a tablet device Shows the freshness of coffee in a coffee maker 1. Detects the AR tag from video feed 2. Retrieves the POI description 3. Retrieves the state of the corresponding linked resource, i.e. mobile agent 4. Displays the state representation in MAR application in the Web browser
Proof-of-concept application (2/2) Coffeemaker Connected to a WSN node, which monitors its power consumption Attached AR tag, with POI description linked to a mobile agent Mobile agent injected to the system to calculate time since last state change of the coffeemaker Thresholding the power consumption for two states: ON and OFF Mobile agent composition POI data structure
Conclusions Augmented reality is a natural way of visualizing information provided by different IoT devices Web browser provides a uniform application development platform for AR applications IoT systems and applications become Web-enabled with the resource- oriented architectural approach Mobile agents enable injecting dynamic application-specific data processing and information visualization into the system The presented prototype application was straightforward to implement with the presented IoT system architecture
Thank you! Questions? For further information, please contact: teemu.leppanen@ee.oulu.fi arto.heikkinen@cie.fi

More Related Content

Augmented Reality Web Applications with Mobile Agents in the Internet of Things

  • 1. Augmented Reality Web Applications with Mobile Agents in the Internet of Things Lepp辰nen, T., Heikkinen, A., Karhu, A., Harjula, E., Riekki, J. and Koskela, T. Augmented Reality Web Applications with Mobile Agents in the Internet of Things, In: 8th International Conference on Next Generation Mobile Applications, Services, and Technologies (NGMAST2014), pp. 54-59, September 10-12, Oxford, UK, 2014. http://dx.doi.org/10.1109/NGMAST.2014.24 Teemu Lepp辰nen1, Arto Heikkinen2, Antti Karhu2, Erkki Harjula2, Jukka Riekki1 and Timo Koskela2 1) Department of Computer Science and Engineering, University of Oulu, Finland 2) Center for Internet Excellence, University of Oulu, Finland
  • 2. Why mobile agents in IoT for AR? Augmented reality technologies provide an enhanced user interface to monitor and control smart environments Mobile agents enable injecting application-specific computations into the system Create agent-based services Re-usable in different applications and services Platform and programming language independent Adapts to fluctuating IoT system environment Transfer code between AR application and system devices Application-specific data processing and UI code for different platforms Mobile agents live in IoT system without explicit coupling to AR applications Facilitates loose coupling between AR application and system resources AR tag mapped to a POI, which is in turn mapped to a system resource, i.e. agent- based service or computation
  • 3. Integration to the Web Resource-oriented architecture Resouce is the main abstraction, exposed with unique URI and state Enables various representations of the resource state for human-machine interactions Resources include: System devices, their physical components (sensors and actuators) and data Mobile agents as representation of the application-specific data processing In AR: POI entities Unified and standardized communication for resource access and control Promotes simplicity and re-usability in the application design HTTP methods: GET, POST, DELETE Create new resources and functionality, retrieve application-specific representations and control the resources Facilitates seamless information retrieval and control for both physical and virtual objects with unified interface for AR applications
  • 4. Web-based mobile AR Framework Enables implementing mobile AR (MAR) applications running plugin-free in a Web browser Relies on common Web technologies HTML5, WebRTC and XML3D Modular design Compatible with all mobile platforms and operating systems Components Sensor-based registration and tracking component detects the devices position and orientation Vision-based registration and tracking component detects AR tags from camera video stream 3D scene manipulation component converts above data for XML3D XML3D provides representations of the content for HTML Web service-interface provides communication
  • 5. IoT system architecture Based on IETF CoRE WGs framework Mobile device hosts the AR application in browser WSN based on 8-bit MCU nodes System components Resource directory stores the system resource descriptions Name, URI, current address, content-type, semantic description, Proxy component translates protocols over disparate networks HTTP (Wi-Fi atop 2.4Ghz) <-> CoAP (6LoWPAN atop 868Mhz) Repositories POI Database hosts POI entity descriptions with corresponding AR tags and linked resources Other repositories host more system resources: e.g. mobile agent code
  • 6. Proof-of-concept application (1/2) MAR application running in a Web browser in a tablet device Shows the freshness of coffee in a coffee maker 1. Detects the AR tag from video feed 2. Retrieves the POI description 3. Retrieves the state of the corresponding linked resource, i.e. mobile agent 4. Displays the state representation in MAR application in the Web browser
  • 7. Proof-of-concept application (2/2) Coffeemaker Connected to a WSN node, which monitors its power consumption Attached AR tag, with POI description linked to a mobile agent Mobile agent injected to the system to calculate time since last state change of the coffeemaker Thresholding the power consumption for two states: ON and OFF Mobile agent composition POI data structure
  • 8. Conclusions Augmented reality is a natural way of visualizing information provided by different IoT devices Web browser provides a uniform application development platform for AR applications IoT systems and applications become Web-enabled with the resource- oriented architectural approach Mobile agents enable injecting dynamic application-specific data processing and information visualization into the system The presented prototype application was straightforward to implement with the presented IoT system architecture
  • 9. Thank you! Questions? For further information, please contact: teemu.leppanen@ee.oulu.fi arto.heikkinen@cie.fi