ºÝºÝߣ

ºÝºÝߣShare a Scribd company logo

Augmented Reality - state of the art

?
?
Alexandre De Saedeleer
Alexandre De Saedeleer

Learn what you can do in Augmented Reality in 2020. Apps and web technologies let you create awesome AR experiences. We give you everything you should know before choosing a technology.

1 of 22
Download to read offline
. AR - State of the Art 1
Augmented Reality - state of the art
. Technological differences between Native and Web environments 3
.4 Native & ARKit/ARCore: World tracking & Scene understanding With ARKit & ARCore on smartphones, the system is able to track the world around it and to understand the scene it is in. As such, a virtual object can be positioned with a true sense of reality, without any marker needed. What¡¯s important to note is that whenever the virtual object is positioned, it stays at its position even if the user walks in the room and points another zone with its camera.
.5 Native & ARKit/ARCore: People occlusion & motion capture AR content realistically passes behind and in front of people in the real world, making AR experiences more immersive while also enabling green screen-style e?ects in almost any environment. With the new releases of ARKit, developers can now use people occlusion and movement capture. That¡¯s what you seen in the pictures below. People occlusion We continue to see the AR object behind someone Motion capture ARKit records people movement and is able to replicate it live
.6 Native & ARKit/ARCore: Also with markers Markers can be used with ARKit/ARCore when one needs to put a virtual object at a very precise location, the marker can then be: - A 2D image (e.g. a museum painting ¡ª see visual hereunder) - A square marker / QR Code - A face - In some cases, a 3D object
.7 Web: Only markers On a web environment, Augmented Reality can be used only with markers. Most of the time the marker needs to be a square marker / QR Code marker. 2D images markers and face recognition markers on the web exist, but their performance can be very deceptive. The marker must always be visible.
.8 Exception for the Web: 3D objects preview with AR Apple has ¡°AR Quick Look¡± which gives the ability to preview on a webpage a 3D object within the reality. Under the hoods it relies on ARKit, but it o?ers very little interaction (especially, as a developer we cannot bring any interaction). Android has an equivalent feature, called ¡°Scene Viewer¡±: https://developers.google.com/ar/develop/java/scene-viewer Try it out: https://developer.apple.com/augmented-reality/quick-look/
.9 Web: why no World tracking and Scene understanding? Web technologies by themselves (i.e. Browser / HTML / JavaScript) aren¡¯t powerful enough to support World tracking and Scene understanding, as it involves advanced software, machine learning and deep access to the hardware. A new web standard, WebXR, is being built to bring this support. Basically, WebXR will bridge the web environment with the underlying ARKit and ARCore on the compatible smartphones and tablets. However, the WebXR standard is not ?nalized yet, and only experimental browsers partly support it: - Firefox WebXR - Google Chrome Canary editions Source: /AugmentedWorldExpo/casper-fabricius- cimmerse-the-state-of-webxr
. What can be achieved with ARKit / ARCore (Native) 10
.11 Marker-based concepts: replacing a 2D image with a 3D virtual object The marker is the image of a painting, and is replaced by a 3D virtual object Advantages with ARKit / ARCore: 1. Once the virtual object is positioned in the world, the user can navigate around the room or around the object without ¡°losing¡± it if the marker is not visible anymore by the camera. 2. The marker can be a 2D image or even a face (no need to be a ¡°square marker¡± or QR-code-like marker) /! Need to be careful about lightning of the 2D image, the angle, etc. ==> impact on time of detection
.12 Markerless + Manual operation In a markerless con?guration, the most common scenario is the following: 1. The user must ?rst scan the environment (moving around his phone) 2. The user taps on the screen to position the object 3. The object is positioned and locked. To test this behaviour with an iPhone: https://developer.apple.com/augmented-reality/quick-look/ This means a manual operation is needed. Advantage with ARKit / ARCore: - Markerless AR is only possible with ARKit / ARCore - The user can walk around in the room, come back and the virtual object will still be at the same (more or less) position.
.13 Markerless + GPS position ==> Precise positioning isn¡¯t an option For IOC, an interesting case would be to position virtual object at exact GPS coordinates e.g. the Sakura trees in front of the museum. While there are numerous projects on the web to achieve such a thing, they all have important limitations: - It is very hard to display 3D objects when working with GPS coordinates - The location is not precise, because on one hand the GPS of the phone is not precise (from a couple meters to tens of meters) and on the other hand it is very hard to have a correct match between GPS coordinates and the ¡°world detected¡± by ARKit / ARCore.
.14 Markerless + GPS position ==> Approximative positioning What is feasible is to position a 3D object in the reality using an approximative positioning, it would basically be geofencing-based: 1. Acquiring GPS position of the user 2. If the user is in a pre-de?ned radius e.g. a speci?c lat/long with a tolerance of a few meters => The experience is activated. We can take into account the compass of the user to be sure of the location he¡¯s looking at. 3. The AR experience is positioned in front of the user. This is the Pokemon Go principle. Totally OK for characters in the park, not precise enough for Sakuras.
.15 Portal doors Portal doors can be done with Native technologies but they su?er the same problem as GPS positioning: they cannot be put for sure at one exact location. However, they can be put at an approximate location, without being a too important problem for the concept.
.16 Other advantages of ARKit / ARCore vs Web Thanks to ¡°World tracking¡± and ¡°Scene understanding¡±, ability to: - Have interactions between virtual objects and the real world: object falling from a table, hitting a wall, etc. - Have objects scaled to the real world without a marker - Take into account light in the room - ARKit 3 / iPhone XS+: People Occlusion - ARKit 3 / iPhone XS+: Motion capture (Kinect-like) - ARKit 3 / iPhone XS+: front+back camera at the same time
. What can be achieved with Web technologies 17
.18 With marker The Web AR can only work with a marker at the moment. Ideally this must be a ¡°square marker¡± or QR-code-like marker. With 2D images as a marker and face marker, it is possible but this can be problematic performance-wise at the moment. IMPORTANT LIMITATION: the marker must be correctly visible at ALL TIME otherwise the object disappear. If it disappear and you come back on the image, there is a ?ickering e?ect (you see the marker for a fraction of second, before it is replaced).
.19 With marker The Web AR can only work with a marker at the moment. Ideally this must be a ¡°square marker¡± or QR-code-like marker. With 2D images as a marker and face marker, it seems to be possible but this might be problematic performance-wise at the moment. IMPORTANT LIMITATION: the marker must be correctly visible at ALL TIME otherwise the object disappear. If it disappear and you come back on the image, there is a ?ickering e?ect (you see the marker for a fraction of second, before it is replaced). https://www.youtube.com/watch?v=0MtvjFg7tik
.20 Multiple markers Supposed to work but seems to be complex / not totally reliable: https://stackover?ow.com/questions/51850930/a-frame-ar-js-multiple-markers-boxes?rq=1
.21 Web-based portal door The web-based portal door is possible, but it requires a marker and this comes with the disadvantages of using a marker: The marker must be seen at all time (this is a HUGE di?culty UX-wise). https://www.youtube.com/watch?v=tcudraEerTY
Augmented Reality - state of the art

More Related Content

Augmented Reality - state of the art

  • 1. . AR - State of the Art 1
  • 4. .4 Native & ARKit/ARCore: World tracking & Scene understanding With ARKit & ARCore on smartphones, the system is able to track the world around it and to understand the scene it is in. As such, a virtual object can be positioned with a true sense of reality, without any marker needed. What¡¯s important to note is that whenever the virtual object is positioned, it stays at its position even if the user walks in the room and points another zone with its camera.
  • 5. .5 Native & ARKit/ARCore: People occlusion & motion capture AR content realistically passes behind and in front of people in the real world, making AR experiences more immersive while also enabling green screen-style e?ects in almost any environment. With the new releases of ARKit, developers can now use people occlusion and movement capture. That¡¯s what you seen in the pictures below. People occlusion We continue to see the AR object behind someone Motion capture ARKit records people movement and is able to replicate it live
  • 6. .6 Native & ARKit/ARCore: Also with markers Markers can be used with ARKit/ARCore when one needs to put a virtual object at a very precise location, the marker can then be: - A 2D image (e.g. a museum painting ¡ª see visual hereunder) - A square marker / QR Code - A face - In some cases, a 3D object
  • 7. .7 Web: Only markers On a web environment, Augmented Reality can be used only with markers. Most of the time the marker needs to be a square marker / QR Code marker. 2D images markers and face recognition markers on the web exist, but their performance can be very deceptive. The marker must always be visible.
  • 8. .8 Exception for the Web: 3D objects preview with AR Apple has ¡°AR Quick Look¡± which gives the ability to preview on a webpage a 3D object within the reality. Under the hoods it relies on ARKit, but it o?ers very little interaction (especially, as a developer we cannot bring any interaction). Android has an equivalent feature, called ¡°Scene Viewer¡±: https://developers.google.com/ar/develop/java/scene-viewer Try it out: https://developer.apple.com/augmented-reality/quick-look/
  • 9. .9 Web: why no World tracking and Scene understanding? Web technologies by themselves (i.e. Browser / HTML / JavaScript) aren¡¯t powerful enough to support World tracking and Scene understanding, as it involves advanced software, machine learning and deep access to the hardware. A new web standard, WebXR, is being built to bring this support. Basically, WebXR will bridge the web environment with the underlying ARKit and ARCore on the compatible smartphones and tablets. However, the WebXR standard is not ?nalized yet, and only experimental browsers partly support it: - Firefox WebXR - Google Chrome Canary editions Source: /AugmentedWorldExpo/casper-fabricius- cimmerse-the-state-of-webxr
  • 10. . What can be achieved with ARKit / ARCore (Native) 10
  • 11. .11 Marker-based concepts: replacing a 2D image with a 3D virtual object The marker is the image of a painting, and is replaced by a 3D virtual object Advantages with ARKit / ARCore: 1. Once the virtual object is positioned in the world, the user can navigate around the room or around the object without ¡°losing¡± it if the marker is not visible anymore by the camera. 2. The marker can be a 2D image or even a face (no need to be a ¡°square marker¡± or QR-code-like marker) /! Need to be careful about lightning of the 2D image, the angle, etc. ==> impact on time of detection
  • 12. .12 Markerless + Manual operation In a markerless con?guration, the most common scenario is the following: 1. The user must ?rst scan the environment (moving around his phone) 2. The user taps on the screen to position the object 3. The object is positioned and locked. To test this behaviour with an iPhone: https://developer.apple.com/augmented-reality/quick-look/ This means a manual operation is needed. Advantage with ARKit / ARCore: - Markerless AR is only possible with ARKit / ARCore - The user can walk around in the room, come back and the virtual object will still be at the same (more or less) position.
  • 13. .13 Markerless + GPS position ==> Precise positioning isn¡¯t an option For IOC, an interesting case would be to position virtual object at exact GPS coordinates e.g. the Sakura trees in front of the museum. While there are numerous projects on the web to achieve such a thing, they all have important limitations: - It is very hard to display 3D objects when working with GPS coordinates - The location is not precise, because on one hand the GPS of the phone is not precise (from a couple meters to tens of meters) and on the other hand it is very hard to have a correct match between GPS coordinates and the ¡°world detected¡± by ARKit / ARCore.
  • 14. .14 Markerless + GPS position ==> Approximative positioning What is feasible is to position a 3D object in the reality using an approximative positioning, it would basically be geofencing-based: 1. Acquiring GPS position of the user 2. If the user is in a pre-de?ned radius e.g. a speci?c lat/long with a tolerance of a few meters => The experience is activated. We can take into account the compass of the user to be sure of the location he¡¯s looking at. 3. The AR experience is positioned in front of the user. This is the Pokemon Go principle. Totally OK for characters in the park, not precise enough for Sakuras.
  • 15. .15 Portal doors Portal doors can be done with Native technologies but they su?er the same problem as GPS positioning: they cannot be put for sure at one exact location. However, they can be put at an approximate location, without being a too important problem for the concept.
  • 16. .16 Other advantages of ARKit / ARCore vs Web Thanks to ¡°World tracking¡± and ¡°Scene understanding¡±, ability to: - Have interactions between virtual objects and the real world: object falling from a table, hitting a wall, etc. - Have objects scaled to the real world without a marker - Take into account light in the room - ARKit 3 / iPhone XS+: People Occlusion - ARKit 3 / iPhone XS+: Motion capture (Kinect-like) - ARKit 3 / iPhone XS+: front+back camera at the same time
  • 17. . What can be achieved with Web technologies 17
  • 18. .18 With marker The Web AR can only work with a marker at the moment. Ideally this must be a ¡°square marker¡± or QR-code-like marker. With 2D images as a marker and face marker, it is possible but this can be problematic performance-wise at the moment. IMPORTANT LIMITATION: the marker must be correctly visible at ALL TIME otherwise the object disappear. If it disappear and you come back on the image, there is a ?ickering e?ect (you see the marker for a fraction of second, before it is replaced).
  • 19. .19 With marker The Web AR can only work with a marker at the moment. Ideally this must be a ¡°square marker¡± or QR-code-like marker. With 2D images as a marker and face marker, it seems to be possible but this might be problematic performance-wise at the moment. IMPORTANT LIMITATION: the marker must be correctly visible at ALL TIME otherwise the object disappear. If it disappear and you come back on the image, there is a ?ickering e?ect (you see the marker for a fraction of second, before it is replaced). https://www.youtube.com/watch?v=0MtvjFg7tik
  • 20. .20 Multiple markers Supposed to work but seems to be complex / not totally reliable: https://stackover?ow.com/questions/51850930/a-frame-ar-js-multiple-markers-boxes?rq=1
  • 21. .21 Web-based portal door The web-based portal door is possible, but it requires a marker and this comes with the disadvantages of using a marker: The marker must be seen at all time (this is a HUGE di?culty UX-wise). https://www.youtube.com/watch?v=tcudraEerTY