AR / UX: Building Augmented Reality Experiences
1 like2,307 views
These are the slides from the presentation given at the joint Tampa Bay User Experience / Front End Design Group meetup by Anitra Pavka and Joey deVilla on the evening of June 14, 2018.
More Related Content
What's hot (7)
Similar to AR / UX: Building Augmented Reality Experiences (20)
Recently uploaded (20)
![[Webinar] Scaling Made Simple: Getting Started with No-Code Web Apps](https://cdn.slidesharecdn.com/ss_thumbnails/webinarscalingmadesimplegettingstartedwithno-codewebapps-mar52025-250305183437-f03c78a3-thumbnail.jpg?width=560&fit=bounds)
AR / UX: Building Augmented Reality Experiences
- 1. AR / UX: BUILDING AUGMENTED REALITY EXPERIENCES Anitra Pavka (@APavka) ? and Joey deVilla (@AccordionGuy)
- 2. ANITRA PAVKA ? Technical Product Manager at Malwarebytes, working on the consumer anti-malware product used by almost 15 million people. ? Contributing author to O¡¯Reilly¡¯s book, HTML5 Cookbook. ? Technical editor for O¡¯Reilly¡¯s book, Universal Design for Web Applications. ? Three-time speaker on accessibility and usability issues at SXSW Interactive. ? Previous Tampa Bay UX Meetup speaker about Apple TV UX.
- 3. JOEY DEVILLA ? Lead Product Manager at Sourcetoad, overseeing the design and implementation of mobile applications. ? Contributing author at the mobile developer tutorial site RayWenderlich.com. ? ARKit session and tutorial presenter at RWDevCon. ? Technical editor for the book ARKit by Tutorials. ? Tampa iOS Meetup organizer. ? Former Microsoft dev evangelist; tried to make Windows Phone 7 happen.
- 5. ¡°THE MASTER KEY¡± (1901) ¡°It consists of this pair of spectacles. While you wear them every one you meet will be marked upon the forehead with a letter indicating his or her character¡¡± ? Good: G ? Evil: E ? Wise: W ? Foolish: F ? Kind: K ? Cruel: C
- 6. IVAN SOUTHERLAND AND ? ¡°THE SWORD OF DAMOCLES¡± (1968)
- 7. TOM CAUDELL COINS THE PHRASE? ¡°AUGMENTED REALITY¡± (1990)
- 8. JOEY¡¯S NEIGHBOR IN TORONTO, STEVE MANN? (1980S AND BEYOND)
- 9. ¡°FIRST AND TEN¡± SYSTEM (1998)
- 13. DEFINING AUGMENTED REALITY The integration of digital information (e.g. graphics, sound, video)¡ ¡into a live view of the user¡¯s physical environment in real time. The word ¡®augmented¡¯ suggests it enriches the real-world view rather than blocking the real environment.
- 14. Augmented? reality Virtual? reality Reality - virtuality continuum Mixed reality Adapted from ¡°Reality-Virtuality Continuum¡± (Milgram, Takemura, Utsumi and Kishino,? University of Toronto, 1994) Real? environment Virtual? environment THE REALITY-VIRTUALITY CONTINUUM
- 15. DEFINING UX ¡°A person¡¯s perceptions and responses that result from the use or anticipated use of a product, system or service¡±. Includes the user¡¯s: ? Emotions ? Beliefs ? Preferences ? Motivations ? Perceptions ? Physical and psychological responses ? Accomplishments that occur before, during and after use of a system User experience is affected by the number and type of interactions within the augmented environment.
- 16. GENERAL USES FOR AR Guide Create Play ? Navigation assistance ? Exploring curated spaces ? Information assistants ? Information overlays and ¡°heads-up¡± displays ? Education ? Virtual construction and measurement tools ? Games ? Social networking ? Shopping assistants
- 17. BENEFITS OF AR Decreased? interaction cost Reduced? cognitive load Minimized? attention switching ? Relevant data is presented without requiring special action ? Efficient interactions with the AR system,? less user effort ? Users are NOT required to learn commands ? Allows users to move information smoothly from one context to another ? Multiple sources of information are combined into a single view ¡ª user¡¯s attention is not divided
- 18. Input method Description Visual Using the camera(s) for tracking, detection, and context identification. Tactile All interfaces that require touching the screen and other controls. Sensors Automatically detecting the environment context. Gesture Moving your hand in a specific way to interact with the environment. Voice Spoken commands for hands-free controls. Auditory For direct voice commands and inferring the user¡¯s context. Kinesthetic Recognizing the user¡¯s movement and posture. INPUT
- 19. Output method Description Visual From simple annotations to complex hyperrealistic 3D graphics. Auditory Sound output, from simple notification chimes to 3D sound. Haptic Vibration, force feedback, and other touch-related sensation. Olfactory Providing smells to increase the sense of immersion. OUTPUT
- 20. CONTEXT / ENVIRONMENT AR experiences occur in expected and unexpected places. ? Where do you expect users to experience this augmentation? ? Where else could they experience this augmentation? ? Could this experience occur with other people? ? Could users have to navigate in the real world while using this augmentation?
- 21. ENVIRONMENT, PART 1 Scout the locations where your app will be used. ? Indoors, outdoors, or both? ? Lighting: Is there bright sunlight or shadows? ? Sound: Is there ambient noise or music? How much? ? How much will the user be moving? ? How much room is there for the user to move around?
- 22. ENVIRONMENT, PART 2 ? How many large, flat surfaces are there? ? Are there many obstacles to movement in the area? ? Is network connectivity important? ? What¡¯s the social setting ¡ª relatively private, or public?? AR requires seamless experiences across environments and devices, where the user is always at the center.
- 23. DEMO #1 HAPPY AR PAINTERBy Joey deVilla
- 24. When initially? engaging in? AR experiences, ? users tend to? remain stationary? due to habits built? by 2D interactions.
- 25. Don¡¯t enforce user movement until it¡¯s necessary.? Introduce movement gradually.
- 26. Design for comfort. Do not put the user in an uncomfortable situation or position. Avoid large or sudden movements.
- 27. Educate the user about specific movements that are required to trigger an experience.
- 28. Provide clear direction when a user needs to transition from one movement to another.
- 29. Give clear guidance on movement range and necessary adjustments to users¡¯ position, pose or gesture.
- 30. Anticipate the limitations of users¡¯ spaces¡ªboth indoor and outdoor, the space¡¯s physical size, or any obstacles including furniture, things, or people.
- 31. Do not ask the user to walk backwards.
- 32. Avoid placing large objects directly in front of users. This may cause them to physically back up.
- 33. Give users a clear idea of the amount of space needed for the experience.
- 34. When the user is not able to move around, provide an alternate experience. For example, activities like interacting with or transforming objects.
- 35. As users fatigue, keep in mind that they may tend to move less.? It¡¯s important to ensure that any text or annotations are clear and legible from any angle. If you walk past the position? of ordinary AR text and look back to read it,? you¡¯ll see something like this. Use things like ¡°billboard constraints¡± to automatically rotate AR text to always face the user.
- 36. SAFETY
- 37. Prioritize the real world over AR content. A balanced experience should not replace the user¡¯s perception of reality with AR.
- 38. Prioritize line of sight. Do not distract or block users from seeing something important in the real world. Remind users to be aware of their surroundings.
- 39. Design for a narrow depth of field. Draw focus to objects only within a reasonable range.
- 40. Don¡¯t make users walk around staring at a small mobile screen the entire time. It¡¯s distracting and dangerous.
- 41. COMFORT
- 42. Challenge users mentally, not physically.
- 43. Minimize flashing or heavy duty animations. Mobile screens induce fatigue and eyestrain even more that desktop screens do.
- 44. Remember the ¡°gorilla arm effect¡±. Users will usually tolerate a higher degree of interaction while sitting or when they¡¯re holding their mobile device at chest level or lower.
- 45. Have the user hold the device with two hands rather than one. It allows them to focus on the experience rather than the device.
- 46. Place commonly-used UI elements in a comfortable to reach area.
- 47. Encourage users to take a break. Allow them to pause, save their current progress, and resume their AR experience later.
- 48. PRIVACY
- 49. Voice commands and audio output can be disruptive and have privacy implications in a public environment.
- 50. Shared environments have security and privacy implications. Accessing someone else¡¯s AR ¡°view¡± may or may not be desirable. 2016 ISACA study: 77% of consumers were concerned that AR- enhanced IoT devices may be more prone to a privacy breach.
- 51. Security breaches may include inappropriate access to protected information or areas and hiding information or real identities.
- 52. GENERAL ADVICE
- 53. Keep it simple. The more an augmentation attempts to grab attention, the more users will focus on it,? instead of reality.
- 54. Do not make your augmented environments too information-dense.
- 55. If you must display instructional text, use approachable terminology. AR jargon may intimidate users. Use less text than a traditional mobile interface. The Sans Serif fonts are better choices than Serifs.
- 56. Design for the small screen first. Consider lighting and contrast so that objects and text can be viewed clearly in different background surroundings and lighting levels.
- 57. Use visual and audio cues to let the user know which elements of the augmented environment can be interacted with, and how to interact with them.
- 58. Use lighting effects, such as shadows and directional lighting to make the augmented environment blend in more realistically with the real world environment.
- 59. Use as much of the display as possible, but do not hide key navigation elements.
- 60. Placement: The information that is presented should not obscure the item of interest.
- 61. Use audio or haptic feedback to enhance the sense that the user has interacted with the augmented environment.
- 62. Provide a continuity of experience ¡ª avoid flows that repeatedly take the user in and out of an AR scene.
- 63. Ensure that the information that is presented is scalable across different form factors.
- 64. Support both portrait and landscape modes to create a more immersive experience and increase user comfort. If this isn¡¯t possible, select the mode that¡¯s ideal for your experience.
- 65. DEMO #2 RAYKEABy Joey deVilla
- 66. INTERACTION
- 67. Indicate when initialization and surface detection is in progress and involve the user.
- 68. Use animation to demonstrate how users are supposed to interact with the application.
- 69. Icons: The user should be able to identify the items¡¯ category at a glance without reading the text label.
- 70. REACTING
- 71. You can enhance an AR experience using known imagery in the user¡¯s environment to trigger the appearance of virtual content.
- 72. Design and display the reference images to optimize detection. Specify the physical size that you expect to find those images in the user¡¯s environment.
- 73. Use detected imagery only as a frame of reference. Virtual content placed on it may not stay in place because changes to the position or orientation of detected imagery are not tracked.
- 74. DEMO #3 BAEDEKARBy Joey deVilla
- 76. Avoid interruptions. Let users adjust virtual objects and their settings in its context because device orientation and position cannot be tracked when AR is inactive.
- 77. Accidental gestures: Avoid using similar gestures to perform different types of interactions.
- 78. Suggest possible fixes to help users recover from common issues and reduce frustration. You can use visual and audio cues, animation, and text.
- 79. If all else fails, allow users to reset and start again to see if they get better results.
- 81. Cameras: ? Dual rear camera (for depth perception) ? Front camera with face-detection capabilities
- 82. Y Roll:? Rotation around? the Y-axis Pitch:? Rotation around? the X-axis X Z Yaw:? Rotation around? the Z-axis MOTION DETECTION: FIRST 3 DEGREES
- 83. MOTION DETECTION: NEXT 3 DEGREES Y Heave:? Movement parallel? to the Y-axis (up and down) Sway:? Movement parallel? to the X-axis (right and left) X Z Surge:? Movement parallel? to the Z-axis (back and forth)
- 84. -Z:? Relative forward +Z: Relative back -X: ? Relative? left +X:? Relative? right +Y: ? Relative? up -Y: Relative? down Origin is device¡¯s? location at start? of AR session CAMERA-BASED ALIGNMENT
- 85. +Y: Up -Z: North +Z: South -X: West +X: East Opposite gravity -Y: Down With gravity Origin is device¡¯s? location at start? of AR session GRAVITY- AND HEADING-BASED ALIGNMENT
- 86. 1. A point on the plane. This can be any one of the feature points identi?ed by the mobile AR framework. 2. A vector normal (perpendicular)? to the plane.? For horizontal planes, this vector is gravity, which mobile device accelerometers can detect. HOW MOBILE AR DETECTS HORIZONTAL PLANES
- 87. DETECTING VERTICAL PLANES IS TRICKIER 1. Finding a point on the plane is easy. It¡¯s just like ?nding a feature point on a horizontal surface. 2. Finding a vertical plane¡¯s normal is hard. With vertical planes, we don¡¯t have a handy force like gravity to rely on to ?nd the normal. (Special bragging rights to the ?rst? person who can answer this question:? ¡°Why can¡¯t we simply use a vector? that¡¯s perpendicular to gravity? as the normal for vertical surfaces?¡±)
- 89. CATEGORIZATION EXERCISE:? WHICH OF THESE IS A HEAVY METAL BAND?
- 90. DEMO #4 VISION QUESTBy Joey deVilla
- 91. HOW TO GET STARTED ON YOUR OWN AR PROJECTS
- 92. IOS Framework: ARKit User system requirements:? Any iDevice with an A9 processor or later? (iPhone 6S or later; iPad Pro or 2017 iPad) Development environment:? Xcode IDE, ? Swift or Objective-C programming language
- 93. ARKit by Tutorials ¡ª available at RayWenderlich.com
- 94. Awesome ARKit ¡ª github.com/olucurious/Awesome-ARKit A regularly updated list of interesting ARKit projects? on GitHub.
- 95. globalnerdy.com/rwdevcon2018-tutorial A free 2-hour tutorial, with all code included!
- 96. AR DEVELOPMENT ON ANDROID Framework: ARCore User system requirements:? Higher-end Android device from 2017 era? (Google Pixel 2, Samsung Galaxy S9,? and similar) Development environment:? Android Studio IDE,? Kotlin or Java programming language
- 97. CROSS-PLATFORM AR DEVELOPMENT Framework: ARKit, with ARCore support coming soon User system requirements:? ARKit-compatible iOS device /? ARCore-compatible Android device Development environment:? Unity IDE, C# programming language
- 98. WEB-BASED AR DEVELOPMENT Framework: JavaScript library that accesses ARKit and ARCore User system requirements:? ARKit-compatible iOS device /? ARCore-compatible Android device Development environment:? Your favorite JavaScript IDE,? JavaScript programming language
- 99. Joey deVilla (@AccordionGuy)? joey@joeydevilla.com Anitra Pavka (@APavka)? anitra@anitrapavka.com