While consumer AR is still some time away and the technology itself has not yet fully solved some of the challenges in rendering realistic digital objects, we might start to think about how a fully realistic and multimodal AR could feel. In doing so, we can use evidence from trials of pilot systems and from psychological experiments on perception and neuroplasticity, the brain’s power to adapt to new kinds of sensory input.
Photo: Makia Minich
Relatively early studies with AR displays highlighted the risk of division of attention (or “attention tunnelling” onto the virtual display), with people missing important natural cues due to attending to the virtual ones. This led to the recommendation that AR be on-demand (with hands-free activation) and that it should be minimal and integrate fully with the scene (Goldiez, 2007) or the “far domain” (Edgar, 2007). We might forsee cues being extremely subtle and perhaps subliminal in some cases, in order to avoid attentional issues.
A promise of AR is that interaction with virtual objects can be as natural and intuitive as normal object manipulation, and the presense of real objects alongside enhance performance and the feeling of presence (Billinghurst et al, 2015). We can anticipate the emergence and constant innovation of social interaction norms that are easy to learn and that relate to everyday object interactions and virtual/real object pairings.
Other features of overlays could confer significant cognitive improvements. Trials of Google Glass with older adults, for instance, showed that the camera feature was attractive as an adjunct to memory (photos with a timestamp allowed people to check back on the day). Also, overlaid instructions for unfamiliar processes were seen as very useful. Of course, without a good way to browse these, it can become frustrating to access the right task or step in a process (Kunze et al, 2014). We can see here that contextual cues and intelligent navigation will contribute a lot to experiences of usefulness.
This all indicates the potential for subtle, natural blending with the world and for new capabilities to be assimilated relatively easily. We know from work on neuroplasticity that new kinds of sense data can be organised by the adaptive brain over time, and come to feel both natural and relied upon:
“I suddenly realized that my perception had shifted. I had some kind of internal map of the city in my head. I could always find my way home. Eventually, I felt I couldn’t get lost, even in a completely new place.” - Udo Wächter, quoted in Bains (2007), after adaping to a direction-sensing vibration belt.
Of course, this also brings the risk that a person would feel sensorially impoverished if that capability is taken away again.
Finally, while a large amount of work has focussed on visual perception (sometimes combined with haptic feedback), there remains a huge area of opportunity in virtualising and blending the other senses. In an augmented environment, interactions between digital and physical elements can also be imagined, making the scene more natural and hightening the senses toward natural phenomena (Shraffenberger & Van Der Heide, 2016).
References
Bains, S. 2007. Mixed Feelings. Wired.com https://www.wired.com/2007/04/esp/
Billinghurst, M., Clark, A. & Lee, G., 2015. A Survey of Augmented Reality. Foundations and Trends in Human-Computer Interaction, 8(2–3), pp.73–272.
Edgar, G.K., 2007. Accommodation, cognition, and virtual image displays: A review of the literature. Displays, 28(2), pp.45–59.
Goldiez, B.F., Ahmad, A.M. & Hancock, P.A., 2007. Effects of augmented reality display settings on human wayfinding performance. IEEE Transactions on Systems, Man and Cybernetics Part C: Applications and Reviews, 37(5), pp.839–845.
Kunze, K., Henze, N. & Kise, K., 2014. Wearable computing for older adults – Initial Insights into Head-Mounted Display Usage. In Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing Adjunct Publication. Seattle, pp. 83–86. Available at: http://dl.acm.org/citation.cfm?doid=2638728.2638747.
Schraffenberger, H. & van der Heide, E., 2016. Multimodal augmented reality: the norm rather than the exception. Proceedings of the 2016 workshop on Multimodal Virtual and Augmented Reality - MVAR ’16, pp.1–6. Available at: http://delivery.acm.org/10.1145/3010000/3001960