1
Overview of Virtual and Augmented Reality: Toward Immersive and Interactive Experiences

1.1 Introduction

Welcome to a journey into the future of human-computer interactions, where the boundaries between reality and imagination blur, and our sensory perceptions are enhanced and augmented. In this first chapter of the book, we will present an overview of the key concepts in virtual reality (VR) and augmented reality (AR) technologies that promise to revolutionize the way we experience and interact with multimedia information and the world around us. As we explore the landscapes these technologies create, we will discover how VR and AR extend our digital capabilities and craft new dimensions of immersive sensory-perceptual experiences. We will explore how these technologies can redefine our relationship with both the digital and physical worlds, offering unprecedented opportunities for innovation and creativity.

The evolution of VR and AR has been marked by significant advancements in transduction, computation, and communication technologies, which have collectively propelled these immersive and interactive experiences from conceptual ideas to practical applications. From gaming and entertainment to education, healthcare, and industry, the applications are vast and varied, promising to enhance every aspect of our lives. This chapter will provide an introduction to these transformative technologies, laying the groundwork for a deeper understanding of their principles, potential, and impact.

As we embark on this exploration, we will examine the technical foundations of virtual and augmented reality and consider the implications these technologies hold for the future. By understanding the mechanisms behind sensory-perceptual immersion, we can appreciate how these technologies can create more engaging and realistic experiences, as well as the significant challenges associated with this endeavor.

We start by reviewing the key concepts and terminologies that are widely used in the field of immersive technologies. Then, we discuss the goals of the book and present an overview of the structure and content flow that will follow.

1.2 Key Concepts in Immersive Technologies

The landscape of immersive technologies has expanded rapidly over the past few decades, driven by advancements in computing, graphics, displays, audio, sensors, and haptics technologies. These innovations have given rise to a range of new tools and platforms that transform how we interact with digital content and the physical world. Central to this transformation are the concepts of VR, AR, merged or mixed reality (MR), extended reality (XR), the metaverse, and spatial computing. Each of these concepts represents unique capabilities and experiences, but they also share common goals: enhancing user immersion, interactivity, and engagement.

Virtual Reality refers to technologies that create a fully immersive digital environment, effectively replacing the user's real-world surroundings with a computer-generated simulation (Burdea & Coiffet, 2003; Sherman & Craig, 2018; Greengard, 2019; LaValle, 2023). This is typically achieved through the use of head-mounted displays (HMDs), which present the user with a stereoscopic view and spatial sound of the virtual world, along with motion tracking systems that capture and interpret the user's movements and map them into the virtual environment. Additional sensory input devices, such as gloves, controllers, and even haptic suits, can enhance the interactivity and realism of the experience. The primary goal of VR is to create a convincing sense of presence, making users feel as though they are truly inside the simulated environment. VR is widely used in various fields, including gaming, where it offers immersive gameplay experiences; simulations for training and education, providing safe and controlled environments for practicing skills; virtual tourism, allowing users to explore distant or inaccessible locations; therapy and rehabilitation, offering controlled environments for therapeutic interventions; social interaction platforms, enabling virtual meetups and collaborative activities; and numerous other immersive applications.

Augmented Reality overlays digital information onto the real-world environment, enhancing and augmenting the user's perception of reality without replacing it (Craig, 2013; Schmalstieg & Hollerer, 2016; Chen et al., 2019). AR devices, which can include smartphones, tablets, smart glasses, and head-up displays (HUDs), project digital elements such as images, videos, sounds, or data onto the user's view of the real world. Unlike VR, which immerses the user in a completely virtual environment, AR integrates digital content with the physical world, providing contextual information and interactive elements that users can see and interact with. AR enhances real-world experiences by adding layers of information that can be accessed in real time. For example, AR can be used in navigation to overlay directions onto the real-world streets, in retail to allow virtual try-ons of clothing or accessories, in education to bring learning materials to life with interactive content, in industrial applications to provide maintenance and repair guidance, and in entertainment through AR games which overlay digital creatures onto real-world environments. AR's ability to blend digital and physical worlds opens up numerous possibilities for enhancing how we interact with our surroundings.

Mixed Reality, a subset of Extended Reality and sometimes referred to as Merged Reality, blends physical and digital worlds, allowing real and virtual elements to coexist and interact seamlessly in real time (Milgram & Kishino, 1994; Wang & Schnabel, 2008; Ohta & Tamura, 2014). MR encompasses both AR and VR, providing a spectrum where digital and real-world objects can interact dynamically. Unlike traditional AR, which can be limited to overlaying digital content onto the real world, MR integrates and anchors digital objects to the physical environment, allowing for more natural and intuitive interactions. MR devices incorporate advanced hardware which can understand and respond to the physical environment through a combination of cameras, sensors, and advanced computing power. This technology enables users to manipulate and interact with both real and virtual elements in a cohesive experience, creating a more immersive and interactive environment. MR is utilized in various fields such as collaborative design, where multiple users can interact with virtual prototypes in a shared physical space; remote assistance, where experts can provide guidance overlaid onto the physical objects being worked on; interactive storytelling, where narratives can unfold around the user; education, providing interactive and engaging learning experiences; and complex data visualization, where abstract data can be visualized in a spatial context.

The Metaverse is the concept of an expansive shared simulated space created by the convergence of physically persistent virtual reality and virtually augmented physical reality (Jaynes et al., 2003; Hazan, 2010; Ball, 2022; Park & Kim, 2022). It represents an interconnected network of immersive digital environments where users can interact, socialize, create, and transact using avatars or digital representations of themselves. The metaverse is envisioned as a successor to the current internet, with an emphasis on shared, persistent, and immersive experiences. Unlike traditional online experiences, which are often isolated and platform-specific, the metaverse aims to be an interconnected and persistent virtual space where activities and interactions continue regardless of individual user presence. Users can move seamlessly between various virtual worlds and environments, each with its own unique characteristics and purposes. The metaverse has the potential to impact numerous aspects of life, including social interactions, where virtual meetups and events can take place; gaming, providing vast and immersive virtual worlds; virtual economies, where users can trade digital goods and services; remote work, offering virtual office spaces and collaboration tools; education, providing immersive learning environments; and entertainment, with virtual concerts, movies, and interactive experiences. Early examples of metaverse concepts include platforms like Second Life, Decentraland, and Meta's Horizon Worlds, which offer glimpses into the potential of these interconnected virtual spaces.

Spatial Computing refers to technologies that enable computers to understand and interact with the physical world in three dimensions, integrating digital information with the user's environment in a way that feels natural and intuitive (Zambonelli & Mamei, 2005; Shekhar & Vold, 2020; Hackl & Cronin, 2024; Xu et al., 2024). It encompasses a range of technologies, including AR, VR, MR, and other sensor-driven environments that recognize the spatial relationship between objects and the environment. Spatial computing allows for more intuitive and natural user interactions with digital content by understanding and responding to the physical context of the user. Devices equipped with spatial computing capabilities can map and understand the physical environment, recognizing objects, surfaces, and spatial relationships. This spatial awareness enables dynamic...