Realizing the Metaverse
Description
A guide to the challenges in making virtual reality, reality
The Metaverse, a version of the internet in which online interactions take place in real time within fully realized virtual spaces, has been promised as the next frontier in wireless communication. It has drawn huge investment from Silicon Valley and widespread media attention. However, the technologies required to make the Metaverse a reality are still in their infancy, and significant barriers must be overcome if this massive step is to be taken.
Realizing the Metaverse provides a systematic overview of these challenges and their likely solutions. Focusing on five key areas-infrastructure, access, intelligence, security, and future developments-it offers one of the first comprehensive, formalized treatments of the Metaverse as a nascent reality. It promises to be an integral contribution to the future development of Metaverse technologies.
Realizing the Metaverse readers will also find:
- An editorial team with extensive research experience in the field
- Detailed discussion of topics such as augmented reality (AR) adaptation, haptic feedback, artificial intelligence, and more
- Enlightening discussion of open questions and future prospects for research
Realizing the Metaverse is ideal for graduate and advanced undergraduate students in wireless technology, network communications, and related fields, as well as for researchers and industry professionals involved with the Metaverse or adjacent technologies.
More details
Other editions
Additional editions
Persons
Wei Yang Bryan Lim, PhD, is a Wallenberg-NTU Presidential Postdoctoral Fellow, Nanyang Technological University, Singapore. He has received Best Paper Awards from the IEEE Wireless Communications and Networking Conference and the IEEE SPCC Technical Committee, and regularly serves as a reviewer for leading journals.
Zehui Xiong, PhD, is an Assistant Professor at Singapore University of Technology and Design. He has published extensively, won many prestigious career/paper awards, and served as the editor for many leading journals in the areas of Internet of Things, edge computing and intelligence. He serves as the Associate Director of Future Communications R&D Programme. He was featured on the list of Forbes Asia 30 under 30 in 2023.
Dusit Niyato, PhD, is a Professor in the School of Computer and Engineering, Nanyang Technological University, Singapore. He is an IEEE Fellow and serves as editor of numerous prestigious journals, including as editor-in-chief of IEEE Communications Surveys and Tutorials.
Junshan Zhang, PhD, is a Professor in the Electrical and Computer Engineering Department at the University of California, Davis, USA. He has researched and published extensively on information networks, data science, 5g, wireless communications, and related subjects, and currently serves as editor-in-chief for IEEE Transactions on Wireless Communication.
Xuemin (Sherman) Shen, PhD, is a University Professor in the Department of Electrical and Computer Engineering, University of Waterloo, Canada. He has published extensively on network resource management, wireless network security, Internet of Things, 5G, and more. He is a Fellow of the Engineering Institute of Canada, the Canadian Academy of Engineering, and the Royal Society of Canada, among others.
Content
About the Editors xiii
List of Contributors xvii
Foreword xix
Preface xxi
1 Introduction 1 Wei Yang Bryan Lim, Zehui Xiong, Dusit Niyato, Junshan Zhang, and Xuemin Sherman Shen
1.1 Introduction 1
1.1.1 Birth of the Metaverse 1
1.2 Architecture, Developments, and Tools of the Metaverse 4
1.2.1 Definition and Architecture 4
1.2.2 Current Developments 9
1.2.2.1 Massive Multiplayer Online Role-playing Games 10
1.2.2.2 Applications of the Metaverse 11
1.2.3 Tools, Platforms, and Frameworks 13
1.2.4 Summary and Lessons Learned 16
1.2.4.1 The Metaverse Cannot Be Realized by a Standalone Technology 16
1.2.4.2 Implementation Challenges 16
1.2.4.3 Standardized Protocols, Frameworks, and Tools 17
1.2.4.4 Metaverse Ecosystem and Economics 17
2 Communication and Computing in Edge-enabled Metaverse 19 Minrui Xu, Wei Chong Ng, Wei Yang Bryan Lim, and Dusit Niyato
2.1 Introduction 19
2.1.1 Rate-Reliability-Latency 3D Multimedia Networks 21
2.1.1.1 Resource Allocation for VR Streaming 25
2.1.1.2 Resource Allocation for AR Adaptation 28
2.1.1.3 Edge Caching for AR/VR Content 30
2.1.1.4 Hologram Streaming 31
2.1.2 Human-in-the-loop Communication 32
2.1.2.1 URLLC for the Tactile Internet 33
2.1.2.2 Semantic/Goal-aware Communication 34
2.1.3 Real-time Physical-Virtual Synchronization 36
2.1.3.1 Resource Allocation for Physical-Virtual Synchronization 37
2.1.3.2 Intelligent Edge Network-aided Virtual-Physical Synchronization 38
2.1.4 Lessons Learned 40
2.1.4.1 Efficient Immersive Streaming and Interaction 40
2.1.4.2 AI for Intelligent Edge Communication 40
2.1.4.3 Context-aware Immersive Content Delivery 40
2.1.4.4 Self-sustainable Physical-Virtual Synchronization 40
2.1.4.5 Decentralized Incentive Mechanism 41
2.2 Computation 41
2.2.1 The Cloud-Edge-End Computing Paradigm 43
2.2.2 Efficient AR/VR Cloud-Edge-End Rendering 45
2.2.2.1 Stochastic Demand and Network Condition 46
2.2.2.2 Stragglers at Edge Networks 48
2.2.2.3 Heterogeneous Tasks 49
2.2.3 Scalable AI Model Training 49
2.2.3.1 Parameter Pruning and Quantization 50
2.2.3.2 Low-rank Approximation 51
2.2.3.3 Knowledge Distillation 52
2.2.4 Privacy and Security 54
2.2.4.1 Trusted Execution Environment 55
2.2.4.2 Federated Learning 56
2.2.4.3 Adversarial Machine Learning 58
2.2.5 Lessons Learned 60
2.2.5.1 Adaptive AR/VR Cloud-Edge-End Rendering 60
2.2.5.2 On-demand and Generalized Model Compression 60
2.2.5.3 User-centric Computing 60
2.2.5.4 Secure Interoperable Computing 61
2.2.5.5 From Distribution to Decentralization 61
2.3 Summary 61
3 Advanced and Future Network Access Technologies for the Metaverse 63 Andrea Garbugli, Domenico Scotece, Luca Foschini, and Paolo Bellavista
3.1 Introduction 63
3.2 Edge-enabled Metaverse 64
3.2.1 Efficient Service Migration at the Edge 65
3.2.1.1 Multi-layer Container Migration 66
3.2.1.2 Metaverse-enabled Service Migration Architecture 68
3.2.1.3 Simulation Results 71
3.3 Ultra-low Latency Communications 73
3.3.1 ULL Networking Protocols 73
3.3.1.1 Time-sensitive Networking (TSN) 74
3.3.1.2 Wireless Networks: 5G and Wi-Fi 6 74
3.3.2 Acceleration Technologies 76
3.3.2.1 Linux eXpress Data Path (XDP) 76
3.3.2.2 Data Plane Development Kit (DPDK) 77
3.3.2.3 Remote Direct Memory Access (RDMA) 77
3.3.2.4 Integration for Metaverse Applications 78
3.3.3 SELENE: A Uniform Acceleration Middleware 78
3.3.3.1 SELENE: The APIs 79
3.3.3.2 SELENE: QoS Policies 81
3.3.3.3 SELENE: The Runtime 82
3.3.3.4 SELENE: Evaluation 85
3.3.3.5 Latency and Throughput Benchmarks 86
3.3.3.6 Lunar Streaming 90
3.4 Summary 92
4 How to Intelligentize the Metaverse 93 Zhe Zhang, Jialin Wu, Ziwen Jin, Xidong Wang, Jiawen Kang, Yanchao Zhao, and Yi Wu
4.1 Seven-layer Model of the Intelligentized Metaverse 93
4.1.1 Experience Layer 93
4.1.2 Exploration and Discovery Layer 94
4.1.3 Creator Economy Layer 95
4.1.4 Spatial Computing Layer 95
4.1.5 Decentralized Layer 95
4.1.6 Human-Computer Interaction Layer 96
4.1.7 Infrastructure Layer 96
4.2 AI for the Metaverse 96
4.2.1 Avatar Creation 96
4.2.2 Empowering 3D Chatbots 97
4.2.3 Language Processing 97
4.2.4 Enabling Human-Computer Interactions 97
4.2.5 3D Scene Rendering 98
4.3 Edge AI for the Metaverse 98
4.3.1 Edge AI Architectures 98
4.3.1.1 Cloud-edge Collaboration Architecture 98
4.3.1.2 Decentralized Architecture 99
4.3.1.3 Federated Learning Architecture 100
4.3.2 How Does Edge AI Contribute to Creating the Metaverse? 100
4.3.2.1 Unique Challenges in the Edge AI-empowered Metaverse 100
4.3.2.2 Benefits of Edge AI for the Metaverse 101
4.4 Edge AI-empowered Metaverse 102
4.4.1 Cloud-edge-end Empowered Metaverse Network 102
4.4.2 Mobile-edge-cloud Empowered Metaverse Network 103
4.4.3 Decentralized Edge AI Empowered Metaverse Network 104
4.4.4 Personalized and Heterogeneous Edge AI Empowered Metaverse Network 105
4.4.5 Space-Air-Ground Empowered Metaverse Network 106
4.5 AI and Edge AI Empowered Metaverse Applications 107
4.5.1 Telecommuting 107
4.5.2 Immersion Education 108
4.5.3 Smart Cities 108
4.5.4 Industrial Metaverse 109
4.5.5 VR Tourism 109
4.5.6 Digital Virtual Human 109
4.6 Open Research Topics and Future Directions 110
4.6.1 Ownership for AI-Generated Content 110
4.6.2 Deepfakes and User Transparency 110
4.6.3 The Right to be Forgotten 110
4.6.4 Balance Between Virtual and Reality 111
4.6.5 Security 111
4.7 Summary 112
5 How to Secure the Metaverse 113 Zefeng Chen, Jiayi Sun, Jiayang Wu, Wensheng Gan, and Philip S. Yu
5.1 Introduction 113
5.2 Security Issues in the Metaverse 117
5.2.1 Data Collection 118
5.2.1.1 Solutions 119
5.2.2 Data Transmission and Network Communication 119
5.2.2.1 Solutions 121
5.2.3 Data Processing 122
5.2.3.1 Solutions 122
5.2.4 Data Storage and Access 123
5.2.4.1 Solutions 123
5.3 Privacy Issues in the Metaverse 124
5.3.1 Data Collection 125
5.3.1.1 Solutions 125
5.3.2 Data Usage and Access 126
5.3.2.1 Solutions 126
5.3.3 Data Transmission, Storage, and Sharing 127
5.3.3.1 Solutions 127
5.4 Open Problems and Opportunities 128
5.4.1 Security Issues of 5G 128
5.4.2 Security Issues of the Internet of Things 129
5.4.3 Security Issues of Extended Reality Technology 130
5.4.4 Security Issues of Cloud Computing 131
5.5 Summary 132
Bibliography 133
Index 173
2
Communication and Computing in Edge-enabled Metaverse
Minrui Xu1, Wei Chong Ng2, Wei Yang Bryan Lim2, and Dusit Niyato1
1School of Computer Science and Engineering, Nanyang Technological University, Singapore, Singapore
2Alibaba Group and Alibaba-NTU Joint Research Institute, Nanyang Technological University, Singapore, Singapore
2.1 Introduction
It is anticipated that mobile edge networks will provide users effective networking and communication capabilities along with fast, low-latency, and expansive wireless coverage, enabling users to fully immerse themselves in the Metaverse and have a seamless, real-time experience [104]. Strict communication criteria must be fulfilled in order to do this (Table 2.1). In contrast to the conventional 2D picture transmission, the existing 5G network infrastructure is heavily taxed by the requirement to send 3D virtual objects and situations in the Metaverse in order to provide consumers an immersive experience. In order to provide immersive content delivery and real-time engagement, for instance, the Metaverse for mobile edge networks, as suggested in [193], shows that enormous connection of Metaverse enablers, such as augmented reality (AR)/virtual reality (VR), tactile Internet, avatars, and Digital twin (DT), requires huge bandwidth and URLLC. Furthermore, Ref. [413] proposes a blockchain-based Metaverse-native communication system that offers anonymous and decentralized connectivity for all virtual and physical entities concerning encrypted addresses. Real-time execution of production choices and virtual concerts are required. All things considered, the following aspects of the Metaverse pose serious new difficulties for mobile edge networks when it comes to offering Metaverse services:
- Immersive 3D Streaming: Millions of users at mobile edge networks may enjoy the full Metaverse experience thanks to 3D streaming, which dissolves the lines between the real and virtual worlds.
Table 2.1 Communication requirements of services in the Metaverse.
Services Reliability (%) Latency (ms) Data Rate (Mbit/s) Connection Density (devices/) VR entertainment 7-15 250 1000-50,000 The tactile Internet 1 1 1000-50,000 Digital twin of smart city 5-10 10 100,000 AR smart healthcare 5 10,000 50 Hologram education (point cloud) 20 500-2000 1000 Hologram education (light field) 20 to 1000 - Multisensory Communications: The Metaverse is made up of several 3D virtual worlds where users may immerse themselves as haptic, kinesthetic, auditory, and visual avatars [52]. Thus, in order for edge users to access 3D multimedia services, including AR/VR and the tactile Internet, at any time or location, they must have ubiquitous connection to the Metaverse.
- Real-time Interaction: Users' real-time interactions with their avatars in the Metaverse are based on massive forms of interactions, such as communications between humans (H2H), machines (H2M), and machines (M2M). Consequently, in real-time interactions, social multimedia services in the Metaverse must be provided under strict guidelines, including motion-to-photon latency [449], interaction latency, and haptic perception latency.
- Seamless Physical-Virtual Synchronization: Both virtual and physical entities must interact and exchange real-time status updates in order to offer smooth synchronization services between them. The age of information (AoI) or the value of information (VoI) may have an impact on the significance and value of synchronizing data [314].
- Multidimensional Collaboration: Maintenance of the Metaverse requires multifaceted cooperation between physical service providers (PSPs) and virtual service providers (VSPs) in the real and virtual worlds [299]. For instance, as covered in Section 1.2.1, in order to guarantee the accuracy of the data used for P2V synchronization, the physical entities in edge networks, such as sensor networks run by PSPs, must routinely gather data from the outside world. Decisions made in virtual environments can be translated into actions in the real world in exchange. Collaboration between several real and virtual entities in various dimensions-such as time and space-is necessary to complete this cycle.
In this chapter, we evaluate state-of-the-art networking and communication options for the edge-enabled Metaverse in order to address these concerns. In order to fully engage with the Metaverse, users must first be provided with seamless access to 3D multimedia services (Section 2.1.1). This enables continuous synchronization between users and physical entities with virtual worlds and includes support for VR streaming and AR adaption. User-centric considerations in the content distribution process should be given priority in communication and networking that supports the Metaverse, going beyond traditional content delivery networks, as demonstrated in Section 2.1.2. The Metaverse's abundance of contextual and tailored content-based services, such as AR/VR and the tactile Internet, is the reason behind this. In addition, the exponential increase of data and the finite bandwidth need a paradigm change from information theory's traditional focus. We examine goal-oriented and semantic communication solutions [424] in response, which can help to ease the spectrum shortage for next-generation multimedia services. Lastly, as will be covered in Section 2.1.3, real-time bidirectional physical-virtual synchronization, or DT, is necessary for the development of the Metaverse. Reconfigurable intelligent surfaces (RIS) and unmanned aerial vehicles (UAVs) are examples of intelligent communication infrastructures that will be used in the sensing and actuation interaction between the physical and virtual worlds. Table 2.2 contains a quick overview of the evaluated articles in terms of scenarios, issues, performance metrics, and mathematical tools. We also provide the instrumental mathematical tools and human-oriented metrics in Figure 2.1.
2.1.1 Rate-Reliability-Latency 3D Multimedia Networks
In terms of transmission rate, reliability, and latency, the Metaverse's seamless and immersive embodied telepresentation via AR/VR lays demanding requirements on mobile edge networks' communication and network infrastructure [328]. Specifically, the large amount of data sent to enable AR/VR services that move between the real and virtual worlds necessitates an all-encompassing performance from the edge networking and communication infrastructure that maximizes the trade-off between latency, throughput, and dependability.
The first factor to take into account is the data rate that facilitates round-trip interactions between the Metaverse and the real world (e.g., for users and sensor networks supporting the synchronization of the virtual and physical worlds) [148, 217, 262]. Another significant obstacle that consumers must overcome to perceive realism in AR/VR services is interaction latency [430]. For instance, mobile edge networks enable players to seamlessly engage in massively multiplayer online games, which depend on extremely low latency. The rationale is that latency controls the speed at which players' replies to other players and the information they get about their position in the virtual worlds. The third dimension pertains to the dependability of physical network services, namely the frequency of break-in-present (BIP) experienced by users establishing connections to the Metaverse (see [370]). Furthermore, the criteria for dependability of AR/VR applications and users could change over time, making it more difficult for VSPs and PSPs to allocate edge resources for the provision of AR/VR services.
Table 2.2 Summary of scenarios, problems, performance metrics, and mathematical tools for AR/VR, the tactile Internet, hologram streaming, and semantic communications.
References Scenarios Problems Performance Metrics Mathematical Tools [129] MBMS over 5G Physical layer design and analysis VR service flexibility, capacity, and coverage Flexible numerology and configurations [111] Interactive mmWave-enabled VR gaming mmWave resource allocation Interaction latency Matching theory [274] D2D-enhanced MBMS with single frequency D2D radio resource management Aggregate data rate and short-term fairness Greedy algorithm and iterative search algorithm [396] Indoor VR over THz/VLC wireless network Virtual access partitioning (VAP) selection and user association VR service reliability and...