Computer-Supported Collaboration
Description
Discover the latest developments in AR, VR, mobile, and wearable technologies for the remote guidance of physical tasks
In Computer-Supported Collaboration: Theory and Practice, an expert team of researchers delivers the latest instruction in using augmented reality (AR), virtual reality (VR), and mobile or wearable technology to support remote guidance on physical tasks. The authors offer an overview of the field before moving on to discuss state-of-the-art research developments in everything from shared visual spaces to the use of hand gestures and gaze information for better collaboration.
The book also describes the hardware devices, software tools, and libraries that can be used to help build remote guidance systems, as well as the industrial systems and applications that have been used in real world settings. Finally, Computer-Supported Collaboration includes a discussion of the current challenges faced by practitioners in the field and likely future directions for new research and development.
Readers will also discover:
- A thorough introduction and review of the art of remote guidance research and engineering
- Comprehensive explorations of the shared visual space used to support common grounding and the remote guidance of physical tasks, as well as mobility support for local workers
- Practical discussions of mobility support of workers and helpers in remote guidance, including systems that support hands-free interaction
- In-depth explorations of communication cues in remote guidance, including systems that support gesturing and sketching on a touch-based display
Perfect for researchers and professionals working in human-computer interaction or computer-supported collaborative work, Computer-Supported Collaboration: Theory and Practice is also an ideal resource for educators and graduate students teaching or studying in these fields.
More details
Other editions
Additional editions
Persons
Weidong Huang, PhD, is an Associate Professor at the University of Technology Sydney, Australia. He received his PhD degree in computer science from the University of Sydney. His research interests are in human-computer interaction, visualization and data science.
Mark Billinghurst, PhD, is the Director of the Empathic Computing Laboratory and a Professor at the University of South Australia in Adelaide, Australia, and also at the University of Auckland in Auckland, New Zealand. He conducts research on how virtual and real worlds can be merged, publishing over 650 papers on augmented reality, virtual reality, remote collaboration, empathic computing, and related topics.
Leila Alem, PhD, is an Adjunct Professor at the University of Technology Sydney Faculty of Engineering and Information Technology. She was a principal research scientist at the CSIRO Digital Productivity and Services based in Sydney. Her main focus of research is in the area of human factors in computer mediated collaboration settings. Drawing on cognitive psychology, social science and human-computer-interface research she has investigated the media, cognitive and social factors at play in telepresence systems and environments.
Chun Xiao, PhD, is a Research Data Insights Analyst at the University of Technology Sydney. She received her PhD in computer science from Otto-von-Guericke University Magdeburg, Germany.
Troels Rasmussen, PhD, was a Faculty Member in the Department of Computer Science at Aarhus University, Denmark, and received his PhD from the same university. His research interests are in augmented reality and human-computer interaction.
Content
About the Authors xiii
Acknowledgments xv
1 Remote Collaboration on Physical Tasks 1
1.1 Introduction 1
1.2 Remote Collaboration in Perspective 2
1.3 Book Audience 5
2 Communication Models for Remote Guidance 7
2.1 Introduction 7
2.2 Overview of Communication Models 9
2.2.1 Linear Communication Models 12
2.2.2 Nonlinear Communication Models 13
2.2.3 Summary 18
2.3 Applying Communication Models 19
2.4 Communication Behaviors in AR Conferencing 22
2.5 A Communication Model for AR 32
2.6 Conclusions 34
3 Communication Cues in Augmented Remote Collaboration 41
3.1 Introduction 41
3.2 The Research Landscape -- Trends Over Time 43
3.3 Communication in Augmented Remote Collaboration 48
3.4 Challenges 65
3.5 Future Directions 67
3.6 Conclusion 68
4 Communication Cues for Remote Guidance 81
4.1 Introduction 81
4.2 Explicit Communication Cues 84
4.3 Implicit Communication Cues 100
4.4 Challenges and Future Directions 104
4.5 Conclusion 105
5 Communicating Eye Gaze Cues in Remote Collaboration on Physical Tasks 115
5.1 Introduction 115
5.2 The Changing Research Landscape -- Research Topic Trends and Teams over the Past Two Decades 117
5.3 Categorization of System Setup Based on the Screened Publications 122
5.4 Gaze Visualization 127
5.5 Functionality of Tracked Gaze in Remote Guidance on Physical Tasks 130
5.6 Challenges of Utilizing Eye Tracking in Remote Collaboration 132
5.7 Future Directions 133
5.8 Conclusion 135
6 Evaluating Augmented Reality Remote Guidance Systems 143
6.1 Introduction 143
6.2 Evaluation Methods for Collaborative AR 148
6.3 Case Studies From Example Systems 152
6.4 Guidelines 163
6.5 Directions for Research 164
6.6 Conclusion 166
7 Supporting Remote Hand Gestures over the Workspace Video 173
7.1 Introduction 173
7.2 RelatedWork 176
7.3 HandsOnVideo 177
7.4 User Testing 183
7.5 Discussion 189
7.6 Conclusion and Future work 190
8 Gesturing in the Air in Supporting Full Mobility 195
8.1 Introduction 195
8.2 Background 197
8.3 System Overview 200
8.4 Usability Study 203
8.5 Discussion 209
8.6 Concluding Remarks and Future Work 210
9 Sharing Hand Gesture and Sketch Cues with a Touch User Interface 215
9.1 Introduction 215
9.2 RelatedWork 217
9.3 Methods and Materials 220
9.4 Results 228
9.5 Discussion 234
9.6 Limitations 237
9.7 Conclusion 238
10 Augmenting Hand Gestures in 3D Mixed Reality 243
10.1 Introduction 243
10.2 RelatedWork 245
10.3 System Overview 248
10.4 Evaluation 253
10.5 A Comparison of User Ratings between HandsInAir and HandsIn3D 260
10.6 Conclusion and Future Work 262
11 Supporting Tailorability to Meet Individual Task Needs 269
11.1 Introduction 269
11.2 Component-Based Design of RemoteAssistKit 271
11.3 Identifying Tailorable Aspects of Remote Assistance 276
11.4 How Users Tailor Remote Assistance 281
11.5 The Importance of Nonverbal Guidance Depends on the Knowledge Relationship 287
11.6 Sharing of Machine Sounds Is Important for Remote Troubleshooting 288
11.7 High-Resolution Views Are Important for Remote Product Quality Optimization 289
11.8 The Manufacturing Context Poses a Challenge for Creating 3D Reconstructions with Depth Cameras 290
11.9 Multiple Cameras SupportWorkspace Awareness in Large Industrial Task Spaces 290
11.10 Concluding Remarks 293
12 Supporting Workspace Awareness with Augmented Reality-Based Multi-camera Visualization and Tracking 299
12.1 Introduction 299
12.2 Augmented Reality for Supporting Awareness During Multi-camera Remote Assistance 302
12.3 Future Research on Multi-camera Remote Assistance 314
12.4 Discussion of 2D vs. 3DWorkspace Information 320
12.5 Concluding Remarks 321
13 Industrial Applications, Current Challenges, and Future Directions 327
13.1 Introduction 327
13.2 Remote Guidance Systems 327
13.3 Technical, Ethical, and Research Challenges and Future Directions 331
13.4 Conclusion 338
References 339
Index 343
1
Remote Collaboration on Physical Tasks
1.1 Introduction
Remote collaboration on physical tasks (or remote guidance/remote assistance) typically involves one or more remote helpers guiding one or more local workers to work collaboratively on the manipulation of physical objects [1, 2]. In this type of remote collaboration, both the workers and helpers are physically distributed. On the one hand, the workers have direct access to the physical objects to be worked on but do not have full skills or knowledge on how to operate or manipulate them; thus, they need to receive help from the remote helpers. On the other hand, the remote helpers know how, but do not have physical access to the objects [3]. Technologies that support remote guidance can greatly improve the productivity and safety of tasks by allowing experts to provide timing guidance and training to individuals remotely without having to travel on-site. It has a wide range of applications in industrial domains (e.g. [4]) and has the potential to revolutionize those industries in terms of how the business operates and how service can be provided to their customers, from manufacturing and construction to healthcare and education, to name a few.
With recent advances in networking, augmented reality (AR), virtual reality (VR), mobile and wearable technologies, it has become increasingly possible in practice to enable helpers to remotely guide individuals in performing complex physical tasks with precision and efficiency [5]. Given the increasing demand for remote guidance technologies from industries and increasing interest and effort in research from academics, this research book explores the latest and typical developments in remote guidance technologies and provides comprehensive reviews of the current state-of-the-art research in this field, including our own research findings and developments in the past 15 years.
1.2 Remote Collaboration in Perspective
The rest of the book has 12 chapters, each focusing on a specific aspect of remote collaboration research. Both technology and communication are essential elements of remote collaboration, and understanding whether and how technology impacts communication behaviors is important for the design of remote collaboration systems. However, this is an area that has not been well-researched. The technology impact can be predicted using communication models. Thus, we dedicate the next chapter of our book to the discussion of how existing communication models can be used to predict the impact of different AR technologies in remote collaboration and if a new communication model needs to be developed. More specifically, we provide a review of various existing communication models and show how they can be used to analyze communication in both AR and non-AR interfaces for remote guidance on physical tasks. We also discuss the limitations of current models, identify research gaps, and explore possible further developments.
The third chapter provides a review of communication cues in remote collaboration. It starts with an overview of the research landscape over the past three decades and then investigates the communication context based on which a remote collaboration is conducted. We categorize communication cues in remote collaboration systems as verbal, visual, haptic, and empathic communication cues and review the systems and experiments that studied each of them to identify advantages and limitations under different situations. Finally, we summarize and address the challenges of multimodality communication modeling and system design for high usability and suggest potential future research directions for augmented remote collaboration system design aiming at effectiveness, reliability, and ease of use.
For remote guidance on physical tasks, in addition to verbal communications, how to convey other communication cues effectively has been researched extensively. Given the importance and variety of possible communication cues as outlined in the third chapter, we presented a review in the fourth chapter to summarize the communication cues being used, approaches that implement the cues, and their effects on remote guidance on physical tasks [6]. In this chapter, we categorize the communication cues into explicit and implicit ones and report our findings. Our review indicates that a number of communication cues have been shown to be effective in improving system usability and helping collaborators to achieve optimal user experience and task performance. More specifically, there is a growing interest in providing a combination of multiple explicit communication cues to cater for the needs of different task purposes and in providing combination of explicit and implicit communication cues.
Although technology for remote collaboration is becoming increasingly more essential and affordable, and eye gaze is an important cue for human-human communication, there is much that remains to be done to explore the use of gaze in remote collaboration, especially for collaboration on physical tasks. Recent advancement in eye tracking technologies enables gaze input to be added to collaborative systems, especially for remote guidance and is expected to bring more promising opportunities to reduce misunderstanding and improve effectiveness. The fifth chapter surveys publications with respect to eye tracking-supported collaborative physical work under remote guidance. We categorize the prototypes and systems presented according to four metrics ranging from eye-tracked subjects to gaze visualization. Then, we summarize the experimental and investigation findings to have an overview of the eye tracking mechanism in remote physical collaboration systems, as well as the roles that eye gaze and its visualization play in common understanding, referential, and social copresence practices.
The sixth chapter provides a summary of how to conduct evaluation studies of AR-based remote guidance systems. As previously discussed in this book, communication is an essential part of remote collaboration, and many technologies have been developed to enable people to better connect and communicate with one another. However, the impact of these technologies can only be measured through conducting evaluation studies and measuring how the technologies change communication behavior between real people. Therefore, the purpose of this chapter is to help the readers become more proficient in their own evaluation studies and create research outputs that will inspire others in the field. More specifically, in this chapter, we present evaluation case studies, derive a number of design guidelines, and discuss methods that can be used to create robust evaluation studies. Finally, this chapter concludes with a list of possible research directions.
From the seventh chapter, we introduce a range of typical remote guidance systems. These systems were developed with different configurations to meet different collaboration requirements and to serve as platforms for us to investigate specific research questions. First, in this chapter, we present a remote guidance system called HandsOnVideo [7], a system that uses a near-eye display to support mobility and unmediated representations of hands to support remote gestures, enabling a remote helper guiding a mobile worker working in nontraditional-desktop environments. The system was designed and developed using a participatory design approach, which allowed us to test and trial a number of design ideas. It also enabled us to understand from a user's perspective some of the design tradeoffs. The usability study with end users indicated that the system is useful and effective. The users were also positive about using the near-eye display for mobility and instructions and using unmediated representations of hands for remote gestures.
The eighth chapter introduces HandsInAir [8], a wearable system for remote guidance. This system is designed to support the mobility of the collaborators and provide easy access to remote expertise. HandsInAir draws on the richness of hand gestures for remote guiding and implements a novel approach that supports unmediated remote gestures and allows the helper to perform natural gestures by hands without the need for physical support. A usability study was also conducted demonstrating the usefulness and usability of HandsInAir. More specifically, the participants were positive about the mobility support provided by the system to the collaborators. According to their feedback, the mobility support allows workers to access a remote helper more easily. Also, helpers are enabled to continuously engage with the system and their partner when they move around during the guiding process. Participants who played the role of helper also considered gesturing in the air as being intuitive and effective.
The ninth chapter introduces HandsInTouch [9], which supports a unique remote collaboration gesture interface by including both raw hand gestures and sketch cues on a live video or still images. We also conducted a user study comparing remote collaboration with the interface that combines hand gestures and sketching (the HandsInTouch interface) to one that only used hand gestures when solving two tasks: Lego assembly and repairing a laptop. It was found from the study that adding sketch cues improved the task completion time, only with the repairing task, as this had complex object manipulation, and that using gesture and sketching together created a higher task load for the...
