1
Remote Collaboration on Physical Tasks

1.1 Introduction

Remote collaboration on physical tasks (or remote guidance/remote assistance) typically involves one or more remote helpers guiding one or more local workers to work collaboratively on the manipulation of physical objects [1, 2]. In this type of remote collaboration, both the workers and helpers are physically distributed. On the one hand, the workers have direct access to the physical objects to be worked on but do not have full skills or knowledge on how to operate or manipulate them; thus, they need to receive help from the remote helpers. On the other hand, the remote helpers know how, but do not have physical access to the objects [3]. Technologies that support remote guidance can greatly improve the productivity and safety of tasks by allowing experts to provide timing guidance and training to individuals remotely without having to travel on‐site. It has a wide range of applications in industrial domains (e.g. [4]) and has the potential to revolutionize those industries in terms of how the business operates and how service can be provided to their customers, from manufacturing and construction to healthcare and education, to name a few.

With recent advances in networking, augmented reality (AR), virtual reality (VR), mobile and wearable technologies, it has become increasingly possible in practice to enable helpers to remotely guide individuals in performing complex physical tasks with precision and efficiency [5]. Given the increasing demand for remote guidance technologies from industries and increasing interest and effort in research from academics, this research book explores the latest and typical developments in remote guidance technologies and provides comprehensive reviews of the current state‐of‐the‐art research in this field, including our own research findings and developments in the past 15 years.

1.2 Remote Collaboration in Perspective

The rest of the book has 12 chapters, each focusing on a specific aspect of remote collaboration research. Both technology and communication are essential elements of remote collaboration, and understanding whether and how technology impacts communication behaviors is important for the design of remote collaboration systems. However, this is an area that has not been well‐researched. The technology impact can be predicted using communication models. Thus, we dedicate the next chapter of our book to the discussion of how existing communication models can be used to predict the impact of different AR technologies in remote collaboration and if a new communication model needs to be developed. More specifically, we provide a review of various existing communication models and show how they can be used to analyze communication in both AR and non‐AR interfaces for remote guidance on physical tasks. We also discuss the limitations of current models, identify research gaps, and explore possible further developments.

The third chapter provides a review of communication cues in remote collaboration. It starts with an overview of the research landscape over the past three decades and then investigates the communication context based on which a remote collaboration is conducted. We categorize communication cues in remote collaboration systems as verbal, visual, haptic, and empathic communication cues and review the systems and experiments that studied each of them to identify advantages and limitations under different situations. Finally, we summarize and address the challenges of multimodality communication modeling and system design for high usability and suggest potential future research directions for augmented remote collaboration system design aiming at effectiveness, reliability, and ease of use.

For remote guidance on physical tasks, in addition to verbal communications, how to convey other communication cues effectively has been researched extensively. Given the importance and variety of possible communication cues as outlined in the third chapter, we presented a review in the fourth chapter to summarize the communication cues being used, approaches that implement the cues, and their effects on remote guidance on physical tasks [6]. In this chapter, we categorize the communication cues into explicit and implicit ones and report our findings. Our review indicates that a number of communication cues have been shown to be effective in improving system usability and helping collaborators to achieve optimal user experience and task performance. More specifically, there is a growing interest in providing a combination of multiple explicit communication cues to cater for the needs of different task purposes and in providing combination of explicit and implicit communication cues.

Although technology for remote collaboration is becoming increasingly more essential and affordable, and eye gaze is an important cue for human–human communication, there is much that remains to be done to explore the use of gaze in remote collaboration, especially for collaboration on physical tasks. Recent advancement in eye tracking technologies enables gaze input to be added to collaborative systems, especially for remote guidance and is expected to bring more promising opportunities to reduce misunderstanding and improve effectiveness. The fifth chapter surveys publications with respect to eye tracking‐supported collaborative physical work under remote guidance. We categorize the prototypes and systems presented according to four metrics ranging from eye‐tracked subjects to gaze visualization. Then, we summarize the experimental and investigation findings to have an overview of the eye tracking mechanism in remote physical collaboration systems, as well as the roles that eye gaze and its visualization play in common understanding, referential, and social copresence practices.

The sixth chapter provides a summary of how to conduct evaluation studies of AR‐based remote guidance systems. As previously discussed in this book, communication is an essential part of remote collaboration, and many technologies have been developed to enable people to better connect and communicate with one another. However, the impact of these technologies can only be measured through conducting evaluation studies and measuring how the technologies change communication behavior between real people. Therefore, the purpose of this chapter is to help the readers become more proficient in their own evaluation studies and create research outputs that will inspire others in the field. More specifically, in this chapter, we present evaluation case studies, derive a number of design guidelines, and discuss methods that can be used to create robust evaluation studies. Finally, this chapter concludes with a list of possible research directions.

From the seventh chapter, we introduce a range of typical remote guidance systems. These systems were developed with different configurations to meet different collaboration requirements and to serve as platforms for us to investigate specific research questions. First, in this chapter, we present a remote guidance system called HandsOnVideo [7], a system that uses a near‐eye display to support mobility and unmediated representations of hands to support remote gestures, enabling a remote helper guiding a mobile worker working in nontraditional‐desktop environments. The system was designed and developed using a participatory design approach, which allowed us to test and trial a number of design ideas. It also enabled us to understand from a user's perspective some of the design tradeoffs. The usability study with end users indicated that the system is useful and effective. The users were also positive about using the near‐eye display for mobility and instructions and using unmediated representations of hands for remote gestures.

The eighth chapter introduces HandsInAir [8], a wearable system for remote guidance. This system is designed to support the mobility of the collaborators and provide easy access to remote expertise. HandsInAir draws on the richness of hand gestures for remote guiding and implements a novel approach that supports unmediated remote gestures and allows the helper to perform natural gestures by hands without the need for physical support. A usability study was also conducted demonstrating the usefulness and usability of HandsInAir. More specifically, the participants were positive about the mobility support provided by the system to the collaborators. According to their feedback, the mobility support allows workers to access a remote helper more easily. Also, helpers are enabled to continuously engage with the system and their partner when they move around during the guiding process. Participants who played the role of helper also considered gesturing in the air as being intuitive and effective.

The ninth chapter introduces HandsInTouch [9], which supports a unique remote collaboration gesture interface by including both raw hand gestures and sketch cues on a live video or still images. We also conducted a user study comparing remote collaboration with the interface that combines hand gestures and sketching (the HandsInTouch interface) to one that only used hand gestures when solving two tasks: Lego assembly and repairing a laptop. It was found from the study that adding sketch cues improved the task completion time, only with the repairing task, as this had complex object manipulation, and that using gesture and sketching together created a higher task load for the...