Foreword by Distinguished Professor Henry Fuchs

It is a testament to the popularity of virtual reality (VR) that we have this wide‐ranging book by Grigore (Greg) Burdea and Philippe Coiffet. Only a few decades ago, even the term “virtual reality” would have been unknown to most people. Now some of the biggest technology companies in the world are vying for leadership in the field, investing heavily in VR research, development, and products.

This book contains a tremendous amount of information. It is logically organized by chapters, in an organization that is easy to follow from the unifying diagram at the beginning page of each chapter. This simple diagram is a straightforward way for the reader to relate the different chapters in the book to each other.

Chapter 1 serves as an appropriate introduction to the book. It contains basic framing of VR, as being immersive and interactive and needing imagination. This chapter also contains a quick introduction to the history of the field. It starts with the playback‐only film‐based immersive Sensorama system of Morton Heilig of the early 1960s. This pioneering system included not only a stereo display but also vibration in the user’s seat, wind (via small fans) in the user’s face, and smells to the user’s nose. The chapter then continues by briefly describing the first true VR system, Ivan Sutherland’s 1968 system, disclosed in his paper entitled, simply, “A head‐mounted three‐dimensional display,” often unfortunately misnamed by others as “The sword of Damocles” (Sutherland humorously used the term Sword of Damocles for the mechanical tube between the head‐mounted display [HMD] and the ceiling, a device that tracked the location of the HMD in the room.) The chapter continues with a brief description of an early force‐feedback (non‐HMD) interactive system of the 1970s, developed at the University of North Carolina at Chapel Hill, by a team lead by Fred Brooks. Next, the chapter describes a pioneering early‐1980s VR system at NASA Ames Research Center. Readers paying close attention will notice that there is no mention of VR systems in the 1970s. The story of VR during that “lost decade” has yet to be written.

Chapter 1 continues with a description of the first commercial VR system, by VPL, a small company led by Jaron Lanier. By offering a complete, turn‐key VR system, VPL opened the world of VR to users who could not, or did not wish to, build their own system from elementary components. Before VPL, a VR user would have had to make one’s own headset extracting, for example, video displays from small “pocket” TVs. Perhaps just as important, Jaron Lanier coined (or at least popularized) the term “virtual reality” and evangelized the vision of VR to the world. Never before had anyone, not even Ivan Sutherland, brought this vision into mainstream media, including the front pages of The New York Times and the Wall Street Journal.

Chapter 1 continues with descriptions of developments in the 1990s and on into the 21st century, including the introduction in 2019 of the Oculus Quest by Facebook/Meta. This was the successor to the 2018 Oculus Go, the first all‐in‐one VR headset. These are complete VR system in itself, that did not need to be connected to any external device (such as a PC). These systems, with introductory prices of $200 and $400, truly brought VR to the masses. These dramatic developments were triggered by the acquisition in 2014 of Oculus by Facebook for over US$2 billion, and the infusion of more billions into research and development (R&D). This was the first period in the 50‐year history of VR that so much money was poured into VR R&D. The chapter concludes with descriptions of other recent developments in VR and with an outline of the remaining chapters in the book.

Chapter 2 covers input devices for tracking human bodies and objects with a wide variety of technologies. Both research and commercial devices are extensively covered. Tracking, in real time, the position and orientation of the user, specifically of the user’s eyes, has been a necessary (and difficult) part of a VR system from the very beginning. Tracking of handheld devices for interaction has also been recognized early on as being very useful, and hand‐device tracking has been implemented in VR systems since early 1970s. The chapter covers a wide variety of these tracking technologies, from electromagnetic trackers, to camera‐based optical trackers (with and without markers), to optical sensors with laser sweeps, to camera‐based “inside‐out” optical trackers worn by the user. Chapter 2 also describes eye‐trackers inside the headset. In addition, the chapter described hybrid internal‐optical trackers as well as very large area tracking using GPS systems. Also described are game controllers and other hand‐tracking systems, as well as more unusual devices such as treadmills and neural interfaces.

Chapter 3 covers graphics displays, starting with an introduction to the human visual system and graphics display characteristics. The chapter also describes smart phones in VR displays, such as the 2015 Samsung – Oculus Gear VR. Attentive readers will notice this kind of device is a predecessor to the revolutionary all‐in‐one VR headsets like the Oculus Quest. The chapter describes several contemporary headsets, including ones capable of lower resolution near the edges of the display screen and ones with built‐in eye tracking. The chapter also includes several sections on non‐head‐worn stereo displays, both ones that require special glasses and autostereoscopic displays that give a stereo percept without the user needing to wear any special glasses. A wide range of designs is described.

Chapter 4 covers output devices for sound, haptic, and olfactory communication to the user. The chapter provides a detailed introduction to the human auditory system and digital processing needed to generate appropriate audio signals to each ear, whether by speakers in a VR headset, or speakers fixed in the room, or speakers worn on the user’s shoulders. An entire section of the chapter is devoted to haptic displays, including a tutorial on the human haptic sensing system, the design of various haptic devices, and devices for communicating hot and cold sensations to the user. The chapter also describes force feedback devices that can exert substantial forces on the user’s hands, for instance. In addition, Chapter 4 includes a section on the human olfactory system and several prototype olfactory displays.

Chapter 5 covers, in considerable detail, computer architectures for VR. The chapter starts by describing the traditional graphics pipeline. As pioneered by Sutherland’s 1968 HMD system: the 3D objects from the database are transformed (by the “Geometry Stage”) into the user’s screen space and then rendered (by the “Rasterizer Stage”) onto the screen. Next in the chapter, more modern rendering pipelines are described, ones with programmable shaders. Several generations of NVIDIA GPUs are described in some detail. Also included are description of various bottlenecks and performance optimization techniques. Desktop gaming architectures are also covered, including a recent Intel CPU design and several recent NVIDIA graphics cards. The chapter concludes with a description of NVIDIA cloud renderers and some of the heat, communication, and latency issues.

Chapter 6 covers the modeling of virtual environments, many of the issues of scanning and modeling 3D objects, both small objects such as human head and large objects such as buildings. The chapter goes into some detail on polygon counts and illumination models and the hierarchical modeling of complicated objects with articulations, such as human hands. Chapter 6 also contains an entire section on physical modeling, including collision detection and collision response techniques and behavior modeling of individual humans and multi‐human crowd scenes as well as hierarchical modeling of complicated objects. Also described are several model management techniques.

Chapter 7 covers VR programming, including scene graphs, toolkits, object libraries, haptic toolkits, game engines (Unity and Unreal), the game production pipeline, and AI in gaming.

Chapter 8 discusses human factors in VR, including a wide range of topics: methodology and technology of human factors research, usability engineering methodology, and user performance studies, including detailed discussion of several user studies as examples. Cybersickness, a barrier to VR adoption, is analyzed in terms of causes and ways to address it. The chapter covers cybersickness and concludes with a discussion of the social implications of VR, including the potential impact of VR on professional life and the potential impact of VR on personal life.

Chapter 9 covers various applications of VR: medical education, rehabilitation after surgery or stroke, and VR skills training for individuals with cognitive impairment. An entire section is devoted to VR in education, arts, and entertainment. The chapter includes a sensitive discussion of VR to help with cultural heritage, appreciating pieces of art, buildings, and historical sites, some of which may no longer exist physically. Entertainment applications are covered in some detail, as these are today’s largest market for VR. These...