1
Perspective

LET'S START WITH an exercise. Hold your index finger out in front of you. Now draw a cube in the air. Think about how you drew that cube: Did you draw a flat cube with a square and diagonal lines to simulate a three-dimensional drawing on a two-dimensional plane? Or did you outstretch your arm to give your cube depth, extending into the third dimension? Chances are if you are an adult, you drew a flat cube. When I try this exercise with young children, they don't have the limitations of thinking about 3D objects as a 2D representation. They think about things how they are in the physical world. We have spent our entire lives viewing content on flat paper, a computer screen, a phone, and it's very difficult to reverse that mentality. But my hope for you is that you can train yourself to think like a child. To think in a more physical way. Because to create virtual worlds, we must drop much of what we've been taught all our lives. We need to remove the limitations that decades of schooling and society have engrained in our minds. It can be done, but it takes work to break those mental habits.

When we work in extended reality, many of the constraints that exist for other media are lifted. The laws of physics no longer exist (unless you want them to). Some of the best features of development engines such as Unity and Unreal are that they simulate physics—gravity, collision, motion, etc. However, these programs also allow you to turn physics off or adapt the experience in a way that is out of this world. Imagine a virtual reality simulation that has the gravity of the moon. Objects react in this experience as if they were on the moon when you throw them, bounce them, drop them. The only component that doesn't react in this way is your actual body (unless you're using some sort of harness to help simulate that too). While many extended reality experiences need to retain some of the earthly laws of physics, it's fun to think outside the realm of what's possible in the physical world and leverage such a powerful medium to achieve just that.

Physical versus Digital

When designing for extended reality, think in 3D. Drop the limitations of your 2D computer screen, phone, and tablet. Even if technically we will still be experiencing the content through flat screens, the way we perceive it is with depth and dimension because of the way it's being anchored or positioned in a space. So what does this mean, to have the freedom of depth and dimension? It means that we can visualize data and content in a way that is immersive. We can interact with content like we interact with objects in the physical world. But in XR, they don't even have to be objects that are bound to the physical world. We can create functional, technological magic.

You'll notice throughout this book that I do not refer to things as “real” and “fake” but rather “physical” and “digital.” This is because the line between real and fake is blurring, and definitions are changing. If someone joins a social space and attends an event on their virtual reality headset, that is still a real experience to them. They made connections, conversed with people across the world, and created a memory—nothing fake about that. Differentiating between extended reality content, or digital content, and the physical world that grounds us isn't made up of real and fake. It's all real in some way, shape, or form. It's different from what we've always known, but that doesn't make it less real.

Sometimes it's a good thing to be able to replicate the physical world. It's not all about creating a digital fantasy where there are no boundaries. Many of the best use cases come from replicating parts of the physical world and their processes. Think of educational content and practice simulations for surgeons. By recreating what's in the physical world these doctors can practice indefinitely and with unlimited resources. There isn't any waste, and there is no liability or potential loss of life with a virtual practice simulation. Simply put, you can practice until you're fully prepared. Another great reason to replicate the physical world is to grant access to culture, travel, and art, for people who don't have the ability or means to access it in person. While in many cases it's hard to compete with the real thing, extended reality can give exposure that can lead to greater understanding around the world, and it's no longer limited to mobility, ability to travel, or socioeconomic status.

Accessibility

Accessibility in the realm of extended reality still has a long way to go, and there is much to be desired, but there are some aspects of accessibility that are made possible by this technology. Leveraging positional data, or where someone is located in a space, and then being able to give detailed audio cues and information based on that positional or spatial data is a powerful way to assist those who are visually impaired. The ability for a program to intake real-time audio and generate voice-to-text displayed directly in the line of sight of an individual via their glasses is an effective way to assist someone who is hard of hearing. Taking that same technology one step further and generating real-time translations could help break global barriers in communication. The ability to lower the interactive components of an environment with the click of a button can help someone who can only experience the world in a seated position. The possibilities for accessibility in these virtual spaces and with augmented tools are going to open a wide array of opportunities, efficiencies, and quality-of-life assistance as never seen before.

Possibilities

So what is possible in XR? When I say the possibilities are endless, that may in fact be true based on the human imagination. However, while there are some capabilities that stand out as winning features, there are limitations as exist with all technology. Starting with digital environments and assets, it's possible to make a 3D space that looks like anything you can think up. By leveraging 3D modeling tools such as Blender or Maya, the creative limitations are bound to your technical modeling skills. Thinking of creating a futuristic otherworldly inhabitation? That can be done! When planning these spaces, ponder about everything you could ever want in an environment. How do you eat? How do you use transportation? What do the trash cans look like? Whether it's a fantasy world or very much rooted in the physical world, this level of detail is crucial. The same goes for single objects that may be experienced one at a time or overlaid onto the physical world. And while creative limitations are boundless, unfortunately, technological limitations exist. Environments and art assets will need to be optimized for the device on which the content is being deployed, so keep that in mind along the way, and consider where you can make compromises if necessary.

Interactivity

Interactions are another imperative factor of any extended reality experience. It's important to think about how to implement interactions in these worlds that you've created or with these digital assets you've produced. Interactions are essential and allow designers to bring what may be impossible in the physical world into a functional virtual instance. Interactions and the complexity of these interactions have a vast range, but the most common is “grab.” The act of picking up an item and placing it, moving it, throwing it, stacking it, dropping it is not only a novelty but a requirement in most extended reality experiences. While it seems the simplest, if “grab” doesn't feel right, it can ruin an experience. “Grab mechanics,” as they're called, are common across simulations in their implementation, and extended reality users have grown to expect a level of consistency across experiences, which is good for the industry and its progress. Interactions can be made using a controller or gesture/hand tracking depending on the hardware's capabilities.

Moving beyond “grab,” most other interactions will involve some sort of physical intersection of the “hand” or controller and the digital object. Think of outstretching your index finger and pushing a physical button with it. That same interaction can be translated into a digital experience using that exact implementation. And a third type of interaction would be user interface, or UI, interaction. This is less immersive, but it can feel more familiar because this is typically used to access menu screens or dialogue boxes, similar to 2D digital content. This is commonly done by using a point-and-click method, like what you'd experience with a computer mouse or even your finger on a phone screen. Interactions are a crucial consideration in any extended reality experience, so take advantage of the possibilities and give thought to implementation.

Movement and locomotion add to the opportunities that extended reality brings to its audience. Moving the user in your experience can be tricky, and detrimental if not considered with care. If you move the user without their consent, it can be nausea inducing, which is an immediate turnoff to the technology in general. On the positive side, in extended reality you're able to move in ways you couldn't in the physical world. Teleportation is one of the most common forms of locomotion. I know you're thinking, “But...