1 Supplementing the Markerless AR with machine learning: Methods and approaches

Abstract

Augmented reality enhances real-world entities with the addition of relevant digital information. This interactive and engaging technology is incorporating itself into wider areas of technical applications. Although it is still in its evolving stage, it has garnered positive responses from audiences who interact with AR-based systems. Industries like gaming, healthcare, and education are benefiting from various augmented reality implementations. The main focus of using AR is to produce more effective interaction with the physical real world as a template and render a separate scenario above it. Initially, a marker-based system was created to project information from a marked region or image with specific characteristics, leading to multiple marketing and retail services adopting this feature. In these applications, a particular marker is sorted through view and detected to project the AR-related information, which further triggers the application’ response towards the marker. The baseline strategy includes image capture, processing for markers, tracking the results, and final rendering. However, to create more innovative forms of augmented reality, this concept was merged with machine learning to improve on the aspects that do not revolve around only particular markers. Markerless AR features superimposing graphics on the basis of locations, contour, or projection. This led to the exploration of various machine learning-based algorithms to improve the decision-making accuracy for AR applications. The methodology includes understanding the real environment and fabricating designs accordingly without the need to look for triggering markers. This chapter enlists various markerless AR techniques that use machine learning methods to increase the capability and coherence. The focus is to identify approaches used from machine learning algorithms to amplify efficacy of augmented reality applications using markerless services.

1.1 Introduction

As humans we are likely learning much from our surrounding environment, and our senses work together to procure us information that is processed, memorized, and stored in our minds. It is this highly valuable practice that initiates and creates a channel to extend our knowledge and zeal for learning or noticing the external environment. Augmented Reality changes this practice slightly by altering the method of interaction with the environment, converting conventional methods to superficial ones. It is this technical enhancement that has amended our perception of vicinity. The implementation of Augmented Reality involves the creation of a specious facade over the real world, thereby creating a separate vision for the actual environment. It creates an approximate or qualitative vision for the surroundings that often shows altered or customized perspectives. The idea of AR is to enhance its user’s point of view towards reality.

AR was initiated back in 1968 when Ian Sutherland created the first HMD (Head-mounted display) headset to increase the sensory perception of the world [1]. Since the 1970s, pivotal growth has been observed in establishing augmented reality as an effective technological establishment in a wide variety of applications. Several attempts have been made to incorporate the ideas of using AR into medicine, marketing, and education. Platforms using Augmented Reality have seen huge increments in users for reasons including growing number of internet users, availability of high-quality handheld devices, and less usage of extra equipment that is necessary. Any successful technical enhancement can grow within the user’s community when it is readily available to its end-users without any extra hardware installation requirements [2]. AR effects generation generally requires the device to be capable of capturing images of surroundings and processing them, which can be easily achieved through the installation of a plugin or application to superimpose upon objects. Neural networks are providing diverse applications used for enhancing the subjectivity of AR-based projections [3], [4], [5]. Since the inception of AR in 1960s, the craze for this technology was enhanced mainly through gaming applications. The first application launched back in 2008, for an advertisement but gradually, as Figure 1.1 shows, the diversity and growth of AR during past two decades have provided a medium to enhance curiosity among users.

Figure 1.1 Mainstreaming of Augmented reality [6].

This chapter introduces the Augmented Reality as a technology, followed by Section 1.2 which discusses the types of AR, with a focus on markerless. Section 1.3 highlights the relevance of machine learning in AR implementation, while Section 1.4 discusses the contributions of other scholars in this area. Section 1.5 provides an in-depth methodology to show where machine learning is used. Furthermore Section 1.6 provides insights into current case studies in AR based on machine learning technologies, and Section 1.7 presents the conclusion.

1.2 Motivation

Augmented Reality has surely piqued the curiosity of many researchers and has played an important role in creation of innovative practices involving it. Machine learning is a pivotal field that is continuously enhancing its properties and increasing its boundaries. There is a growing amount of literature that supports the benefits of incorporating these two powerful technologies, as the strength of AR can be improved and enhanced through well-developed and practiced methods of ML. Researching through these papers, the results suggest considerable efforts of joining these technologies. This chapter aims to shed light on some of these pivotal studies while investigating and understanding the intricate balance between AR and ML. One major challenge that poses a current research gap in this approach lies within the devices projecting AR models. As multiple instances use common smartphones that have limited resources for computation, issues such as latency, speed, and dynamics of rendering AR models are delayed due to longer computations. To date, only a few studies have investigated this aspect as well.

1.3 How have AI and blockchain transformed AR?

Classifying AR into various different categories depends on the type of hardware used or how the end result is projected. In accordance with the theme of this chapter, the classification paradigm is the basis of how AR understands the real world and projects the final rendering [7].

As Figure 1.2 shows, the main classification at this level is between Marked AR and Markerless AR. Marker-based AR is an approach where a target object is identified, and a pivot is used to situate the rendered object on that specific surface.

Figure 1.2 Types of augmented reality.

This is an initial approach that is generally found in smartphone cameras to project different filters. Identification of a point is relevant in this form of AR. [8] have compared on how CNN and SVM can be used to identify the spot to render AR, showing 96.7 % and 92.5 % respectively. Their application is designed to recognize and project the alphabets for children in a learning app. Figure 1.3 shows a person making a digital purchase through a QR-code.

Figure 1.3 A marker-based approach requires markers such as QR codes to trigger an event. Image credit: [9].

Marker-based AR is successful when the point of reference is well-defined and output is based around it. However, in situations where the marker is not detected it proves to be unfruitful [2]. This led to creation of Markerless AR, which relies on incoming data from various sensors to place a rendered output in context with the real world. Mainly, as listed in Table 1.1, a simultaneous localization and mapping criteria are used, which creates an estimate to place objects in the real world.