Adding Sensorial Effects to Media Content

Keywords

Sensorial effects; sensorial effect rendering; sensory effect metadata

Contents

2.1 Introduction

MPEG-V, Part 3: Sensory information (ISO/IEC 23005-3), specifies the Sensory Effect Description Language (SEDL) [1] as an XML schema-based language that enables one to describe sensorial effects (SEs) such as light, wind, fog, and vibration that trigger human senses. The actual SEs are not part of the SEDL but are defined within the Sensory Effect Vocabulary (SEV) for extensibility and flexibility, allowing each application domain to define its own SEs. A description conforming to SEDL is referred to as Sensory Effect Metadata (SEM) and may be associated with any type of multimedia content (e.g., movies, music, Web sites, games). The SEM is used to steer actuators such as fans, vibration chairs, and lamps using an appropriate mediation device to increase the user experience. That is, in addition to the audiovisual (AV) content of a movie, e.g., the user will also perceive other effects such as those described above, giving the user the sensation of being part of the particular media content, which will result in a worthwhile, informative user experience. The concept of receiving SEs in addition to AV content is depicted in Figure 2.1.

The media and corresponding SEM may be obtained from a Digital Versatile Disc (DVD), Blu-ray Disc (BD), or any type of online service (i.e., download/play or streaming). The media processing engine, which is also referred to as the adaptation engine, acts as the mediation device and is responsible for playing the actual media content resource and accompanied SEs in a synchronized way based on the user’s setup in terms of both the media content and rendering of the SE. Therefore, the media processing engine may adapt both the media resource and the SEM according to the capabilities of the various rendering devices.

The SEV defines a clear set of actual SEs to be used with the SEDL in an extensible and flexible way. That is, it can be easily extended with new effects or through a derivation of existing effects thanks to the extensibility feature of the XML schema. Furthermore, the effects are defined based on the authors’ (i.e., creators of the SEM) intention independent from the end user’s device setting, as shown in Figure 2.2.

The sensory effect metadata elements or data types are mapped to commands that control the actuators based on their capabilities. This mapping is usually provided by the Virtual-to-Real adaptation engine and was deliberately not defined in this standard, i.e., it is left open for industry competitors. It is important to note that there is not necessarily a one-to-one mapping between elements or data types of the SE data and ACs. For example, the effect of hot/cold wind may be rendered on a single device with two capabilities, i.e., a heater or air conditioner, and a fan or ventilator.

As shown in Figure 2.3, the SEs can be adjusted into adapted SEs (i.e., defined in MPEG-V, Part 5, as device commands) in accordance with the capabilities of the actuators (ACs, defined in MPEG-V, Part 2) and actuation preferences (APs, defined in MPEG-V, Part 2, as user sensory preferences).

Figure 2.4 shows an example of combining SEs (SEs in MPEG-V, Part 3) with sensed information (SI in MPEG-V, Part 5) to generate adapted actuator commands (ACmd in MPEG-V, Part 5). For example, the SE corresponding to the scene might be cooling the temperature to 5°C and adding a wind effect with 100% intensity. Assume instead that the current room temperature is 12°C. It would be unwise to deploy the cooling and wind effect as described in the SE data because the current temperature inside the room is already low, and users may feel uncomfortable with the generated SEs. Therefore, a sensor measures the room temperature and the adaptation engine generates the adapted SEs (i.e., ACmds), which are a reduced wind effect (20% intensity) and a heating effect (20°C), for instance.

This chapter is organized as follows. Section 2.2 describes the details of the SEDL. Section 2.3 presents the SEV, which specifies the data formats used for creating SEs. Section 2.4 presents XML instances using SEDL and SEV. Finally, Section 2.5 concludes the chapter.

2.2 Sensory Effect Description Language

2.2.1 SEDL Structure

The SEDL is a language providing basic building blocks to instantiate sensory effect metadata defined by the MPEG-V standard based on XML that can be authored by content providers.

2.2.2 Base Data Types and Elements of SEDL

There are two base types in the SEDL. The first base type is SEMBaseAttributes, which includes six base attributes and one base attribute Group. The schema definition of SEMBaseAttributes is shown in Table 2.1. The activate attribute describes whether the SE shall be activated.

The duration attribute describes the duration of any SE rendering. The fade attribute describes the fade time within which the defined intensity is reached. The alt attribute describes an alternative effect identified by the uniform resource identifier (URI). For example, an alternative effect is chosen because the original intended effect cannot be rendered owing to a lack of devices supporting this effect. The priority attribute describes the priority for effects with respect to other effects in the same group of effects sharing the same point in time when they should become available for consumption. A value of 1 indicates the highest priority, and larger values indicate lower priorities. The location attribute describes the location from where the effect is expected to be received from the user’s perspective according to the X, Y, and Z axes, as depicted in Figure 2.5. A classification scheme that may be used for this purpose is LocationCS, as defined in Annex A of ISO/IEC 23005-6. For example, urn:mpeg:mpeg-v:01-SI-LocationCS-NS:left:*:midway defines the location as follows: left on the X-axis, any location on the Y-axis, and midway on the Z-axis. That is, it describes all effects on the left-midway side of the user. The SEMAdaptabilityAttributes contains two attributes related to the adaptability of the SEs. The adaptType attribute describes the preferred type of adaptation using the following possible instantiations: strict, i.e., an adaptation by approximation may not be performed, i.e., an adaptation by approximation may be performed with a smaller effect value than the specified effect value, i.e., an adaptation by approximation may be performed with a greater effect value than the specified effect value, and i.e., an adaptation by approximation may be performed between the upper and lower bounds specified by adaptRange. The adaptRange attribute describes the upper and lower bounds in terms of percentage for adaptType.

There are five base elements (Table 2.2), i.e., Declaration, GroupOfEffects, Effect, ReferenceEffect, and Parameter, which are explained in detail in the following sections, extended from the abstract SEMBaseType type...