The Engineering of Mixed Reality Systems (eBook)

Contents 6
Contributors 9
1 Introduction 14
1.1 Mixed Reality Systems: A Booming Domain 14
1.1.1 Variety of Mixed Reality Systems 14
1.1.2 Variety of Application Domains 15
1.2 Mixed Reality Engineering 15
1.2.1 Interaction Design 15
1.2.1.1 Elements of Design 16
1.2.1.2 Specific Design Issues 16
1.2.1.3 Structuring the Design 17
1.2.2 Software Design and Implementation 17
1.2.2.1 Technical Solutions and Interaction Techniques 17
1.2.2.2 Platform: Prototyping, Development and Authoring Tools 17
1.2.2.3 Life Cycle 18
1.2.3 Application of Mixed Reality 18
Part I Interaction Design 20
2 An Integrating Framework for Mixed Systems 21
2.1 Introduction 21
2.2 Illustrative Examples 23
2.3 Integrating Framework for Describing and Classifying Mixed Systems 25
2.3.1 Modeling of a Mixed Object 25
2.3.2 Mixed Object: Intrinsic Characterization 26
2.3.2.1 Characteristics of the Linking Modalities (Devices and Languages) 26
2.3.2.2 Characteristics of the Physical Properties 27
2.3.2.3 Characteristics of the Digital Properties 29
2.3.3 Modeling Mixed Interaction: Putting Mixed Objects into Interaction Context 31
2.3.4 Mixed Object: Extrinsic Characterization 33
2.3.4.1 Characteristics of the Roles 33
2.3.4.2 Characteristics of the Physical Properties 33
2.3.4.3 Characteristics of the Digital Properties 35
2.4 Integrating Framework for Designing Mixed Systems: The Case of Roam 36
2.4.1 Extrinsic Design 38
2.4.2 Intrinsic Design 40
2.5 Conclusion 41
References 42
3 A Holistic Approach to Design and Evaluation of Mixed Reality Systems 44
3.1 Introduction 44
3.2 Related Work 45
3.2.1 Mixed Reality Systems 46
3.2.2 Usefulness 47
3.2.3 Technology in Context 47
3.3 User Involvement in the Development Process 50
3.3.1 The Method Used in the Case Studies 51
3.3.2 The Design Process Used in the Case Studies 51
3.4 The First Case Study An Instructional Task 53
3.4.1 Equipment Used in the Study 54
3.4.2 The User Task 54
3.4.3 Participants and Procedure 55
3.4.4 Results of the Study 56
3.5 The Second Case Study A Collaborative MR Application 57
3.5.1 Equipment Used in the Study 58
3.5.2 The User Task 58
3.5.3 Participants and Procedure 59
3.5.4 Results of the Study 60
3.6 Discussion 62
3.7 Conclusions and Future Direction 63
References 64
4 Embedded Mixed Reality Environments 67
4.1 Introduction 67
4.2 Three Embedded Mixed Reality Environments 69
4.2.1 The MackRoom: Co-visiting an Exhibition from Different Physical Locations 69
4.2.1.1 MackRoom in Use and Experience 71
4.2.2 Mixed Reality Architecture: Flexible Audio-Visual Connections Between Distributed Offices 72
4.2.2.1 MRA in Use and Experience 74
4.2.3 Uncle Roy All Around You: A Mobile Mixed Reality Performance 74
4.2.3.1 URAAY in Use and Experience 76
4.3 Designing for Embeddedness 77
4.3.1 Creating Space for Interaction 77
4.3.1.1 Setting Up the Physical Interaction Space 77
4.3.1.2 Extent of Physical Interaction Space 78
4.3.2 Asymmetries in the Interface Between Digital and Physical Environments 79
4.3.2.1 User Representations 80
4.3.2.2 Spatial Mapping 80
4.3.2.3 Content Mapping 81
4.3.3 Social Interaction in Embedded Mixed Reality Environment 81
4.3.3.1 Role Taking 82
4.3.3.2 Social Rules and Norms 83
4.4 Reflection 84
4.5 Conclusions 86
References 86
5 The Semantic Environment: Heuristics for a Cross-Context HumanInformation Interaction Model 89
5.1 Introduction 89
5.2 A Holistic Framework 90
5.3 From Information Retrieval to HumanInformation Interaction 91
5.4 Resilience 92
5.4.1 Scenario: The Resilient Supermarket 93
5.5 Place 95
5.5.1 Hansel and Gretel or Getting Lost in the Woods 96
5.5.2 Berry-Picking 96
5.5.3 Information Scent 97
5.5.4 Scenario: Sense of Place in the Supermarket 98
5.6 Choice 101
5.6.1 Hick's Law 102
5.6.2 Reducing the Load: Organize and Cluster, Focus, and Magnify 102
5.6.3 Scenario: Choice in the Supermarket 103
5.7 Correlation 103
5.7.1 Scenario: Correlation in the Supermarket 104
5.8 The Semantic Supermarket 105
5.9 Conclusions: Toward a Cross-Context HumanInformation Interaction Model 106
References 107
6 Tangible Interaction in Mixed Reality Systems 110
6.1 Introduction 110
6.2 State of the Art of Tangible User Interface Models 112
6.2.1 The Seminal Tangible Interaction Model 112
6.2.2 The Extended Tangible Interaction Model 113
6.3 Designing Tangible Interaction Techniques in MR Environments 114
6.3.1 Categorizations of Tangible User Interfaces 114
6.3.2 A Multidisciplinary and Participatory Approach 115
6.3.3 Taking into Account the Skills of Users 115
6.3.4 The Design Process 116
6.4 Case Studies 117
6.4.1 A Tangible User Interface for 3D CAD Parts Assembly: ESKUA 117
6.4.2 A Tangible Tabletop for Geoscience: GeoTUI 118
6.4.3 A Tangible User Interface for the Virtual Reassembly of Fractured Archaeological Objects: ArcheoTUI 120
6.4.4 Illustration of the Design Approach on Case Studies 121
6.5 User Studies in the Workplace: Feedback 122
6.5.1 Evaluation: Setup, Metrics, Analysis 122
6.5.1.1 ESKUA 122
6.5.1.2 GeoTUI 123
6.5.1.3 ArcheoTUI 124
6.5.2 Lessons Learnt from the User Studies 124
6.5.2.1 Recommendations Derived from Our User Studies on Tangible Interaction 125
6.5.2.2 Some Questions as a Guide 126
6.6 Conclusion: The Benefits of Tangible Interaction in Mixed Reality Systems 126
References 127
7 Designing a Mixed Reality Intergenerational EntertainmentSystem 130
7.1 Introduction 130
7.2 Related Work 132
7.3 Design Methodology 133
7.3.1 Problem Identification 134
7.3.2 Problem Exploration 135
7.3.3 Design Goals 136
7.4 Design Requirements and Ideas Generation 137
7.4.1 Resources and Time Constraints 137
7.4.2 User Needs 137
7.4.3 Context of Use 138
7.4.4 Design Ideas Generation 138
7.5 Prototype Iterations and System Description 139
7.5.1 Prototype Iterations 139
7.5.2 Current System Architecture 139
7.5.3 Game Play 140
7.6 Intergenerational Player Study 143
7.6.1 Introduction 143
7.6.2 Methods 143
7.6.3 Physical Interface Design Issues 144
7.6.4 Physicality Issues of the Virtual and Physical Player Roles 146
7.6.5 Focus Group Session with Older Players 147
7.7 Conclusion 148
References 149
8 Auditory-Induced Presence in Mixed Reality Environments and Related Technology 151
8.1 Audio in Mixed Realities 151
8.2 Presence and Auditory Displays 154
8.3 Spatial Sound Rendering and Presentation Technologies 155
8.3.1 Multichannel Loudspeaker Reproduction 155
8.3.2 Headphone Reproduction 156
8.3.3 Presentation Systems -- Design Considerations 158
8.3.4 Virtual Acoustics Synthesis and Optimization 158
8.4 Auditory Presence in Mediated Environments: Previous Findings 159
8.4.1 Presence and the Auditory Background 160
8.4.2 Spatial Properties 161
8.4.3 Sound Quality and Sound Content 162
8.4.4 Consistency Across and Within Modalities 163
8.5 Example Scenario: The MR Museum of Music History 164
8.5.1 Displays and Interaction Devices 165
8.5.2 Exhibition Displays 165
8.6 Discussion 167
References 168
9 An Exploration of Exertion in Mixed Reality Systems via the Table Tennis for Three Game 172
9.1 Introduction 173
9.2 Related Work 174
9.3 Table Tennis for Three 176
9.3.1 The Table Tennis for Three Experience 177
9.4 Design of Table Tennis for Three 179
9.4.1 Choice of Tangible Equipment 179
9.4.1.1 Supporting Bodily Skill Training 179
9.4.1.2 Utilizing Existing Sport Advances 179
9.4.1.3 Uncertainty of the Real World 179
9.4.1.4 Supporting Proprioception and Force-Feedback 181
9.4.1.5 Avoiding Complex Equipment Such as Head-Mounted Displays 181
9.4.2 Implementation 182
9.4.2.1 Impact Detection Mechanism 182
9.4.2.2 Videoconferencing 184
9.4.2.3 Gameplay 184
9.5 Feedback from Users 185
9.6 Future Work 186
9.7 Discussion and Conclusions 186
References 188
10 Developing Mixed Interactive Systems: A Model-Based Process for Generating and Managing Design Solutions 190
10.1 Introduction 190
10.1.1 Existing MIS Development Support 191
10.1.2 Objective and Goal 192
10.1.3 A Case Study 193
10.2 Articulating MIS Task Analysis and Mixed Interaction Design 194
10.2.1 Presentation of the Two Selected Models: K-MAD and ASUR 194
10.2.1.1 Task Analysis with K-MAD 194
10.2.1.2 Mixed Interaction Design with ASUR 195
10.2.2 Articulation Between K-MAD and ASUR 195
10.2.2.1 Task to Mixed Interaction Model Transformations 195
10.2.2.2 Applying the Task to Mixed Interaction Model Transformations on the Case Study 196
10.2.3 Designer and ASUR Refinement 198
10.2.4 Advantages and Limits of the Transformation Process 200
10.3 Articulating Mixed Interaction Design with MIS Implementation 201
10.3.1 MIS Architecture Requirements 201
10.3.1.1 Modifiability, Portability and Development Efficiency 201
10.3.1.2 ASUR-IL Metamodel 202
10.3.1.3 Model-Driven Engineering Tools 203
10.3.2 ASUR to ASUR-IL Transformation Principles 203
10.3.2.1 Mixed Interaction to Software Architecture Model Transformation 203
10.3.2.2 Applying the Mixed Interaction to Software Architecture Model Transformation on the Case Study 204
10.3.3 From a Software Architecture Model for MIS to mplementation 206
10.3.4 Limits and Interests of These Articulations 207
10.4 Outcomes of the Design Process in an Iterative Development Context 208
10.4.1 K-MAD Level 209
10.4.2 ASUR Level 209
10.4.3 ASUR-IL Level 210
10.4.4 WComp Level 211
10.5 Conclusions and Perspectives 211
References 213
Part II Software Design and Implementation 216
11 Designing Outdoor Mixed Reality Hardware Systems 217
11.1 Introduction 217
11.2 Previous Work on Outdoor MR 220
11.3 The Tinmith System 221
11.4 Hardware for Outdoor MR Systems 222
11.4.1 Head-Mounted Electronics 224
11.4.2 Main Enclosure 226
11.4.3 Batteries 227
11.5 Input Devices 227
11.5.1 Pinch Gloves 228
11.5.2 Button Box 228
11.5.3 Additional Input Devices 229
11.6 Wearable Mixed Reality System Design 230
11.6.1 Manufacturing Techniques 230
11.6.2 Belt vs. Backpack 231
11.6.3 Electrical and Magnetic Interference 232
11.7 System Management 232
11.7.1 Power Management 232
11.7.2 Configuration Selection 233
11.7.3 Input Management 234
11.7.4 External Display 234
11.8 Conclusion 236
References 236
12 Multimodal Excitatory Interfaces with Automatic Content Classification 238
12.1 Motivation 238
12.2 Background Review 240
12.3 Inertial Sensing 240
12.4 Object Dynamics 242
12.4.1 Accelerometer Mapping 242
12.4.2 Friction and Stiction 243
12.4.3 Springs 243
12.4.4 Impacts 244
12.5 Message Transformation 244
12.5.1 PPM Language Model 244
12.5.1.1 Potential Enhancements 247
12.5.1.2 Test Model Classes 247
12.5.1.3 Certainty Filtering 249
12.5.2 Exploration 249
12.5.2.1 Identity Sieving 249
12.5.2.2 Time-Sequenced ''Rain'' 250
12.6 Auditory and Vibrotactile Display 250
12.6.1 Vibrotactile Events 251
12.6.2 Audio Synthesis 252
12.6.2.1 Sample Banks 252
12.6.2.2 Audio Transformations 253
12.7 Further-Work Active Selection 254
12.8 Conclusions 254
References 254
13 Management of Tracking for Mixed and AugmentedReality Systems 256
13.1 Motivation 256
13.1.1 Requirements 257
13.1.2 Related Work 258
13.1.3 The Ubitrack and trackman Approach 258
13.2 The Ubitrack Framework 259
13.2.1 Spatial Relationship Graphs 259
13.2.1.1 Use of Cycles for Sensor Calibration and Object Registration 260
13.2.1.2 Edge Characteristics 260
13.2.2 Data Flow Networks 261
13.2.3 Spatial Relationship Patterns 261
13.2.3.1 Basic Concept 261
13.2.3.2 Synchronization Issues 263
13.2.3.3 Pattern Categories 263
13.2.4 SRG Design Activities 265
13.3 trackman: Interactive Modeling of Spatial Relationships 265
13.3.1 System Architecture 265
13.3.2 Graphical Layout 266
13.3.3 Interactive SRG Generation 267
13.3.4 Interactive Deduction of Spatial Relationships 267
13.3.5 More Modeling Functionality 268
13.3.6 Ordering of Design Activities 269
13.4 Advanced Interactive Modeling Concepts 270
13.4.1 Semi-automatic Modeling 270
13.4.2 Meta Patterns 271
13.5 Tools to Analyze and to Interact with Data Flows 272
13.5.1 Tools for Calibration and Registration 272
13.5.2 Tools for Online Analysis of Tracking Environments 274
13.6 Application Examples 275
13.7 Conclusion 276
References 277
14 Authoring Immersive Mixed Reality Experiences 279
14.1 Introduction 279
14.1.1 Definitions and Assumptions 280
14.1.1.1 Mixed Reality 280
14.1.1.2 Immersion 280
14.1.1.3 Absolute vs. Relative Coordinate Systems 280
14.2 Background: Mixed Reality Environments in the Arts 281
14.2.1 Motivation for Using Mixed Reality in the Arts 281
14.2.2 Examples of Mixed Reality in the Arts 282
14.2.2.1 Example: Markerless Magic Books 282
14.2.2.2 Mixed Reality as a Presentation Medium 282
14.2.2.3 Crossing Borders: Interactive Cinema 282
14.3 Related Work: Authoring Tools 283
14.4 Authoring Content for Mixed Reality Environments 284
14.4.1 Engineering and Authoring Platform: VGE 284
14.4.1.1 Overall Architecture 285
14.4.1.2 Perspectives 285
14.4.1.3 Sensors and Algorithms 286
14.4.2 Designing the Real World 286
14.4.2.1 Geometry and Visual Appearance 287
14.4.2.2 Lighting 287
14.4.3 Mixing Virtual Images 287
14.4.3.1 Test Setup 288
14.4.4 Directing the User Experience 290
14.4.4.1 Prototyping Tool: Interactive Table 291
14.4.5 Case Study: Exercise in Immersion 4 292
14.5 Conclusions 293
References 294
15 Fiia: A Model-Based Approach to Engineering Collaborative Augmented Reality 296
15.1 Introduction 296
15.2 Example: Collaborative Game Prototyping with Raptor 297
15.3 Related Work 301
15.3.1 Modeling Collaborative Augmented Reality 301
15.3.2 Toolkits for Collaborative AR 301
15.4 Fiia Notation 302
15.4.1 Notation for Collaborative AR 303
15.4.1.1 Adapters 304
15.4.1.2 Data Sharing 305
15.4.2 Scenario-Based Design 306
15.4.3 Mapping Fiia Diagrams to Code 307
15.4.4 Summing Up the Fiia Notation 308
15.5 The Fiia.Net Toolkit 308
15.5.1 Conceptual Framework 308
15.5.2 Distribution Architecture 309
15.5.3 Adapters 311
15.6 Implementing Fiia 312
15.7 Experience 313
15.8 Conclusion 314
References 314
16 A Software Engineering Method for the Design of Mixed Reality Systems 316
16.1 Introduction 316
16.2 Extending an SE Method for Mixed Reality Systems 318
16.2.1 Extending Symphony for the Design of Mixed Reality Systems 318
16.2.2 Case Study 320
16.3 The Functional Branch 321
16.3.1 Introduction 321
16.3.2 Initiating the Development 321
16.3.3 Conceptual Specifications of Requirements 322
16.3.4 Organizational and Interaction-Oriented Specification of Requirements 323
16.3.5 Analysis 327
16.3.6 Main Points Discussed 329
16.4 The Technical Branch 329
16.4.1 Description of the Applicative Architecture 330
16.4.2 Description of the Technical Architecture 331
16.4.3 Main Points Discussed 332
16.5 The Junction of the Functional and Technical Branches 333
16.5.1 Design 333
16.5.2 Main Points Discussed 335
16.6 Conclusions and Future Work 335
References 336
Part III Applications of Mixed Reality 338
17 Enhancing Health-Care Services with Mixed Reality Systems 339
17.1 Health Care and Mixed Reality Systems 339
17.1.1 Augmented and Mixed Reality 340
17.1.2 Usability Evaluation Techniques 341
17.1.3 Security Aspects 342
17.1.4 Work Structure 342
17.2 Overview of the Development Approach 342
17.2.1 Process Evaluation of the Health-Care Service 343
17.2.2 Evaluation of the Existing Information Systems 343
17.2.3 Identification of Decision Paths and Actions That Can Benefit from Mixed Reality Systems 344
17.2.4 Implementation of the Mixed Reality System 344
17.3 System Design and Implementation 344
17.3.1 Design of the System 345
17.3.2 ASUR Model of the System 347
17.3.2.1 Real Objects (Components R) 347
17.3.2.2 Person as User (Component U) 348
17.3.2.3 Adapters (Components A) 348
17.3.2.4 Systems (Components S) 349
17.3.2.5 Relationships Between the ASUR Components of the System 349
17.3.3 Addressing Critical Aspects of Mixed Reality Systems for Health-Care Services 349
17.3.3.1 Context Awareness 350
17.3.3.2 Timeliness and High Assurance 350
17.3.3.3 Fault Tolerance 351
17.3.3.4 Interoperability 351
17.3.4 Addressing Software Design Requirements 352
17.3.4.1 Distributed and Cooperating Services 352
17.3.4.2 Security and Privacy 353
17.3.4.3 Lookup and Discovery 353
17.3.4.4 Performance and Availability 354
17.3.5 Technology Environment and Architectural Approach 354
17.4 Conclusion and Outlook 356
References 356
18 The eXperience Induction Machine: A New Paradigm for Mixed-Reality Interaction Design and Psychological Experimentation 359
18.1 Introduction 359
18.1.1 Mixed-Reality Installations and Spaces 361
18.1.2 Why Build Such Spaces? Epistemological Rationale 363
18.1.3 Mixed and Virtual Reality as a Tool in Psychological Research 365
18.1.4 Challenges of Using Mixed and Virtual Realities in Psychological Research 367
18.2 The eXperience Induction Machine 369
18.2.1 System Architecture 369
18.2.1.1 Design Principles 369
18.2.1.2 Interfaces to Sensors and Effectors 370
18.3 XIM as a Platform for Psychological Experimentation 372
18.3.1 The Persistent Virtual Community 372
18.3.2 A Space Explains Itself: The ''Autodemo'' 373
18.3.3 Cooperation and Competition: Playing Football in Mixed Reality 375
18.4 Conclusion and Outlook 377
References 378
19 MyCoach: In Situ User Evaluation of a Virtual and Physical Coach for Running 382
19.1 Introduction 382
19.1.1 Virtual Trainer/Coach 383
19.2 MyCoach 384
19.3 User Experiment 386
19.3.1 Runners 386
19.3.2 Measurement 387
19.4 Results 388
19.4.1 Pre-trial Results: Running and Training Habits 388
19.4.2 During Trial Results: Use of the MyCoach System 390
19.5 Usage Data 391
19.6 Netnography 392
19.6.1 Post-trial: Evaluation of MyCoach 393
19.7 Conclusions 396
19.8 Further Development of MyCoach 396
References 397
20 The RoboCup Mixed Reality League - A Case Study 399
20.1 Introduction 399
20.2 Hardware Architecture 403
20.2.1 Micro-Robots 403
20.2.1.1 Battery Charger 406
20.2.1.2 Infrared Transmitter 407
20.2.1.3 Firmware Uploading Interface Board 407
20.2.2 Augmented Reality Display 408
20.2.3 Tracking Camera 408
20.2.4 Computer 409
20.3 Software Architecture 409
20.3.1 Vision-Tracking Module 411
20.3.2 Application Modules 411
20.3.3 Graphics Module 413
20.3.4 Robot Control Module 413
20.3.5 Agents 415
20.4 Experience 415
20.4.1 Development Process 415
20.4.2 Soccer System 415
20.4.3 Racing Application 416
20.4.4 Future Developments 417
20.5 Summary and Conclusions 417
References 417
21 Mixed-Reality Prototypes to Support Early Creative Design 419
21.1 Introduction 419
21.2 Profession-Centered Methodology and User-Centered Design 420
21.3 Context and Needs 423
21.3.1 Architectural Design 423
21.3.2 Sketch-Based Preliminary Design 425
21.3.3 Distant Collaborative Design 426
21.3.4 Why Mixed Reality Should Be a Good Way of Responding to These Needs 427
21.4 Technological Solutions 428
21.4.1 Introduction 428
21.4.2 The Virtual Desktop 428
21.4.3 EsQUIsE 431
21.4.3.1 Introduction 431
21.4.3.2 The Entry Module 431
21.4.3.3 The Interpretation Module 432
21.4.3.4 The Evaluation Module 432
21.4.4 SketSha 434
21.5 Evaluations 436
21.5.1 Usability 437
21.5.2 Sketches 437
21.5.3 Immersion 438
21.5.4 Design Process 440
21.6 Characterization of These Mixed-Reality Systems 441
21.7 Discussions 442
References 443
Index 446