3D Visual Content Creation, Coding and Delivery (eBook)

​Pedro A. Amado Assunção received the Licenciado and M.Sc. degrees in Electrical Engineering from the University of Coimbra, Portugal, in 1988 and 1993, respectively, and the Ph.D. in Electronic Systems Engineering from the University of Essex, UK, in 1998. He is currently Professor of Electrical and Computer Engineering, Electronics and Multimedia Communication Systems at the Polytechnic Institute of Leiria and senior researcher at Instituto de Telecomunicacoes, Portugal. He has been involved in several projects in the field of multimedia communications and he served as a reviewer and/or technical programme committee of many international conferences and journals. He is author/co-author of more than one hundred papers published in international scientific conferences and journals, one book, eight book chapters and four US patents. His current research interests include ultra-high definition 3D multiview and light-field video coding, communications and processing, video codec complexity control and networking adaptation of coded video streams, error concealment and quality assessment. He was the Chair of the EU COST Action IC1105, 3D Content Creation, Coding and Transmission over Future Media Networks (3D-ConTourNet) and General Chair of 3DTV-Con 2015. He is a Senior Member of the IEEE. Atanas Gotchev received M.Sc. degrees in radio and television engineering (1990) and applied mathematics (1992) and Ph.D. degree in telecommunications (1996) from the Technical University of Sofia, and D.Sc. (Tech.) degree in information technologies from Tampere University of Technology (2003). He is Professor at Tampere University of Technology and Director of the Centre for Immersive Visual Technologies. He has published more than 30 articles in referred scientific journals, 4 book chapters and more than 145 papers in refereed scientific edited volumes and conference proceedings. He has actively contributed to the organization of the 3DTV series of conferences (being twice General Chair and four times Technical Program Chair), and served as a committee member and reviewer in a number of conferences such as EUSIPCO, IEEE ICME, IEEE ICIP, etc. He was the Scientific Coordinator of the EU FP7 project Mobile3DTV and currently he is the Coordinator of the European Training Network of Full Parallax Imaging. His recent research concentrates on methods and algorithms for multi-sensor 3-D scene capture, transform-domain light-field reconstruction, and Fourier analysis of 3-D displays.

Contents 6
1 Introduction 12
2 Emerging Imaging Technologies: Trends and Challenges 15
Abstract 15
2.1 Introduction 16
2.2 Multiview Video Plus Depth 19
2.3 Standardization—The Status and Current Activities 24
2.3.1 Standardization in Multimedia 24
2.3.2 Basic Technologies 26
2.3.3 Multiview Video Coding 27
2.3.4 3D Video Coding 28
2.3.5 New Standardization Projects 31
2.4 Lightfield Super-Multiview with Camera Array 31
2.5 Lightfield with Microlens Array 35
2.5.1 Lightfield Raw Data-Based Approach 37
2.5.2 Multiview-Based Approach 38
2.5.3 Subsampled Grid of MIs Plus Disparity Approach 40
2.6 Free Navigation and Free Viewpoint Television 41
Acknowledgements 43
References 43
3 3D Content Acquisition and Coding 50
Abstract 50
3.1 Introduction 51
3.2 Effect of an Incorrect Camera Alignment on the Accuracy of the Spatial Reconstruction and Stereo Perception 52
3.2.1 The Influence of Inaccurate Camera Alignment 53
3.2.2 Influence of the Camera System Parameters and Spatial Position of the Object 57
3.2.3 Remarks 64
3.3 Compression Tools for Stereoscopic and Multiview Video 64
3.3.1 Stereoscopic Frame-Compatible Formats 65
3.3.2 Compression Tools for Stereoscopic Video 67
3.3.3 Performance Analysis of Compression Tools 70
3.3.4 Remarks 76
3.4 Multiview Video Compression for Arbitrary Camera Locations 77
3.4.1 Adaptation of 3D-HEVC to Nonlinear Camera Arrangements 77
3.4.2 Methodology of Evaluation 80
3.4.3 Results 81
3.4.4 Remarks 82
3.5 Recent Developments in Video Compression with Capabilities Beyond HEVC 82
3.5.1 UltraHD Video Compression Performance Beyond HEVC 84
3.5.2 Conversion and Coding for HDR/WCG Video 90
3.5.3 Projection Conversions and Coding for 360° Video 95
Acknowledgements 101
References 101
4 Efficient Depth-Based Coding 105
Abstract 105
4.1 Introduction 106
4.2 Depth Map Coding for Efficient Virtual View Synthesis 106
4.2.1 Algorithm Overview 107
4.2.2 Flexible Block Partitioning 108
4.2.3 Directional Intra Prediction 108
4.2.4 Constrained Depth Modelling Mode 110
4.2.5 Residual Signal Coding 111
4.2.6 Rate-Distortion Performance 112
4.3 Depth Compression Using Standard Coding Techniques 113
4.3.1 Bitrate Distribution 113
4.3.2 Depth Map Quality 114
4.3.3 Bitrate Distribution Between Texture and Depth 115
4.3.4 Coding Depth with Reduced Resolution 119
4.4 Conclusion 121
Acknowledgements 121
References 121
5 Error Concealment Methods for Multiview Video and Depth 123
Abstract 123
5.1 Introduction 124
5.2 Error Concealment for Multiview Video 125
5.2.1 Basic Methods Using Neighbouring Regions 125
5.2.2 Recent Advances in EC for Multiview Video 128
5.3 Methods for Error Concealment of Depth Maps 136
5.4 Conclusions 146
References 147
6 Light Field Image Compression 150
Abstract 150
6.1 Introduction 151
6.2 Light Field Image Representation 152
6.3 Light Field Image Coding Formats 153
6.3.1 Light Field Image Coding Using HEVC 154
6.3.1.1 Coding Efficiency 157
6.4 Scalable Light Field Coding for Backward Display Compatibility 159
6.4.1 Display Scalable Coding Architecture 162
6.4.2 Hierarchical Content Generation 163
6.4.3 Efficient LF Enhancement Layer Coding Solution 164
6.4.3.1 Self-similarity (SS) Prediction 164
6.4.3.2 Inter-layer (IL) Prediction 165
6.4.3.3 Intra Prediction 167
6.4.3.4 Header Formatting and CABAC 167
6.4.4 Performance Assessment 168
6.4.4.1 Overall DS-LFC RD Performance 170
6.4.4.2 Quality of Rendered Views 171
6.5 Sparse Set of Micro-lens Images and Disparities for an Efficient Scalable Coding of Light Field Images 173
6.5.1 Scalability 173
6.5.2 Displacement Intra and Inter Prediction Scheme 174
6.5.3 Encoding 174
6.5.3.1 Sparse Set of Micro-lens Images 175
6.5.3.2 Disparity Maps 176
6.5.3.3 Refinement by Inter and Intra Prediction 177
6.5.4 Decoding and Reconstruction 177
6.5.5 Evaluation 179
6.5.6 Remarks 180
6.6 Conclusions 180
References 181
7 Impact of Packet Losses in Scalable Light Field Video Coding 184
Abstract 184
7.1 Introduction 184
7.2 Scalable Light Field Coding 186
7.3 Mitigation of Packet Loss Impact on Scalable Light Field Coding 187
7.3.1 Relevant Factors for the Inter-layer Prediction Accuracy 188
7.3.2 Proposed Error Concealment Algorithm 190
7.4 Experimental Results 192
7.5 Conclusions 199
Acknowledgements 199
References 199
8 Transmission of 3D Video Content 201
Abstract 201
8.1 Introduction 202
8.2 DVB-T/T2, C/C2, and S/S2 Systems 202
8.2.1 DVB-T 203
8.2.2 DVB-T2 205
8.2.3 DVB-S/S2 207
8.2.4 DVB-C/C2 208
8.2.5 Transport of 3D Video in DVB Systems 210
8.3 Hybrid Broadcast/Broadband 3DTV 214
8.4 3D Video Delivery Over IP 216
8.4.1 HTTP and RTP-Based 3D/Multi-view Streaming 216
8.4.2 3D Video Distribution Over P2P Networks 218
8.5 3D Video Distribution in ICN 220
8.6 3D Stereo and Multi-view Video in Wireless Networks 221
8.7 Conclusion 224
References 224
9 3D Video Tools 228
Abstract 228
9.1 Introduction 229
9.2 Software Tools for 3D Video Compression 230
9.2.1 H.264 and 3D Extensions 230
9.2.2 HEVC and 3D Extensions 232
9.2.3 FFmpeg 233
9.2.3.1 FFmpeg as a Research Tool 234
9.3 Streamers and 3D Video Players 236
9.3.1 OpenSVC Decoder 236
9.3.1.1 3D Open SVC Decoder Extensions 237
9.3.2 VLC Player 239
9.4 Network Simulators, Emulators, Testbeds and Network Analysis Tools 239
9.4.1 Simulators 240
9.4.1.1 Sirannon 240
9.4.1.2 DVB-T Simulator in Simulink 241
9.4.1.3 DVB-T2 Simulator in MATLAB 242
9.4.1.4 NS-2 242
9.4.1.5 NS-3 244
9.4.1.6 ndnSIM 248
9.4.2 Emulators 248
9.4.2.1 Network Link Emulators 248
9.4.2.2 Virtual Network Emulators 249
9.4.2.3 Omnet++ 249
9.4.2.4 MANE 250
9.4.3 Testbeds 252
9.4.3.1 PlanetLab 252
9.4.3.2 Network-Impairing Multimedia Testbed 253
9.4.4 Network Analysis Tools 255
9.4.4.1 Wireshark 255
9.5 3D Video Evaluation Tools 257
9.5.1 Generator of Degradations in 3D SBS Video Sequences 257
9.5.2 Crowd3D 259
9.5.3 3D MOS Using DSCQS 262
9.6 Conclusion 266
References 266
10 Quality of Experience and Quality of Service Metrics for 3D Content 271
Abstract 271
10.1 Introduction 272
10.2 Perceptual Characteristics of Multiview Content 275
10.2.1 Previous Work on QoE and QoS Assessment 278
10.2.2 Quality Assessment Based on Geometric and Spatial Distortions 279
10.2.3 Quality Based on Depth Map Analysis and Distortion 281
10.3 Subjective Quality Evaluation 284
10.3.1 Standard Methods for Subjective Quality Evaluation 284
10.3.2 Psychology/Neuroscience-Based Methodologies 286
10.3.3 High-Level QoE Factors 288
10.4 Conclusions and Future Directions 292
10.4.1 Measurement of Different Perceptual Attributes 293
10.4.2 Lack of 3D Image/Video Databases 293
10.4.3 Visual Attention Models to Develop RR and NR Quality Metrics 294
10.4.4 Need for a Standard for Subjective Experiments 294
References 295
11 3D Visual Content Datasets 302
Abstract 302
11.1 Introduction 303
11.2 Stereoscopic and Multiview Visual Content Datasets 304
11.2.1 Stereo Dataset Generation for Different Scene Cases 304
11.2.2 Multiview Camera Content for 3D Reconstruction, Modeling, and Visualization 307
11.3 Characterization and Selection of Light-Field Content for Perceptual Assessment 311
11.4 Special Point-Cloud and Holographic Content Datasets 314
11.4.1 JPEG Pleno Database: Point-Cloud Datasets 316
11.4.2 JPEG Pleno Database: Holographic Datasets 317
11.5 Datasets Annotated with Ratings from Subjective Experiments 317
11.5.1 3D Image Quality Databases 318
11.5.2 3D Video Quality Databases 320
11.5.3 3D Models Quality Databases 322
11.5.4 Eye-Tracking 3D Databases 322
11.6 Conclusions 324
References 324