1
Introduction

The scene: my video coding research lab at the Robert Gordon University, Aberdeen, Scotland, 1994. ‘You're working on video compression?’ asked my visitor. ‘Isn't that going to be unnecessary in a few years? After all, network bandwidth is increasing every year. We've already got 10 Mbps Ethernet and we'll soon have 100 Mbps. Video compression will be redundant by the year 2000’.

My visitor predicted that research into video compression would fade away as ever‐increasing bandwidth and storage capacity made it possible to send uncompressed, full‐quality video with ease. He offered me a wager, that my research topic, video coding, would be redundant by the year 2000. If only I had taken that bet! In fact, at the present time, the need for efficient and effective video coding is perhaps greater than ever. Generation and consumption of video content have grown more quickly than available bandwidth has in recent years. This is because of the emergence of user‐generated content that comes most notably from mobile devices. Consumption has also increased through changing viewing patterns – often with multiple simultaneous streams being viewed in a single household – and the move to higher‐resolution content, such as High Definition and beyond.

This book is about video coding, also known as video compression. A video encoder converts digital video into a compressed form, that is, a form that takes up less storage or transmission capacity, known as coded video. A video decoder does the opposite, converting coded video into uncompressed digital video. The combination of a video encoder and a video decoder is known as a video codec, and the processes of compression and decompression are often described as video coding.

1.1 Why Write This Book?

I have written four books about video coding. Why write another one?

It is now 2024, and video coding is embedded in modern life as never before. Compressed video made a major impact in the 1990s with the advent of digital television and DVD video. Other applications followed throughout the early 2000s, and now compressed video is a fundamental part of an ever‐expanding list of consumer and business applications. These include, but are not limited to:

• Recording, playing and sharing video on mobile devices.
• Video calling and videoconferencing, both of which received a massive boost following the COVID lockdowns of 2020.
• Video streaming, which is now overtaking digital TV broadcasting as the dominant way in which we watch video.
• Video in social media, with platforms such as Facebook, Instagram and TikTok increasingly relying on video as well as still images.
• Security and surveillance, from commercial security cameras to police bodycams and video‐enabled doorbells.
• Automotive video, from dashcams to in‐car entertainment.

Since I wrote my last book in 2010, we have seen a significant increase in the use of video compression and a shift in the way coded video is used, as well as the release of a series of new industry standards and formats for video compression. These formats include H.265/High Efficiency Video Coding (), H.266/Versatile Video Coding, VP9, AV1 and more.

These new standards and formats are publicly available documents. However, the published standards are not intended for the casual reader and can be challenging to understand. This is due in part to their focus on highly detailed technical descriptions of how standard‐based video codecs work. Even a reader with a solid background in computer science or electronic engineering may need a more approachable way into the video coding standards.

I have spent much of my career explaining video coding to students, researchers, engineers and professionals. I have always tried to find approachable and intuitive explanations of how video coding works. I wrote this book primarily to gather this material in one place, and I hope that it can provide a more user‐friendly guide to video coding that will enable readers to engage with the video coding standards and understand how to get the most out of video compression.

1.2 What Is in the Book?

In this book, I give an overview of digital video and video compression and a short history of some of the key concepts and standards. I then discuss each of the main core concepts of video coding. I explain each concept in general terms and discuss how each is put into practice in certain video coding standards. I have chosen to focus on the widely used H.26x series of standards, with a particular focus on H.265/. Each chapter is liberally illustrated with diagrams and examples.

In Chapter 2, I provide an overview of a video codec and introduce the concepts of performance and visual quality.

Chapter 3 is a short history of video coding. This chapter begins with a description of the basic concepts that developed between the 1950s and 1990s and goes on to describe the major video coding standards from H.261 through to H.266 and AV1.

Chapter 4 explains the structural elements used in video coding. A video codec processes a video by breaking it down into structures such as Groups of Pictures, pictures, slices, tiles, basic coding units, prediction blocks and transform blocks.

Chapters 5 and 6 deal with prediction. The information in a block of video pixels can be significantly reduced if we can predict the pixels from previously sent information. Intra prediction (see Chapter 5) involves predicting blocks using information that is available in the same frame of video. This is known as intra‐frame or spatial prediction. Inter prediction (see Chapter 6) creates predictions using information that is available in other, previously coded frames. This is known as inter‐frame or temporal prediction.

Chapter 7 examines transforms and quantisation. A transform converts a block of pixels, or difference samples, into another domain in which the important visual information is concentrated into a smaller number of values. Quantisation removes some of the resulting information, deliberately reducing the precision of the data in order to compress it, hopefully without sacrificing too much visual quality.

Chapter 8 looks at the final stage in a video encoder, entropy coding, in which data are converted into a compressed bitstream. I discuss the main types of entropy coding, with a particular emphasis on arithmetic coding, which has become the method of choice for entropy coding in recent video coding standards.

Chapter 9 deals with filtering, in particular the types of video filtering that are carried out as part of a video encoding or decoding process. In this context, video pixels are filtered to try to improve the performance of all the other compression processes.

In Chapter 10, I discuss how coded video can be stored and transmitted. Video is often stored together with audio in container files such as an MP4 file. Transport or transmission of video is an area that continues to develop, with the emergence of adaptive bitrate streaming as one of the most important ways of sending video across the internet.

Chapter 11 considers how video codecs are implemented in software or hardware and how the performance of video codecs can be measured and compared. I provide some suggestions for trying this yourself, for example, by experimenting with the various public‐domain software video codecs and by analysing coded video to see how it is actually put together.

1.3 How Should You Use This Book?

I wrote this book with several audiences in mind. Each chapter starts by setting out basic concepts and continues into more technical detail. I have deliberately not assumed specialised background knowledge on the part of the reader. I hope that the book will be useful to:

• Students who are studying multimedia processing and communications.
• Academics and researchers, as the book presents a way to understand and approach the extensive research literature on video coding and provides a platform to develop ideas and research topics.
• Engineers and implementers, as a bridge between the basic concepts of video coding and the often challenging density of the video coding standards.
• Other professionals, as a more accessible way into this important technical topic.

I have written the book so that it can be read from start to finish, and I hope that at least some readers will do just that. Of course, you may want to dip in and out and concentrate on particular topics of interest. I would recommend reading Chapter 2, which will give an overview of how each topic fits into the overall video codec system or model. From Chapter 4 onwards, each chapter begins with an explanation of the basic concepts of the video compression process such as prediction and transform. The chapter then explains how the process is put into practice according to specific standards with a particular focus on H.265/HEVC.

At some point, you may want to go deeper into the topics presented here. For example, you might want to find out exactly how motion vectors are communicated in H.265 or H.266. There are many excellent articles in journals that can explain topics further, such as the IEEE Transactions on Circuits and Systems for Video Technology, which has published several special issues and special sections on H.265/HEVC [1] and a special section on...