Video Data Compression

B.G. Haskell    Visual Communications Research Department, AT&T Bell Laboratories, Holmdel, New Jersey

1.1 Introduction

A considerable effort has been underway for some time to develop inexpensive transmission techniques that take advantage of recent advances in electronic technology as well as expected future developments. Most of the attention has been focused on digital systems because, as is well known, noise does not accumulate in digital regenerators as it does in analog amplifiers and, in addition, signal processing is much easier in a digital format.

Progress is being made on two fronts. First, the present high cost per bit of transmitting a digital data stream has generated interest in a number of methods that are currently being evaluated for cost reduction. While these methods have general applications and are not confined to a data stream produced by a video signal source, it is important to remember that video bit rates tend to be considerably higher than those required for voice or data transmission. The most promising techniques for more economical digital transmission include optical fibers, digital satellite, broadband ISDN, and digital transmission over the air, among others.

The second front on which progress is being made involves reducing the number of bits that have to be transmitted in a video communication system. Bit-rate reduction is accomplished by eliminating, as much as possible, the substantial amount of redundant information that exists in a video signal as it leaves the camera. The amount of signal processing required to reduce the redundancy determines the economic feasibility of using this method in a given system. The savings that accrue from lowering the transmission bit rate must more than offset the cost of the required signal processing if redundancy reduction is to be economical.

Present costs of digital logic and digital memory are low enough to make this type of signal processing economically very attractive for use in long distance videoconferencing links over existing facilities. Furthermore, it is expected that the cost of digital logic and memory will continue to decline. Therefore, it is conjectured by those knowledgeable in the field that signal processing for bit-rate reduction will have an important part to play in all video systems, and in many cases, it could become the overriding factor determining economic feasibility.

To transmit video information at the minimum bit rate for a given quality of reproduction, it is necessary to exploit our understanding of many branches of science. Ideally the engineer should have an appreciation of motion pictures, colorimetry, human vision, signal theory, display devices, and so on. As might be expected any individual can have only a smattering of knowledge on such a diverse range of topics, and a specialist in any one topic will readily confess to a certain amount of ignorance even in his or her chosen field. As engineers we are concerned with complex stimuli and their human perception, as well as the final utilization of the perceived information. Knowledge of these is often unavailable or sketchy, forcing us to design encoders based on a relatively primitive understanding of the problem. The limits of bit-rate compression will be approached, we believe, only as our knowledge of stimuli, perception, and utilization increases.

Thus, in opening a discussion of video bit-rate compression we are very aware of our own limitations. Our modest objective of defining the state of the art is, we are well aware, open to the criticisms of oversimplification, serious omissions, and factual disagreement. As for where the subject is heading and its inherent limitations, we confess myopia and will not be surprised by a discovery that could not have been extrapolated from existing thinking and known ignorances. But first let us set the stage for our discussion. The conventional representation of a digital communication link for the transmission of audio or pictorial information is shown in Fig. 1.1. The function of the source encoder is to operate on an analog of audio or pictures, x(t), and to convert it into a stream of binary digits, s(t). The source decoder at the receiver accepts a binary signal S(t) and produces a continuous signal X(t). It may not be necessary to ensure X(t) = x(t), but what does matter is that after transduction, e.g., loudspeaker or TV tube, X(t) should be perceived as x(t), subject to an acceptable quality criterion. Although x(t) does not always have to be identical to X(t), system engineers prefer s(t) = S(t); i.e., the channel appears ideal. Most practical channels contain dispersion, nonlinearities, additive noise, multipath fading, interference from other channels, and so on. These imperfections are overcome largely by preprocessing and postprocessing the binary signals s(t) and S(t) by the channel codec and terminal equipment. The transmitting terminal equipment operates on c(t) to produce (perhaps by conversion to multilevel, modulation, filtering, etc.) a signal f(t) that is suitable for combating the imperfections of the communication channel. The signal F(t) that emerges from the channel may differ considerably from f(t). After demodulation, a binary signal C(t) is regenerated using adaptive equalization of the channel and adaptive detection strategies. The binary signal C(t) is then channel and source decoded to produce S(t).

The purpose of this book is to discuss mostly source encoding. However, Fig. 1.1 demonstrates that S(t) is dependent on the channel terminal equipment, the channel codec, and of course, the channel. Thus, encoding picture signals is not merely a source encoding problem, but may include the complete communication system. For example, if the channel is known to result in a high bit error rate (ber), then the effect on the recovered signal X(t) may be mitigated by altering the modulation and regeneration strategies, increasing the length of the check bits in the channel coding words, altering the source encoding algorithm, or combinations of all of these. The conventional arrangement of source and channel codecs may be altered, even merged. Postprocessing of X(t) can also be successfully employed.

Thus, we are interested in the source codec, its algorithms, how they relate to the signals it encodes, how the bit rate can be reduced by exploiting the source signal statistics and properties of human perception, the variety of quality criteria, the codec complexity, and above all, how these phenomena are interrelated and can be traded to approach an optimum design.

We therefore present a discussion of picture sources and our scant knowledge of the salient properties of human perception. Armed with this we describe the current state of the art in waveform and parameter coding and conclude with directions for the future, guessing at where we believe some ultimate limitations may be found.

1.1.1 Picture Sources

Video processing or transmission systems typically start with a two-dimensional distribution of light intensity. Thus, three-dimensional scenes must first be projected onto a two-dimensional plane by an optical imaging system. Color pictures can usually be represented by three such light intensity distributions in three primary bands of wavelengths. If moving objects are to be accommodated, the light intensity must change with time.

The two-dimensional light intensity distribution is then usually raster scanned to produce a one-dimensional waveform. Facsimile involves single pictures, while in television the scene is repetitively raster scanned (usually with interlace to avoid flicker). Black/white pictures, e.g., printed or handwritten text, line drawings, weather maps, produce a two-level or binary waveform.

Color pictures produce three such waveforms corresponding to the three primaries. These are then usually converted by linear combination into a luminance (monochrome brightness) component and two chrominance (hue and saturation) components. Multiplexing methods for further combining these components into a single composite waveform are well known and widely used; however, the luminance component usually takes up most of the channel capacity.

1.1.2 The Eye and Seeing

The eye is the organ of sight, having at its rear an inner nervous coating known as the retina. Rays of light pass through the cornea, aqueous humor, lens, and vitreous body to form an image on the retina. The central area of the retina, known as the fovea, provides high resolution and good color vision in about 1 degree of solid angle. The images on the retinas are sent along two optic nerves, one for each eye, until they meet at the optic chiasma, where half the fibers of each nerve diverge to opposite sides of the brain. This enables observations in three dimensions.

The eye behaves as a two-dimensional low-pass filter for...