Why Joint Source and Channel Decoding?

This chapter introduces theoretical justifications for joint source-channel coding (JSCC) and identifies certain expectations that one may have regarding JSCC and joint source-channel decoding (JSCD). While we mainly focus on a justification for JSCC, we also discuss JSCD toward the end of the chapter.

While JSCC was, initially, of interest to only a small group of researchers, it has, more recently, gained widespread attention. For some of the readers just learning about this field, the main concept may seem somewhat awkward: Is it really possible to forget Shannon’s separation theorem? For other readers, Shannon’s work may be remembered only vaguely. For these reasons, we start with a brief review of Shannon theory, we carefully explain the origin of the separation theorem, we outline limitations on source coding and on channel coding, and we define an important quantity – the optimum performance theoretically attainable (OPTA) – for the communication of a particular source over a particular channel. From these foundations, the assumptions under which the separation theorem holds become clear, and the expectations that one has when working on JSCC can be made more explicit.

These expectations are illustrated in two ways: In a first step, we describe some simple scenarios where, even though the separation theorem holds, separation is not the only approach to optimal system design. In a second step, we summarize key ways in which practical situations differ from the abstract model that was used to justify the separation theorem. We concentrate on the broadcast scenario, which, given modern applications, is more prevalent than the point-to-point scenario, and which, by its nature, results in imperfectly known channel parameters. In this case, it is shown that the separation theorem does not hold.

Our presentation closely follows the studies of Zahir Azami et al. (1996) and Gatspar et al. (2003).

2.1 INFORMATION THEORETIC PRELIMINARIES

This section provides the basic information theoretic tools that are needed to understand the Shannon bounds in their various incarnations. While, with channel coding, it is quite common to check the performance of a proposed system against the best attainable performance, it is not as common in source coding, even when one knows the rate-distortion curves. One of the reasons for this is that the corresponding computations are likely to be intractable, as a result of the high correlation found in practical source signals like video. In addition, it is less well known that a similar bound exists for JSCC, which is the real target of this book. In fact, one can evaluate, for simple sources, the minimum rate that is required to communicate the given source over a particular channel with a specified maximum end-to-end distortion. This point is clarified later in this section.

2.1.1 The Situation of Interest

When deriving bounds, one typically requires a simple model for the situation of interest. We use here the classical transmission model, as depicted in Figure 2.1. We now carefully analyze this situation, while keeping in mind the constraints that are required for the theoretical results to have practical utility.

We consider only block processing; the inputs and all intermediate quantities are vector valued, as defined below:

• The initial and reconstructed source words have k components (i.e., the “source symbols”), X = (X1, X2, …, Xk), and ^=(X^1,X^2,…,X^k).

• The channel input and output words have n components (i.e., the “channel symbols”), B = (B1, B2, …, Bn), and Y = (Y1, Y2, …, Yn).

All variables are modeled as random variables and, for generality, are not assumed to be either discrete or continuous. For this reason, following the notations in the studies by Rioul (2007) and Zahir Azami et al. (1996), we use

to simultaneously denote summation, for discrete variables, and integration, for continuous variables (where we omit the measure symbol d(.) for simplicity). With this notation, for example, expectation reads

  (2.1)

This way, we can address a wide variety of sources and channels using a common notation.

The source is characterized by its probability distribution p(x) = p(x1, …, xk). For example, a zero-mean i.i.d. Gaussian source is described by

(x)=1(2πσx2)k/2exp(−‖x‖22σx2),

where x2 denotes the source variance, and a memoryless binary symmetric source (BSS) is described by the uniform distribution

(x)=12k.

The channel is described by the transition probabilities p(y|b) that relate the input b to the output y, i.e., the probability distribution of the output variable Y for a given realization of the input b. For example, an additive white Gaussian noise (AWGN) channel is characterized by

(y|b)=1(2πσn2)n/2exp(−‖ y−b ‖22σn2),

where n2 denotes the noise variance. A binary symmetric channel (BSC) is modeled by its crossover probability p such that

(y|b)=pd(1−p)n−d,

where d = dH(b, y) denotes the Hamming distance between the channel input and output.

We allow any combination of input and channel types, e.g., a Gaussian signal over a Gaussian channel, a Gaussian signal over a BSC, a binary signal over a Gaussian channel, a binary signal over a Gaussian channel, or even multivalued signals. Unless stated otherwise, the equations we provide hold in all of these cases.

Of course, when mixing input and channel types, one must adapt the source to the channel; this is the role of the encoder. The encoder is described by the mapping b = C(x). Specifying C amounts to designing channel code words characterized by their probability distribution p(b).

Conversely, one needs a decoder to convert the channel output to a reconstructed word in the source domain. The decoder is described by the mapping ^=D(y). Specifying D amounts to designing source code words according to the distribution (x^|x).

The model is not yet complete since important degrees of freedom still remain. For example, when designing the transmission system, one needs to choose the channel input power. While increased channel input power generally yields improved performance (except in a few pathological cases), it comes with practical costs. For this reason, a power constraint is typically adopted. In a similar manner, one needs to decide on an acceptable level of end-to-end reconstruction error ^−X. Various measures can be used to quantify the error, including error probability (when the source is discrete), mean square error (MSE), and so on. The best choice depends on the...