Information

D.T. Pham

Publisher Summary

The blind source separation (BSS) is aimed at reconstructing the sources from the observations. In a blind context, the separation of sources can only rely on the basic knowledge, which is their mutual independence. The mutual information and the independence criterion offering several benefits are adopted. First, it is invariant with respect to invertible. In particular, it is scale invariant thus avoids a prewhitening step, which is needed in many other separation methods. Second, it is a very general and complete independence criterion: it is non-negative and can be zero if and only if there is independence. Some other criteria such as the cumulants are only partial: to ensure independence, one needs to check that all (cross) cumulants vanish, but in practice only a finite number of them can be considered. Finally, the mutual information can be interpreted in terms of entropy and the Kullback–Leibler divergence, and is closely related to the expected log likelihood. This chapter covers the use of the mutual information between the observations at a given time, which is suitable for instantaneous (linear) mixtures, in which the temporal dependence of the source sequences is ignored. It also covers the use of the information rate between stationary processes, which is necessary to treat the case of convolutive (linear) mixtures and can also be useful for the case of instantaneous mixtures when there is strong temporal dependence of the source sequences.

Chapter Outline

2.1 Introduction

Blind source separation (BSS) deals typically with a mixing model of the form1(⋅)=A{s(⋅)} where (n) and (n) represent the source and observed vectors at time and is a transformation, which can be instantaneous (operating on each (n) to produce (n)), or global (operating on the whole sequence (⋅) of source vectors. The goal of BSS is to reconstruct the sources from the observations. Clearly, for this task to be possible should not be completely unknown: it should belong to some class of transformations given a priori. Most common classes are the class of linear (affine) instantaneous transformations and that of (linear) convolutions. They correspond to linear instantaneous mixtures and (linear) convolutive mixtures respectively. Nonlinear transformations have also been considered. For example, may be constituted of linear instantaneous (or convolutive) transformation followed by nonlinear instantaneous transformation operating component-wise. The corresponding mixtures are called post-nonlinear (or convolutive post-nonlinear). This chapter deals primarily with linear mixtures; nonlinear mixtures are treated elsewhere (see Chapter 14).

In a blind context, the separation of sources can only rely on the basic knowledge which is their mutual independence. It is thus natural to try to achieve separation by minimizing an independence criterion between the components of −1{x(⋅)} among all ∈A, where −1 denotes the inverse transformation of . We adopt here the mutual information and the independence criterion. This is a popular criterion and has many appeals. Firstly, it is invariant with respect to invertible transformation (see Lemma 2.1 below and what follows). In particular, it is scale invariant thus avoids a prewhitening step, which is needed in many other separation methods. Secondly, it is a very general and complete independence criterion: it is non-negative and can be zero if and only if there is independence. Some other criteria such as the cumulants are only partial: to ensure independence, one needs to check that all (cross) cumulants vanish, but in practice only a finite number of them can be considered. Finally, the mutual information can be interpreted in terms of entropy and Kullback-Leibler divergence [3], and is closely related to the expected log likelihood [1]. The downsize is that this criterion requires the knowledge of the joint density of the components of −1{x(⋅)}, which is unknown in practice and hence must be replaced by some nonparametric estimate. The estimation procedure can be quite costly computationally. We shall however introduce some methods which are not much costlier than using simpler criteria (such as the cumulants).

The rest of this chapter contains two parts: the first one concerns the use of the mutual information between the observations at a given time, which is suitable for instantaneous (linear) mixtures, in which the temporal dependence of the source sequences are ignored (for simplicity or because it is weak). The second part concerns the use of the information rate between stationary processes, which is necessary to treat the case of convolutive (linear) mixtures, but can also be useful for the case of instantaneous mixtures when there is strong temporal dependence of the source sequences. Note that for the convolutive mixture, the sources can be recovered only up to filtering (as it will be seen later) and one may require temporal independence of the source to lift this ambiguity. The problem then may be viewed as the multi-channel blind deconvolution problem as it reduces to the well-known deconvolution problem when both source and observed sequences are scalar. We however call this problem blind separation-deconvolution as it aims to both recover the sources (separation) and make them temporally independence (deconvolution).

2.2 Methods based on mutual information

We first define mutual information and provide its main properties.

2.2.1 Mutual information between random vectors

Let 1,…,yP, be random vectors with joint densities y1,…,yP and marginal density y1,…,pyP. The mutual information between these vectors is defined as the Kullback-Leibler divergence (or relative entropy) between the densities k=1Ppyk and y1,…,yP:

{y1…,yP}=−Elogpy1(y1)⋯pyP(yP)py1,…,yP(y1,…,yP).

This measure is non-negative (but can be ∞) and can vanish only if the random vectors are mutually independent [4]. It can also be written as:

{y1…,yP}=∑i=1PH{yk}−H{y1…,yP} (2.1)

where (y1…,yP) and (y1),…,H(yP) are the joint entropy and the marginal entropies of 1,…,yP:

{y1…,yP}=−Elogpy1,…,yP(y1…,yP) (2.2)

and {yk} is defined in the same way with yk in place of y1,…,yP. The notation {y1…,yP} is the same as {[y1T⋯yPT]T}, T denoting the transpose.

The entropy possesses the following interesting property of equivariance with respect to invertible transformation.

This result can be easily obtained from the definition (2.2) of entropy and the equality x(x)=py[g(x)]|detg′(x)|. It follows immediately from (2.1) that if is a transformation operating component-wise (that is the -th component of (x) depends only on the -th component of ) then the mutual information between the components of the transformed random vector (x) is the same as that between the components of the original random vector . Thus the mutual information between the random variables 1,…,yP remains unchanged when one applies to each of them an invertible transformation. Clearly, it is also unchanged if one permutes the random variables. This is consistent with the fact that independent random variables remain independent under the above operations. As a result, one can separate the sources from their linear mixtures only up to a scaling and a permutation (by exploiting their independence only).

The application of Lemma 2.1 is most interesting in the case of the linear (affine) mixture, since then one is dispensed with an expectation calculation: if...