Bayesian Bounds for Parameter Estimation and Nonlinear Filtering/tracking

Harry L. Van Trees, ScD, is University Professor Emeritus at George Mason University. His career spans a variety of academic, government, and industry positions. He spent fourteen years as professor of electrical engineering at Massachusetts Institute of Technology. During this period he developed the three-volume series, Detection, Estimation, and Modulation Theory, which are the classical texts in the area and still widely used throughout the world. At George Mason University, he completed Part IV of the DEMT series, Optimum Array Processing. He served as chief scientist of the U.S. Air Force, principal deputy assistant secretary of defense for C3I, president of M/A-Com-Linkabit Government Systems, and founding director of the C3I Center at George Mason University. He is currently engaged in academic research at George Mason University, and is a consultant to several companies and government agencies. Kristine L. Bell, PhD, is Associate Professor in the Department of Statistics and C3I Center at George Mason University. In her doctoral thesis, she developed a significant generalization of the Ziv-Zakai bound that expanded its range of applicability. Her current research interests are in the areas of robust, adaptive signal processing techniques and performance bounds for source location and tracking with applications in radar, sonar, aeroacoustics, and satellite communications.

Preface. Introduction (Harry L. Van Trees and Kristine L. Bell). 1 Bayesian Estimation: Static Parameters. 1.1 Maximum Likelihood and Maximum a Posteriori Estimation. 1.1.1 Nonrandom Parameters. 1.1.2 Random Parameters. 1.1.3 Hybrid Parameters. 1.1.4 Examples. 1.2 Covariance Inequality Bounds. 1.2.1 Covariance Inequality. 1.2.2 Bayesian Bounds. 1.2.3 Scalar Parameters. 1.2.3.1 Bayesian Cramer-Rao Bound. 1.2.3.2 Weighted Bayesian Cramer-Rao Bound. 1.2.3.3 Bayesian Bhattacharyya Bound. 1.2.3.4 Bobrovsky-Zakai Bound. 1.2.3.5 Weiss-Weinstein Bound. 1.2.4 Vector Parameters. 1.2.4.1 Bayesian Cramer-Rao Bound. 1.2.4.2 Weighted Bayesian CRB. 1.2.4.3 Bayesian Bhattacharyya Bound. 1.2.4.4 Bobrovsky-Zakai Bound. 1.2.4.5 Weiss-Weinstein Bound. 1.2.5 Combined Bayesian Bounds. 1.2.6 Nuisance Parameters. 1.2.6.1 Nonrandom Unwanted Parameters. 1.2.6.2 Random Parameters. 1.2.7 Hybrid Parameters. 1.2.8 Functions of the Parameter Vector. 1.2.8.1 Scalar Parameters. 1.2.8.2 Vector Parameters. 1.2.9 Summary: Covariance Inequality Bounds. 1.3 Ziv-Zakai Bounds. 1.3.1 Scalar Parameters. 1.3.2 Equally Likely Hypotheses. 1.3.3 Vector Parameters. 1.4 Method of Interval Estimation. 1.5 Summary. 2 Bayesian Estimation: Random Processes. 2.1 Continuous-Time Processes and Continuous-Time Observations. 2.1.1 Nonlinear Models. 2.1.1.1 Linear AWGN Process and Observations. 2.1.1.2 Linear AWGN Process, Nonlinear AWGN Observations. 2.1.1.3 Nonlinear AWGN Process and Observations. 2.1.1.4 Nonlinear Process and Observations. 2.1.2 Bayesian Cramer-Rao Bounds: Continuous-Time. 2.2 Continuous-Time Processes and Discrete-Time Observations. 2.2.1 Extended Kalman Filter. 2.2.2 Bayesian Cramer-Rao Bound. 2.2.3 Discretizing the Continuous-Time State Equation. 2.3 Discrete-Time Processes and Discrete-Time Observations. 2.3.1 Linear AWGN Process and Observations. 2.3.2 General Nonlinear Model. 2.3.2.1 MMSE and MAP Estimation. 2.3.2.2 Extended Kalman Filter. 2.3.3 Recursive Bayesian Cramer-Rao Bounds. 2.4 Global Recursive Bayesian Bounds. 2.5 Summary. 3 Outline of the Book. Part I Bayesian Cramer-Rao Bounds. 1.1 H. L. Van Trees, Excerpts from Part I of Detection, Estimation, and Modulation Theory, pp. 66-86, Wiley, New York, 1968 (reprinted Wiley 2001). 1.2 M. P. Shutzenberger," A generalization of the Frechet-Cramer inequality in the case of Bayes estimation," Bulletin of the American Mathematical Society, vol. 63, no. 142, 1957. Part II Global Bayesian Bounds. 2.1 B. Z. Bobrovsky, E. Mayer-Wolf, and M. Zakai, "Some classes of global Cramer-Rao bounds," Ann. Stat., vol. 15, pp. 1421-1438, 1987. 2.2 H. L. Van Trees, Excerpts from Part I of Detection, Estimation, and Modulation Theory, pp. 273-286, Wiley, New York, 1968 (reprinted 2001). 2.3 D. Rife and R. Boorstyn, "Single-tone parameter estimation from discrete-time observations," IEEE Trans. Inform. Theory, vol. IT-20, no. 5, pp. 591-598, September 1974. 2.4 R. J. McAulay and E. M. Hostetter, "Barankin bounds on parameter estimation," IEEE Trans. Info. Theory, vol. IT-17, no. 6, pp. 669-676, November 1971. 2.5 R. Miller and C. Chang, "A modified Cramer-Rao bound and its applications, IEEE Trans. Info. Theory, vol. 24, no. 3, pp. 398-400, May 1978. 2.6 A. Weiss and E. Weinstein, "A lower bound on the mean-square error in random parameter estimation," IEEE. Trans. Info. Theory, vol. 31, no. 5, pp. 680-682, September 1985. 2.7 E. Weinstein and A. J. Weiss, "Lower bounds on the mean square estimation error," Proceedings of the IEEE, vol. 73, no. 9, pp. 1433-1434, September 1985. 2.8 E. Weinstein and A. J. Weiss, "A general class of lower bounds in parameter estimation," IEEE Trans. Info. Theory, vol. 34, no. 2, pp. 338-342, March 1988. 2.9 J. S. Abel, "A bound on mean-square-estimate error," IEEE. Trans. Info. Theory, vol. 39, no. 5, pp. 1675-1680, September 1993. 2.10 A. Renaux, P. Forster, P. Larzabal, and C. Richmond, "The Bayesian Abel bound on the mean square error," ICASSP 2006, vol. 3, pp. III-9-12, Toulouse, France. 2.11 J. Ziv and M. Zakai, "Some lower bounds on signal parameter estimation," IEEE. Trans. Info. Theory, vol. IT-15, no. 3, pp. 386-391, May 1969. 2.12 L. P. Seidman, "Performance limitations and error calculations for parameter estimation," Proc. IEEE, vol. 58, no. 5, pp. 644-652, May 1970. 2.13 D. Chazan, M. Zakai, and J. Ziv, "Improved lower bounds on signal parameter estimation," IEEE Trans. Info. Theory, vol. IT-21, no. 1, pp. 90-93, Jan. 1975. 2.14 S. Bellini and G. Tartara, "Bounds on error in signal parameter estimation," IEEE. Trans. Commun., vol. COM-22, pp. 340-342, March 1974. 2.14a S. Bellini and G. Tartara, "Corrections to 'Bounds on error in signal parameter estimation,' " IEEE Trans. Commun., vol. 23, no. 4, p. 486, April 1975. 2.15 M. Wax and J. Ziv, "Improved bounds on the local mean-square error and the bias of parameter estimators," IEEE. Trans. Info. Theory, vol. 23, no. 4, pp. 529-530, July 1977. 2.16 E. Weinstein, "Relations between Belini-Tartara, Chazan-Zakai-Ziv, and Wax-Ziv lower bounds," IEEE. Trans. Info. Theory, vol. 34, no. 2, pp. 342-343, March, 1988. 2.17 K. L. Bell, Y. Steinberg, Y. Ephraim, and H. L. Van Trees, "Extended Ziv-Zakai lower bound for vector parameter estimation," IEEE. Trans. Info. Theory, vol. 43, no. 2, pp. 624-637, March 1997. 2.18 K. L. Bell, Y. Ephraim, and H. L. Van Trees, "Explicit Ziv-Zakai lower bound for bearing estimation," IEEE. Trans. Signal Process., vol. 44, no. 11, pp. 2810-2814, November 1996. 2.19 S. Basu and Y. Bresler, "A global lower bound on parameter estimation error with periodic distortion functions," IEEE. Trans. Info. Theory, vol. 46, no. 3, pp. 1145-1150, May 2000. 2.20 H. L. Van Trees, Detection, Estimation, and Modulation Theory, Part III, Chapter 10, pp. 275-308, Wiley, New York, 1971 (reprinted Wiley 2001). 2.21 F. Athley, "Threshold region performance of maximum likelihood direction of arrival estimators," IEEE Trans. on Signal Process., vol. 53, no. 4, pp. 1359-1373, April 2005. 2.22 C. D. Richmond, "Capon algorithm mean-squared error threshold SNR prediction and probability of resolution," IEEE Trans. on Signal Process., vol. 53, no. 8, pp. 2748-2764, August 2005. 2.23 C. Richmond, "Mean-squared error and threshold SNR prediction of maximum-likelihood signal parameter estimation with estimated colored noise covariances," IEEE. Trans. Info. Theory, vol. 52, no. 5, pp. 2146-2164, May 2006. 2.24 L. Najjar-Atallah, P. Larzabal, and P. Forster, "Threshold region determination of ML estimation in known phase data-aided frequency synchronization," IEEE Signal Process. Letters, vol. 12, no. 9, pp. 605-608, September 2005. 2.25 L. D. Brown and R. C. Liu, "Bounds on the Bayes and minimax risk for signal parameter estimation," IEEE. Trans. Info. Theory, vol. 39, no. 4, pp. 1386-1394, July 1993. 2.26 J. K. Thomas, L. L. Scharf, and D. W. Tufts, "The probability of a subspace swap in the SVD," IEEE Trans. Signal Process., vol. 43, no. 3, pp. 730-736, March 1995. Part III Hybrid Bayesian Bounds. 3.1 Y. Rockah and P. M. Schultheiss, "Array shape calibration using sources in unknown locations-Part I: far-field sources," IEEE Trans. Acoust., Speech Signal Process., vol. 35, no. 3, pp. 286-299, March 1987. 3.2 I. Reuven and H. Messer, "A Barankin-type lower bound on the estimation error of a hybrid parameter vector," IEEE. Trans. Info. Theory, vol. 43, no. 3, pp. 1084-1093, May 1997. 3.3 J. Tabrikian and J. Krolik, "Efficient computation of the Bayesian Cramer-Rao bound on estimating parameters of Markov models," IEEE Conf. Acoustics, Speech, and Sig. Process., ICASSP'99, pp. 1761-1764, 1999. 3.4 S. Buzzi, M. Lops, and S. Sardellitti, "Further results on Cramer-Rao bounds for parameter estimation in long-code DS/CDMA systems," IEEE Trans. Sig. Process., vol. 53, no. 3, pp. 1216-1221, March 2005. 3.5 P. Tichavsky' and K. Wong, "Quasi-fluid-mechanics-based quasi-Bayesian Cramer-Rao bounds for deformed towed-array direction finding," IEEE Trans. Signal Process., vol. 52, no. 1, pp. 36-47, Jan. 2004. Part IV Constrained Cramer-Rao Bounds. 4.1 J. D. Gorman and A. O. Hero, "Lower bounds for parametric estimation with constraints," IEEE. Trans. Info. Theory, vol. 36, no. 6, pp. 1285-1301, November 1990. 4.2 T. L. Marzetta, "A simple derivation of the constrained multiple parameter Cramer-Rao bound," IEEE. Trans. Signal Process., vol. 41, no. 6, pp. 2247-2249, June 1993. 4.3 P. Stoica and B. C. Ng, "On the Cramer-Rao bound under parametric constraints," IEEE Signal Process Letters, vol. 5, no. 7, pp. 177-179, July 1998. 4.4 T. J. Moore, B. M. Sadler, and R. J. Kozick, "Regularity and strict identifiability in MIMO systems," IEEE. Trans. Signal Process., vol. 50, no. 8, pp. 1831-1842, August 2002. 4.5 S. T. Smith, "Covariance, subspace, and intrinsic Cramer-Rao bounds," IEEE. Trans. Signal Process., vol. 53, no. 5, pp. 1610-1630, May 2005. 4.6 A. O. Hero, J. A. Fessler, and M. Usman, "Exploring estimator bias-variance tradeoffs using the uniform CR bound," IEEE. Trans. Signal Process., vol. 44, no. 8, pp. 2026-2041, August 1996. 4.7 A. Nehorai and M. Hawkes, "Performance bounds for estimating vector systems," IEEE. Trans. Signal Process., vol. 48, no. 6, pp. 1737-1749, June 2000. 4.8 L. T. McWhorter and L. L. Scharf, "Properties of quadratic covariance bounds," Proc. 27th Annual Asilomar Conf. on Signals, Systems, and Computers, Asilomar, CA, vol. 2, pp. 1176-1180, November 1993. Part V Applications: Static Parameters. 5.1 A. Weiss and E. Weinstein, "Fundamental limitations in passive time delay estimation-Part I: Narrow-band systems," IEEE Trans. Acoustics, Speech, Sig. Process., vol. ASSP-31, no. 2, pp. 472-486, April 1983. 5.2 A. Bartow and H. Messer, "Lower bound on the achievable DSP performance for localizing step-like continuous signals in noise," IEEE. Trans. Signal Process., vol. 46, no. 8, pp. 2195-2201, August 1998. 5.3 B. Sadler and R. Kozick, "A survey of time delay estimation performance bounds," Fourth IEEE Workshop on Sensor Array and Multichannel Process., pp. 282-288, 12-14 July 2006. 5.4 W. Xu, A. Baggeroer, and C. Richmond, "Bayesian bounds for matched-field parameter estimation," IEEE. Trans. Signal Process., vol. 52, no. 12, pp. 3293-3305, December, 2004. 5.5 J. Tabrikian and J. Krolik, "Barankin bounds for source localization in an uncertain ocean environment," IEEE. Trans. Signal Process., vol. 47, no. 11, pp. 2917-2927, November 1999. 5.6 U. Oktel and R. Moses, "A Bayesian approach to array geometry design," IEEE. Trans. Signal Process., vol. 53, no. 5, pp. 1919-1923, May 2005. 5.7 F. Athley, "Optimization of element positions for direction finding with sparse arrays," 11th IEEE SPW on Stat. Proc., pp. 516-519, August 2001. 5.8 F. Athley and C. Engdahl, "Direction-of-arrival estimation using separated subarrays," 34th IEEE Asilomar Conf. on SSC, vol. 1, pp. 585-589, October 2001. 5.9 H. Nguyen and and H. L. Van Trees, "Comparison of performance bounds for DOA estimation," IEEE Seventh SP Workshop on Statistical Signal and Array Processing, pp. 313-316, June 1994. 5.10 Y. Qi and H. Kobayashi, "On geolocation accuracy with prior information in non-line-of-sight environment," IEEE 56th Vehicular Tech. Conf. Proc., vol. 1, pp. 285-288, Sept. 2002. 5.11 S. C. White and N. C. Beaulieu, "On the application of the Cramer-Rao and detection theory bounds to mean square error of symbol timing recovery," IEEE Trans. Comm., vol. 40, no. 10, pp. 1635-1643, October 1992. 5.12 A. Pinkus and J. Tabrikian, "Barankin bound for range and Doppler estimation using orthogonal signal transmission," IEEE Conf. on Radar, pp. 94-99, April 2006. 5.13 J. Tabrikian, "Barankin bounds for target localization by MIMO radars," Fourth IEEE Workshop on Sensor Array and Multichannel Process., pp. 278-281, July 2006. 5.14 A. Renaux, "Weiss-Weinstein bound for data-aided carrier estimation," IEEE Sig. Proc. Letters, vol. 14, no. 4, pp. 283-286, April 2007. Part VI Nonlinear Stochastic Dynamic Systems. 6.1 H. J. Kushner, "On the differential equations satisfied by conditional probability densities of Markov processes, with applications," J. SIAM on Control, vol. 2, pp. 106-119, 1964. 6.2 Y. C. Ho and R. C. K. Lee, "A Bayesian approach to problems in stochastic estimation and control," IEEE Trans. Auto. Control, vol. 9, no. 4, pp. 333-339, October 1964. 6.3 H. Cox, "On the estimation of state variables and parameters for noisy dynamic systems," IEEE Trans. Auto. Control, vol. 9, no. 1, pp. 5-12, January 1964. 6.4 H. L. Van Trees, "Bounds on the accuracy attainable in the estimation of continuous random processes," IEEE Trans. Info. Theory, vol. 12, no. 3, pp. 298-305, July 1966. 6.5 D. L. Snyder and I. B. Rhodes, "Filtering and control performance bounds with implication on asymptotic Separation," Automatica, vol. 8, pp. 747-753, Nov. 1972. 6.6 B. Z. Bobrovsky, and M. Zakai, "A lower bound on the estimation error for Markov processes," IEEE Trans. Auto. Control, vol. 20, no. 6, 785-788, December 1975. 6.7 B. Z. Bobrovsky and M. Zakai, "A lower bound on the estimation error for certain diffusion processes," IEEE Trans. Info. Theory, vol. IT-22, no. 1, pp. 45-52, January 1976. 6.8 B. Bobrovsky, M. Zakai, and O. Zeitouni, "Error ounds for the nonlinear filtering of signals with small diffusion coefficients," IEEE Trans. Info. Theory, vol. 34, no. 4, pp. 710-721, July 1988. 6.9 J. H. Taylor, "The Cramer-Rao estimation error lower bound computation for deterministic nonlinear systems," IEEE Trans. Auto. Control, vol. 24, pp. 343-344, April 1979. 6.10 C. B. Chang, "Two lower bounds on the covariance for nonlinear estimation problems," IEEE Trans. Auto. Control, vol. AC-26, no. 6, pp. 1294-1297, Dec. 1981. 6.11 H. L. Van Trees, Excerpts from Part II of Detection, Estimation, and Modulation Theory, pp. 134-153, Wiley, New York, 1971 (reprinted Wiley 2003). 6.12 M. Zakai and J. Ziv, "Lower and upper bounds on the optimal filtering error of certain diffusion processes," IEEE Trans. Info. Theory, vol. IT-18, no. 3, pp. 325-331, May 1972. 6.13 P. Tichavsky, C. Muravchik, and A. Nehorai, "Posterior Cramer-Rao bounds for discrete-time nonlinear filtering," IEEE Trans. Signal Process., vol. 46, no. 5, pp. 1386-1396, May 1998. 6.14 M. Simandl, J. Kralovec, and P. Tichavsky, "Filtering, predictive and smoothing Cramer-Rao bounds for discrete-time nonlinear dynamic systems," Automatica, vol. 37, pp. 1703-1716, 2001. 6.15 I. Rapoport and Y. Oshman, "Recursive Weiss-Weinstein lower bounds for discrete-time nonlinear filtering," 43rd IEEE Conf. on Decision and Control, vol. 3, pp. 2662-2667, Dec. 2004. 6.16 S. Reece and D. Nicholson, "Tighter alternatives to the Cramer-Rao lower bound for discrete-time filtering," 7th Intl. Conf. on Info. Fusion, vol. 1, pp. 101-106, 25-28 July 2005. 6.17 M. S. Arulampalam, S. Maskell, N. Gordon and T. Clapp, "A tutorial on particle filters for online nonlinear/non-Gaussian Bayesian tracking," IEEE Trans. Sig. Process, vol. 50, no. 2, pp. 174-188, February 2002. Part VII Applications: Nonlinear Dynamic Systems. 7.1 V. Aidala and S. Hammel, "Utilization of modified polar coordinates for bearings-only tracking," IEEE Trans. Auto. Control, vol. AC-28, no. 3, pp. 283-294, March 1983. 7.2 M. Hernandez, B. Ristic, and A. Farina, "A performance bound for manoeuvering target tracking using best-fitting Gaussian distribution," Proc. 7th Int. Conf. Information Fusion, FUSION 2005," Philadelphia, PA, pp. 1-8, July 2005. 7.3 J. M. Passerieux and D. Van Cappel, "Optimal observer maneuver for bearings-only tracking," IEEE Trans. Aero. and Elect. Syst., vol. 34, no. 3, pp. 777-788, July 1998. 7.4 K. L. Bell and H. L. Van Trees, "Posterior Cramer-Rao bound for tracking target bearing," 13th Annual Workshop on Adaptive Sensor Array Process. (ASAP 2005), MIT Lincoln Lab, Lexington, MA, June 2005. 7.5 R. Niu, P. Willett and Y. Bar-Shalom, "Matrix CRLB scaling due to measurements of uncertain origin," IEEE Trans. Signal Process., vol. 49, no. 7, pp. 1325-1335, July 2001. 7.6 M. Hernandez, A. Farina, and B. Ristic, "PCRLB for tracking in cluttered environments: Measurement sequence conditioning approach," IEEE Trans. Aero. Elect. Syst., vol. 42, no. 2, pp. 680-704, April 2006. 7.7 X. Zhang, P. Willett, and Y. Bar-Shalom, "Dynamic Cramer-Rao bound for target tracking in clutter," IEEE Trans. Aero. Elect. Syst., vol. 41, no. 4, pp. 1154-1167, October 2005. 7.8 B. Ristic, A. Farina, and M. Hernandez, "Cramer-Rao lower bound for tracking multiple targets," IEE Proc. on Radar, Sonar and Navig., vol. 151, no. 3, pp. 129-134, June 2004. 7.9 C. Hue, J-P. Le Cadre, and P. Perez, "Posterior Cramer-Rao bounds for multi-target tracking," IEEE Trans. Aero. Elect. Syst., vol. 42, no. 1, pp. 37-49, January 2006. 7.10 F. E. Daum, "Bounds on performance for multiple target tracking," IEEE Trans. Auto. Control, vol. 35, no. 4, pp. 443-446, April 1990. 7.11 N. Bergman, L. Ljung, and F. Gustafsson, "Point-mass filter and Cramer-Rao bound for terrain-aided navigation," IEEE Proc. of the 36th IEEE Conf. on Decision & Control, vol. 1, pp. 565-570, December 1997. 7.12 H. L. Van Trees, K. L. Bell, and Y. Wang, "Bayesian Cramer-Rao bounds for multistatic radar," IEEE International Waveform Diversity & Design Conference, January 2006. 7.13 I. Rapoport and Y. Oshman, "Weiss-Weinstein lower bounds for Markovian systems. Part 2: Applications to fault-tolerant filtering," IEEE Trans. Signal Process., vol. 55, no. 2, pp. 2031-2042, May 2007. 7.14 K. L. Bell and H. L. Van Trees, "Combined Cramer-Rao/Weiss-Weinstein bound for tracking target bearing," 4th Annual IEEE Workshop on Sensor Array and Multi-Channel Processing (SAM 2006),Waltham, MA, pp. 273-277, July 2006. Part VIII Statistical Literature. 8.1 R. D. Gill and B. Y. Levit, "Applications of the Van Trees inequality: A Bayesian Cramer-Rao bound," Bernoulli 1 (1/2), pp. 59-79, 1995. 8.2 B. L. S. Prakasa Rao, "On Cramer-Rao type integral inequalities." Calcutta Stat. Assoc. Bulletin, (H. K. Nandi Mem. Spec. Vol.), vol. 40, nos. 157-60, pp. 183-205, 1991. 8.3 J. J. Gart, "An extension of the Cramer-Rao inequality." Annals Math. Stat., vol. 30, no. 2, pp. 367-380, 1959. 8.4 M. Ghosh, "Cramer-Rao bounds for posterior variances." Stat. Prob. Letters, vol. 17, pp. 173-178, 1993. References. Author Index.