Matrix Differential Calculus with Applications in Statistics and Econometrics

JAN R. MAGNUS is Emeritus Professor at the Department of Econometrics & Operations Research, Tilburg University, and Extraordinary Professor at the Department of Econometrics & Operations Research, Vrije University, Amsterdam. He is research fellow of CentER and the Tinbergen Institute. He has co-authored nine books and is the author of over 100 scientific papers. HEINZ NEUDECKER (1933-2017) was Professor of Econometrics at the University of Amsterdam from 1972 until his retirement in 1998.

Preface xiii

Part One — Matrices

1 Basic properties of vectors and matrices 3

1 Introduction 3

2 Sets 3

3 Matrices: addition and multiplication 4

4 The transpose of a matrix 6

5 Square matrices 6

6 Linear forms and quadratic forms 7

7 The rank of a matrix 9

8 The inverse 10

9 The determinant 10

10 The trace 11

11 Partitioned matrices 12

12 Complex matrices 14

13 Eigenvalues and eigenvectors 14

14 Schur’s decomposition theorem 17

15 The Jordan decomposition 18

16 The singular-value decomposition 20

17 Further results concerning eigenvalues 20

18 Positive (semi)definite matrices 23

19 Three further results for positive definite matrices 25

20 A useful result 26

21 Symmetric matrix functions 27

Miscellaneous exercises 28

Bibliographical notes 30

2 Kronecker products, vec operator, and Moore-Penrose inverse 31

1 Introduction 31

2 The Kronecker product 31

3 Eigenvalues of a Kronecker product 33

4 The vec operator 34

5 The Moore-Penrose (MP) inverse 36

6 Existence and uniqueness of the MP inverse 37

7 Some properties of the MP inverse 38

8 Further properties 39

9 The solution of linear equation systems 41

Miscellaneous exercises 43

Bibliographical notes 45

3 Miscellaneous matrix results 47

1 Introduction 47

2 The adjoint matrix 47

3 Proof of Theorem 3.1 49

4 Bordered determinants 51

5 The matrix equation AX = 0 51

6 The Hadamard product 52

7 The commutation matrix Kmn 54

8 The duplication matrix Dn 56

9 Relationship between Dn+1 and Dn, I 58

10 Relationship between Dn+1 and Dn, II 59

11 Conditions for a quadratic form to be positive (negative) subject to linear constraints 60

12 Necessary and sufficient conditions for r(A : B) = r(A) + r(B) 63

13 The bordered Gramian matrix 65

14 The equations X1A + X2B′ = G1,X1B = G2 67

Miscellaneous exercises 69

Bibliographical notes 70

Part Two — Differentials: the theory

4 Mathematical preliminaries 73

1 Introduction 73

2 Interior points and accumulation points 73

3 Open and closed sets 75

4 The Bolzano-Weierstrass theorem 77

5 Functions 78

6 The limit of a function 79

7 Continuous functions and compactness 80

8 Convex sets 81

9 Convex and concave functions 83

Bibliographical notes 86

5 Differentials and differentiability 87

1 Introduction 87

2 Continuity 88

3 Differentiability and linear approximation 90

4 The differential of a vector function 91

5 Uniqueness of the differential 93

6 Continuity of differentiable functions 94

7 Partial derivatives 95

8 The first identification theorem 96

9 Existence of the differential, I 97

10 Existence of the differential, II 99

11 Continuous differentiability 100

12 The chain rule 100

13 Cauchy invariance 102

14 The mean-value theorem for real-valued functions 103

15 Differentiable matrix functions 104

16 Some remarks on notation 106

17 Complex differentiation 108

Miscellaneous exercises 110

Bibliographical notes 110

6 The second differential 111

1 Introduction 111

2 Second-order partial derivatives 111

3 The Hessian matrix 112

4 Twice differentiability and second-order approximation, I 113

5 Definition of twice differentiability 114

6 The second differential 115

7 Symmetry of the Hessian matrix 117

8 The second identification theorem 119

9 Twice differentiability and second-order approximation, II 119

10 Chain rule for Hessian matrices 121

11 The analog for second differentials 123

12 Taylor’s theorem for real-valued functions 124

13 Higher-order differentials 125

14 Real analytic functions 125

15 Twice differentiable matrix functions 126

Bibliographical notes 127

7 Static optimization 129

1 Introduction 129

2 Unconstrained optimization 130

3 The existence of absolute extrema 131

4 Necessary conditions for a local minimum 132

5 Sufficient conditions for a local minimum: first-derivative test 134

6 Sufficient conditions for a local minimum: second-derivative test 136

7 Characterization of differentiable convex functions 138

8 Characterization of twice differentiable convex functions 141

9 Sufficient conditions for an absolute minimum 142

10 Monotonic transformations 143

11 Optimization subject to constraints 144

12 Necessary conditions for a local minimum under constraints 145

13 Sufficient conditions for a local minimum under constraints 149

14 Sufficient conditions for an absolute minimum under constraints 154

15 A note on constraints in matrix form 155

16 Economic interpretation of Lagrange multipliers 155

Appendix: the implicit function theorem 157

Bibliographical notes 159

Part Three — Differentials: the practice

8 Some important differentials 163

1 Introduction 163

2 Fundamental rules of differential calculus 163

3 The differential of a determinant 165

4 The differential of an inverse 168

5 Differential of the Moore-Penrose inverse 169

6 The differential of the adjoint matrix 172

7 On differentiating eigenvalues and eigenvectors 174

8 The continuity of eigenprojections 176

9 The differential of eigenvalues and eigenvectors: symmetric case 180

10 Two alternative expressions for dλ 183

11 Second differential of the eigenvalue function 185

Miscellaneous exercises 186

Bibliographical notes 189

9 First-order differentials and Jacobian matrices 191

1 Introduction 191

2 Classification 192

3 Derisatives 192

4 Derivatives 194

5 Identification of Jacobian matrices 196

6 The first identification table 197

7 Partitioning of the derivative 197

8 Scalar functions of a scalar 198

9 Scalar functions of a vector 198

10 Scalar functions of a matrix, I: trace 199

11 Scalar functions of a matrix, II: determinant 201

12 Scalar functions of a matrix, III: eigenvalue 202

13 Two examples of vector functions 203

14 Matrix functions 204

15 Kronecker products 206

16 Some other problems 208

17 Jacobians of transformations 209

Bibliographical notes 210

10 Second-order differentials and Hessian matrices 211

1 Introduction 211

2 The second identification table 211

3 Linear and quadratic forms 212

4 A useful theorem 213

5 The determinant function 214

6 The eigenvalue function 215

7 Other examples 215

8 Composite functions 217

9 The eigenvector function 218

10 Hessian of matrix functions, I 219

11 Hessian of matrix functions, II 219

Miscellaneous exercises 220

Part Four — Inequalities

11 Inequalities 225

1 Introduction 225

2 The Cauchy-Schwarz inequality 226

3 Matrix analogs of the Cauchy-Schwarz inequality 227

4 The theorem of the arithmetic and geometric means 228

5 The Rayleigh quotient 230

6 Concavity of λ1 and convexity of λn 232

7 Variational description of eigenvalues 232

8 Fischer’s min-max theorem 234

9 Monotonicity of the eigenvalues 236

10 The Poincar´e separation theorem 236

11 Two corollaries of Poincar´e’s theorem 237

12 Further consequences of the Poincar´e theorem 238

13 Multiplicative version 239

14 The maximum of a bilinear form 241

15 Hadamard’s inequality 242

16 An interlude: Karamata’s inequality 242

17 Karamata’s inequality and eigenvalues 244

18 An inequality concerning positive semidefinite matrices 245

19 A representation theorem for ( ∑api )1/p 246

20 A representation theorem for (trAp)1/p 247

21 Hölder’s inequality 248

22 Concavity of log|A| 250

23 Minkowski’s inequality 251

24 Quasilinear representation of |A|1/n 253

25 Minkowski’s determinant theorem 255

26 Weighted means of order p 256

27 Schlömilch’s inequality 258

28 Curvature properties of Mp(x, a) 259

29 Least squares 260

30 Generalized least squares 261

31 Restricted least squares 262

32 Restricted least squares: matrix version 264

Miscellaneous exercises 265

Bibliographical notes 269

Part Five — The linear model

12 Statistical preliminaries 273

1 Introduction 273

2 The cumulative distribution function 273

3 The joint density function 274

4 Expectations 274

5 Variance and covariance 275

6 Independence of two random variables 277

7 Independence of n random variables 279

8 Sampling 279

9 The one-dimensional normal distribution 279

10 The multivariate normal distribution 280

11 Estimation 282

Miscellaneous exercises 282

Bibliographical notes 283

13 The linear regression model 285

1 Introduction 285

2 Affine minimum-trace unbiased estimation 286

3 The Gauss-Markov theorem 287

4 The method of least squares 290

5 Aitken’s theorem 291

6 Multicollinearity 293

7 Estimable functions 295

8 Linear constraints: the case M(R′) ⊂M(X′) 296

9 Linear constraints: the general case 300

10 Linear constraints: the case M(R′) ∩M(X′) = {0} 302

11 A singular variance matrix: the case M(X) ⊂M(V ) 304

12 A singular variance matrix: the case r(X′V +X) = r(X) 305

13 A singular variance matrix: the general case, I 307

14 Explicit and implicit linear constraints 307

15 The general linear model, I 310

16 A singular variance matrix: the general case, II 311

17 The general linear model, II 314

18 Generalized least squares 315

19 Restricted least squares 316

Miscellaneous exercises 318

Bibliographical notes 319

14 Further topics in the linear model 321

1 Introduction 321

2 Best quadratic unbiased estimation of σ2 322

3 The best quadratic and positive unbiased estimator of σ2 322

4 The best quadratic unbiased estimator of σ2 324

5 Best quadratic invariant estimation of σ2 326

6 The best quadratic and positive invariant estimator of σ2 327

7 The best quadratic invariant estimator of σ2 329

8 Best quadratic unbiased estimation: multivariate normal case 330

9 Bounds for the bias of the least-squares estimator of σ2, I 332

10 Bounds for the bias of the least-squares estimator of σ2, II 333

11 The prediction of disturbances 335

12 Best linear unbiased predictors with scalar variance matrix 336

13 Best linear unbiased predictors with fixed variance matrix, I 338

14 Best linear unbiased predictors with fixed variance matrix, II 340

15 Local sensitivity of the posterior mean 341

16 Local sensitivity of the posterior precision 342

Bibliographical notes 344

Part Six — Applications to maximum likelihood estimation

15 Maximum likelihood estimation 347

1 Introduction 347

2 The method of maximum likelihood (ML) 347

3 ML estimation of the multivariate normal distribution 348

4 Symmetry: implicit versus explicit treatment 350

5 The treatment of positive definiteness 351

6 The information matrix 352

7 ML estimation of the multivariate normal distribution: distinct means 354

8 The multivariate linear regression model 354

9 The errors-in-variables model 357

10 The nonlinear regression model with normal errors 359

11 Special case: functional independence of mean and variance parameters 361

12 Generalization of Theorem 15.6 362

Miscellaneous exercises 364

Bibliographical notes 365

16 Simultaneous equations 367

1 Introduction 367

2 The simultaneous equations model 367

3 The identification problem 369

4 Identification with linear constraints on B and Γ only 371

5 Identification with linear constraints on B, Γ, and ∑ 371

6 Nonlinear constraints 373

7 FIML: the information matrix (general case) 374

8 FIML: asymptotic variance matrix (special case) 376

9 LIML: first-order conditions 378

10 LIML: information matrix 381

11 LIML: asymptotic variance matrix 383

Bibliographical notes 388

17 Topics in psychometrics 389

1 Introduction 389

2 Population principal components 390

3 Optimality of principal components 391

4 A related result 392

5 Sample principal components 393

6 Optimality of sample principal components 395

7 One-mode component analysis 395

8 One-mode component analysis and sample principal components 398

9 Two-mode component analysis 399

10 Multimode component analysis 400

11 Factor analysis 404

12 A zigzag routine 407

13 A Newton-Raphson routine 408

14 Kaiser’s varimax method 412

15 Canonical correlations and variates in the population 414

16 Correspondence analysis 417

17 Linear discriminant analysis 418

Bibliographical notes 419

Part Seven — Summary

18 Matrix calculus: the essentials 423

1 Introduction 423

2 Differentials 424

3 Vector calculus 426

4 Optimization 429

5 Least squares 431

6 Matrix calculus 432

7 Interlude on linear and quadratic forms 434

8 The second differential 434

9 Chain rule for second differentials 436

10 Four examples 438

11 The Kronecker product and vec operator 439

12 Identification 441

13 The commutation matrix 442

14 From second differential to Hessian 443

15 Symmetry and the duplication matrix 444

16 Maximum likelihood 445

Further reading 448

Bibliography 449

Index of symbols 467

Subject index 471