Probability Concepts and Theory for Engineers

Professor Harry Schwarzlander, Department of Electrical Engineering and Computer Science, Syracuse University, Syracuse, New York, USA Harry Schwarzlander is Associate Professor Emeritus at Syracuse University and has been with the university since 1964 where he has developed and taught 25 courses to electrical engineering graduate and undergraduate students. He was an Instructor in the Department of Electrical Engineering at Purdue University from 1960 to 1964, and before that, an Engineer and Project Engineer for General Electronic Laboratories, Inc., Cambridge, Massachusetts. Professor Schwarzlander is a Registered Professional Engineer in New York and a Life Member of IEEE, taking posts as Secretary and Chairman between 1967 and 1969. In 2004 he was awarded Doctor Honoris Causa 'in recognition of outstanding accomplishments, exemplary educational leadership and distinguished service to mankind' by The International Institute for Advanced Studies in Systems Research and Cybernetics. He holds one patent for the RMS-Measuring Voltmeter, 1959. Currently Executive Director of The New Environment, Inc. and Editor of New Environment Bulletin (the monthly newsletter of the New Environment Association), Professor Schwarzlander has contributed to over 65 publications and presentations. He researches into a range of different areas, including interference testing of electronic equipment and information storage and retrieval.

Preface xi

Introduction xiii

Part I The Basic Model

Part I Introduction 2

Section 1 Dealing with ‘Real-World’ Problems 3

Section 2 The Probabilistic Experiment 6

Section 3 Outcome 11

Section 4 Events 14

Section 5 The Connection to the Mathematical World 17

Section 6 Elements and Sets 20

Section 7 Classes of Sets 23

Section 8 Elementary Set Operations 26

Section 9 Additional Set Operations 30

Section 10 Functions 33

Section 11 The Size of a Set 36

Section 12 Multiple and Infinite Set Operations 40

Section 13 More About Additive Classes 44

Section 14 Additive Set Functions 49

Section 15 More about Probabilistic Experiments 53

Section 16 The Probability Function 58

Section 17 Probability Space 62

Section 18 Simple Probability Arithmetic 65

Part I Summary 71

Part II The Approach to Elementary Probability Problems

Part II Introduction 74

Section 19 About Probability Problems 75

Section 20 Equally Likely Possible Outcomes 81

Section 21 Conditional Probability 86

Section 22 Conditional Probability Distributions 91

Section 23 Independent Events 99

Section 24 Classes of Independent Events 104

Section 25 Possible Outcomes Represented as Ordered k-Tuples 109

Section 26 Product Experiments and Product Spaces 114

Section 27 Product Probability Spaces 120

Section 28 Dependence Between the Components in an Ordered k-Tuple 125

Section 29 Multiple Observations Without Regard to Order 128

Section 30 Unordered Sampling with Replacement 132

Section 31 More Complicated Discrete Probability Problems 135

Section 32 Uncertainty and Randomness 140

Section 33 Fuzziness 146

Part II Summary 152

Part III Introduction to Random Variables

Part III Introduction 154

Section 34 Numerical-Valued Outcomes 155

Section 35 The Binomial Distribution 161

Section 36 The Real Numbers 165

Section 37 General Definition of a Random Variable 169

Section 38 The Cumulative Distribution Function 173

Section 39 The Probability Density Function 180

Section 40 The Gaussian Distribution 186

Section 41 Two Discrete Random Variables 191

Section 42 Two Arbitrary Random Variables 197

Section 43 Two-Dimensional Distribution Functions 202

Section 44 Two-Dimensional Density Functions 208

Section 45 Two Statistically Independent Random Variables 216

Section 46 Two Statistically Independent Random Variables—Absolutely Continuous Case 221

Part III Summary 226

Part IV Transformations and Multiple Random Variables

Part IV Introduction 228

Section 47 Transformation of a Random Variable 229

a) Transformation of a discrete random variable 229

b) Transformation of an arbitrary random variable 231

c) Transformation of an absolutely continuous random variable 235

Section 48 Transformation of a Two-Dimensional Random Variable 238

Section 49 The Sum of Two Discrete Random Variables 243

Section 50 The Sum of Two Arbitrary Random Variables 247

Section 51 n-Dimensional Random Variables 253

Section 52 Absolutely Continuous n-Dimensional R.V.’s 259

Section 53 Coordinate Transformations 263

Section 54 Rotations and the Bivariate Gaussian Distribution 268

Section 55 Several Statistically Independent Random Variables 274

Section 56 Singular Distributions in One Dimension 279

Section 57 Conditional Induced Distribution, Given an Event 284

Section 58 Resolving a Distribution into Components of Pure Type 290

Section 59 Conditional Distribution Given the Value of a Random Variable 293

Section 60 Random Occurrences in Time 298

Part IV Summary 304

Part V Parameters for Describing Random Variables and Induced Distributions

Part V Introduction 306

Section 61 Some Properties of a Random Variable 307

Section 62 Higher Moments 314

Section 63 Expectation of a Function of a Random Variable 320

a) Scale change and shift of origin 320

b) General formulation 320

c) Sum of random variables 322

d) Powers of a random variable 323

e) Product of random variables 325

Section 64 The Variance of a Function of a Random Variable 328

Section 65 Bounds on the Induced Distribution 332

Section 66 Test Sampling 336

a) A Simple random sample 336

b) Unbiased estimators 338

c) Variance of the sample average 339

d) Estimating the population variance 341

e) Sampling with replacement 342

Section 67 Conditional Expectation with Respect to an Event 345

Section 68 Covariance and Correlation Coefficient 350

Section 69 The Correlation Coefficient as Parameter in a Joint Distribution 356

Section 70 More General Kinds of Dependence Between Random Variables 362

Section 71 The Covariance Matrix 367

Section 72 Random Variables as the Elements of a Vector Space 374

Section 73 Estimation 379

a) The concept of estimating a random variable 379

b) Optimum constant estimates 379

c) Mean-square estimation using random variables 381

d) Linear mean-square estimation 382

Section 74 The Stieltjes Integral 386

Part V Summary 393

Part VI Further Topics in Random Variables

Part VI Introduction 396

Section 75 Complex Random Variables 397

Section 76 The Characteristic Function 402

Section 77 Characteristic Function of a Transformed Random Variable 408

Section 78 Characteristic Function of a Multidimensional Random Variable 412

Section 79 The Generating Function 417

Section 80 Several Jointly Gaussian Random Variables 422

Section 81 Spherically Symmetric Vector Random Variables 428

Section 82 Entropy Associated with Random Variables 435

a) Discrete random variables 435

b) Absolutely continuous random variables 438

Section 83 Copulas 443

Section 84 Sequences of Random Variables 454

a) Preliminaries 454

b) Simple gambling schemes 455

c) Operations on sequences 458

Section 85 Convergent Sequences and Laws of Large Numbers 461

a) Convergence of sequences 461

b) Laws of large numbers 464

c) Connection with statistical regularity 468

Section 86 Convergence of Probability Distributions and the Central Limit Theorem 470

Part VI Summary 477

Appendices 479

Answers to Queries 479

Table of the Gaussian Integral 482

Part I Problems 483

Part II Problems 500

Part III Problems 521

Part IV Problems 537

Part V Problems 556

Part VI Problems 574

Notation and Abbreviations 587

References 595

Subject Index 597