Understanding Markov Chains (eBook)

The author is an associate professor from the Nanyang Technological University (NTU) and is well-established in the field of stochastic processes and a highly respected probabilist. He has authored the book, Stochastic Analysis in Discrete and Continuous Settings: With Normal Martingales, Lecture Notes in Mathematics, Springer, 2009 and was a co-editor for the book, Stochastic Analysis with Financial Applications, Progress in Probability, Vol. 65, Springer Basel, 2011. Aside from these two Springer titles, he has authored several others. He is currently teaching the course M27004-Probability Theory and Stochastic Processes at NTU. The manuscript has been developed over the years from his courses on Stochastic Processes.

Introduction1 Probability Background1.1 Probability Spaces and Events1.2 Probability Measures1.3 Conditional Probabilities and Independence1.4 Random Variables1.5 Probability Distributions1.6 Expectation of a Random Variable1.7 Conditional Expectation1.8 Moment and Probability Generating FunctionsExercises2 Gambling Problems2.1 Constrained Random Walk2.2 Ruin Probabilities2.3 Mean Game DurationExercises3 Random Walk3.1 Unrestricted Random Walk3.2 Mean and Variance3.3 Distribution3.4 First Return to ZeroExercises4 Discrete-Time Markov Chains4.1 Markov Property4.2 Transition matrix4.3 Examples of Markov Chains4.4 Higher Order Transition Probabilities4.5 The Two-State Discrete-Time Markov ChainExercises5 First Step Analysis5.1 Hitting Probabilities5.2 Mean Hitting and Absorption Times5.3 First Return Times5.4 Number of ReturnsExercises6 Classication of States6.1 Communicating States6.2 Recurrent States6.3 Transient States6.4 Positive and Null Recurrence6.5 Periodicity and AperiodicityExercises7 Long-Run Behavior of Markov Chains7.1 Limiting Distributions7.2 Stationary Distributions7.3 Markov Chain Monte CarloExercises8 Branching Processes8.1 Defnition and Examples8.2 Probability Generating Functions8.3 Extinction ProbabilitiesExercises9 Continuous-Time Markov Chains9.1 The Poisson Process9.2 Continuous-Time Chains9.3 Transition Semigroup9.4 Infinitesimal Generator9.5 The Two-State Continuous-Time Markov Chain9.6 Limiting and Stationary Distributions9.7 The Discrete-Time Embedded Chain9.8 Mean Absorption Time and ProbabilitiesExercises10 Discrete-Time Martingales10.1 Filtrations and Conditional Expectations10.2 Martingales - Definition and Properties10.3 Ruin Probabilities10.4 Mean Game DurationExercises11 Spatial Poisson Processes11.1 Spatial Poisson (1781-1840) Processes11.2 Poisson Stochastic Integrals11.3 Transformations of Poisson Measures11.4 Moments of Poisson Stochastic Integrals11.5 Deviation InequalitiesExercises12 Reliability Theory12.1 Survival Probabilities12.2 Poisson Process with Time-DependentIntensity12.3 Mean Time to FailureExercisesSome Useful IdentitiesSolutions to the ExercisesReferencesIndex