Modeling and Analysis of Stochastic Systems

Vidyadhar G. Kulkarni

IntroductionWhat in the World is a Stochastic Process?How to Characterize a Stochastic ProcessWhat Do We Do with a Stochastic Process?Discrete-Time Markov Chains: Transient BehaviourDefinition and CharacterizationExamplesDTMCs in Other FieldsMarginal DistributionsOccupancy TimesComputation of Matrix PowersModeling ExercisesComputational ExercisesConceptual ExercisesDiscrete-Time Markov Chains: First Passage TimesDefinitionsCumulative Distribution Function of TAbsorption ProbabilitiesExpectation of TGenerating Function and Higher Moments of TComputational ExercisesConceptual ExercisesDiscrete-Time Markox Chains: Limiting BehaviourExploring the Limiting Behaviour by ExamplesClassification of StatesDetermining Recurrence and Transience: Finite DTMCsDetermining Recurrence and Transience: Infinite DTMScLimiting Behaviour of Irreducible DTMCsExamples: Limiting Behaviour of Infinite State-Space Irreducible DTMCsLimiting Behaviour of Reducible DTMCsDTMCs with Costs and RewardsReversibilityComputational ExercisesConceptual ExercisesPoisson ProcessesExponential DistributionsPoisson Process: DefinitionsEvent Times in a Poisson ProcessSuperposition and Splitting of Poisson ProcessesNon-Homogeneous Poisson ProcessCompound Poisson ProcessComputational ExercisesConceptual ExercisesContinuous-Time Markov ChainsDefinitions and Sample Path PropertiesExamplesCTMCs in Other FieldsTransient Behaviour: Marginal DistributionTransient Behaviour: Occupancy TimesComputation of P(t): Finite State-SpaceComputation of P(t): Infinite State-SpaceFirst-Passage TimesExploring the Limiting Behaviour by ExamplesClassification of StatesLimiting Behaviour of Irre