Markov Decision Processes with Their Applications (eBook)

Contents 6
List of Figures 10
List of Tables 11
Preface 12
Acknowledgments 14
INTRODUCTION 15
1. A Brief Description of Markov Decision Processes 15
2. Overview of the Book 18
3. Organization of the Book 20
DISCRETE TIME MARKOV DECISION PROCESSES: TOTAL REWARD 24
1. Model and Preliminaries 24
1.1 System Model 24
1.2 Some Concepts 25
1.3 Finiteness of the Reward 27
2. Optimality Equation 30
2.1 Validity of the Optimality Equation 30
2.2 Properties of the Optimality Equation 34
3. Properties of Optimal Policies 38
4. Successive Approximation 43
5. Sufficient Conditions 45
6. Notes and References 47
Problems 48
DISCRETE TIME MARKOV DECISION PROCESSES: AVERAGE CRITERION 52
1. Model and Preliminaries 52
2. Optimality Equation 56
2.1 Properties of ACOE and Optimal Policies 57
2.2 Sufficient Conditions 61
2.3 Recurrent Conditions 63
3. Optimality Inequalities 66
3.1 Conditions 67
3.2 Properties of ACOI and Optimal Policies 70
4. Notes and References 73
Problems 74
CONTINUOUS TIME MARKOV DECISION PROCESSES 75
1. A Stationary Model: Total Reward 75
1.1 Model and Conditions 75
1.2 Model Decomposition 79
1.3 Some Properties 83
1.4 Optimality Equation and Optimal Policies 89
2. A Nonstationary Model: Total Reward 97
2.1 Model and Conditions 97
2.2 Optimality Equation 99
3. A Stationary Model: Average Criterion 107
4. Notes and References 113
Problems 114
SEMI-MARKOV DECISION PROCESSES 116
1. Model and Conditions 116
1.1 Model 116
1.2 Regular Conditions 118
1.3 Criteria 121
2. Transformation 122
2.1 Total Reward 123
2.2 Average Criterion 126
3. Notes and References 130
Problems 131
MARKOV DECISION PROCESSES IN SEMI- MARKOV ENVIRONMENTS 132
1. Continuous Time Markov Decision Processes in Semi- Markov Environments 132
1.1 Model 132
1.2 Optimality Equation 138
1.3 Approximation byWeak Convergence 148
1.4 Markov Environment 151
1.5 Phase Type Environment 154
2. Semi-Markov Decision Processes in Semi-Markov Environments 159
2.1 Model 159
2.2 Optimality Equation 163
2.3 Markov Environment 169
3. Mixed Markov Decision Processes in Semi-Markov Environments 171
3.1 Model 171
3.2 Optimality Equation 174
3.3 Markov Environment 181
4. Notes and References 185
Problems 186
OPTIMAL CONTROL OF DISCRETE EVENT SYSTEMS: I 187
1. System Model 187
2. Optimality 190
2.1 Maximum Discounted Total Reward 192
2.2 Minimum Discounted Total Reward 196
3. Optimality in Event Feedback Control 196
4. Link to Logic Level 199
5. Resource Allocation System 204
6. Notes and References 211
Problems 212
OPTIMAL CONTROL OF DISCRETE EVENT SYSTEMS: II 213
1. System Model 213
2. Optimality Equation and Optimal Supervisors 217
3. Language Properties 223
4. System Based on Automaton 225
5. Supervisory Control Problems 228
5.1 Event Feedback Control 228
5.2 State Feedback Control 232
6. Job-Matching Problem 233
7. Notes and References 240
Problems 240
OPTIMAL REPLACEMENT UNDER STOCHASTIC ENVIRONMENTS 242
1. Optimal Replacement: Discrete Time 243
1.1 Problem and Model 243
1.2 Total Cost Criterion 247
1.3 Average Criterion 250
2. Optimal Replacement: Semi-Markov Processes 253
2.1 Problem 253
2.2 Optimal Control Limit Policies 256
2.3 Markov Environment 259
2.4 Numerical Example 267
3. Notes and References 269
Problems 271
OPTIMAL ALLOCATION IN SEQUENTIAL ONLINE AUCTIONS 273
1. Problem and Model 273
2. Analysis for Private Reserve Price 275
3. Analysis for Announced Reserve Price 279
4. Monotone Properties 281
5. Numerical Results 290
6. Notes and References 292
Problems 293
References 295
Index 303