Motivated Reinforcement Learning (eBook)

Preface 5
Acronyms 9
Contents 10
Part I Non-Player Characters and Reinforcement Learning 14
Chapter 1 Non-Player Characters in Multiuser Games 15
1.1 Types of Multiuser Games 16
1.1.1 Massively Multiplayer Online Role-Playing Games 16
1.1.2 Multiuser Simulation Games 17
1.1.3 Open-Ended Virtual Worlds 17
1.2 Character Roles in Multiuser Games 20
1.3 Existing Artificial Intelligence Techniques for Non- Player Characters in Multiuser Games 21
1.3.1 Reflexive Agents 21
1.3.2 Learning Agents 24
1.3.3 Evolutionary Agents 26
1.3.4 Smart Terrain 26
1.4 Summary 27
1.5 References 27
Chapter 2 Motivation in Natural and Artificial Agents 29
2.1 Defining Motivation 29
2.2 Biological Theories of Motivation 32
2.2.1 Drive Theory 32
2.2.2 Motivational State Theory 34
2.2.3 Arousal 35
2.3 Cognitive Theories of Motivation 38
2.3.1 Curiosity 38
2.3.2 Operant Theory 40
2.3.3 Incentive 41
2.3.4 Achievement Motivation 42
2.3.5 Attribution Theory 43
2.3.6 Intrinsic Motivation 45
2.4 Social Theories of Motivation 47
2.4.1 Conformity 47
2.4.2 Cultural Effect 48
2.4.3 Evolution 48
2.5 Combined Motivation Theories 49
2.5.1 Maslow’s Hierarchy of Needs 50
2.5.2 Existence Relatedness Growth Theory 50
2.6 Summary 51
2.7 References 52
Chapter 3 Towards Motivated Reinforcement Learning 56
3.1 Defining Reinforcement Learning 56
3.1.1 Dynamic Programming 58
3.1.2 Monte Carlo Methods 59
3.1.3 Temporal Difference Learning 60
3.2 Reinforcement Learning in Complex Environments 63
3.2.1 Partially Observable Environments 63
3.2.2 Function Approximation 64
3.2.3 Hierarchical Reinforcement Learning 65
3.3 Motivated Reinforcement Learning 68
3.3.1 Using a Motivation Signal in Addition to a Reward Signal 69
3.3.2 Using a Motivation Signal Instead of a Reward Signal 75
3.4 Summary 78
3.5 References 79
Chapter 4 Comparing the Behaviour of Learning Agents 82
4.1 Player Satisfaction 82
4.1.1 Psychological Flow 83
4.1.2 Structural Flow 84
4.2 Formalising Non-Player Character Behaviour 84
4.2.1 Models of Optimality for Reinforcement Learning 85
4.2.2 Characteristics of Motivated Reinforcement Learning 89
4.3 Comparing Motivated Reinforcement Learning Agents 92
4.3.1 Statistical Model for Identifying Learned Tasks 94
4.3.2 Behavioural Variety 94
4.3.3 Behavioural Complexity 96
4.4 Summary 97
4.5 References 98
Part II Developing Curious Characters Using Motivated Reinforcement Learning 100
Chapter 5 Curiosity, Motivation and Attention Focus 101
5.1 Agents in Complex, Dynamic Environments 101
5.1.1 States 103
5.1.2 Actions 104
5.1.3 Reward and Motivation 104
5.2 Motivation and Attention Focus 105
5.2.1 Observations 106
5.2.2 Events 108
5.2.3 Tasks and Task Selection 110
5.2.4 Experience-Based Reward as Cognitive Motivation 112
5.2.5 Arbitration Functions 118
5.2.6 A General Experience-Based Motivation Function 119
5.3 Curiosity as Motivation for Support Characters 121
5.3.1 Curiosity as Interesting Events 121
5.3.2 Curiosity as Interest and Competence 126
5.4 Summary 129
5.5 References 129
Chapter 6 Motivated Reinforcement Learning Agents 131
6.1 A General Motivated Reinforcement Learning Model 131
6.2 Algorithms for Motivated Reinforcement Learning 133
6.2.1 Motivated Flat Reinforcement Learning 133
6.2.2 Motivated Multioption Reinforcement Learning 136
6.2.3 Motivated Hierarchical Reinforcement Learning 141
6.3 Summary 143
6.4 References 144
Part III Curious Characters in Games 145
Chapter 7 Curious Characters for Multiuser Games 146
7.1 Motivated Reinforcement Learning for Support Characters in Massively Multiplayer Online Role-Playing Games 147
7.2 Character Behaviour in Small-Scale, Isolated Game Locations 150
7.2.1 Case Studies of Individual Characters 151
7.2.2 General Trends in Character Behaviour 154
7.3 Summary 157
7.4 References 158
Chapter 8 Curious Characters for Games in Complex, Dynamic Environments 159
8.1 Designing Characters That Can Multitask 160
8.1.1 Case Studies of Individual Characters 163
8.1.2 General Trends in Character Behaviour 164
8.2 Designing Characters for Complex Tasks 167
8.2.1 Case Studies of Individual Characters 167
8.2.2 General Trends in Character Behaviour 169
8.3 Games That Change While Characters Are Learning 171
8.3.1 Case Studies of Individual Characters 172
8.3.2 General Trends in Character Behaviour 175
8.4 Summary 177
8.5 References 178
Chapter 9 Curious Characters for Games in Second Life 179
9.1 Motivated Reinforcement Learning in Open-Ended Simulation Games 179
9.1.1 Game Design 180
9.1.2 Character Design 180
9.2 Evaluating Character Behaviour in Response to Game Play Sequences 184
9.2.1 Discussion 195
9.3 Summary 196
9.4 References 197
Part IV Future 198
Chapter 10 Towards the Future 199
10.1 Using Motivated Reinforcement Learning in Non-Player Characters 199
10.2 Other Gaming Applications for Motivated Reinforcement Learning 200
10.2.1 Dynamic Difficulty Adjustment 200
10.2.2 Procedural Content Generation 201
10.3 Beyond Curiosity 201
10.3.1 Biological Models of Motivation 201
10.3.2 Cognitive Models of Motivation 202
10.3.3 Social Models of Motivation 202
10.3.4 Combined Models of Motivation 202
10.4 New Models of Motivated Learning 203
10.4.1 Motivated Supervised Learning 203
10.4.2 Motivated Unsupervised Learning 204
10.5 Evaluating the Behaviour of Motivated Learning Agents 204
10.6 Concluding Remarks 204
10.7 References 205
Index 206