Applied Reinforcement Learning with Python (eBook)

Table of Contents 5
About the Author 9
About the Technical Reviewer 10
Acknowledgments 11
Introduction 12
Chapter 1: Introduction to Reinforcement Learning 13
History of Reinforcement Learning 14
MDPs and their Relation to  Reinforcement Learning 15
Reinforcement Learning Algorithms and RL Frameworks 19
Q Learning 22
Actor-Critic Models 23
Applications of Reinforcement Learning 24
Classic Control Problems 24
Super Mario Bros. 25
Doom 26
Reinforcement-Based Marketing Making 27
Sonic the Hedgehog 28
Conclusion 29
Chapter 2: Reinforcement Learning Algorithms 30
OpenAI Gym 30
Policy-Based Learning 31
Policy Gradients Explained Mathematically 33
Gradient Ascent Applied to Policy Optimization 35
Using Vanilla Policy Gradients on the Cart Pole Problem 36
What Are Discounted Rewards and Why Do We Use Them? 40
Drawbacks to Policy Gradients 47
Proximal Policy Optimization (PPO) and Actor-Critic Models 48
Implementing PPO and Solving Super Mario Bros. 49
Overview of Super Mario Bros. 50
Installing Environment Package 51
Structure of the Code in Repository 51
Model Architecture 52
Working with a More Difficult Reinforcement Learning Challenge 58
Dockerizing Reinforcement Learning Experiments 61
Results of the Experiment 63
Conclusion 64
Chapter 3: Reinforcement Learning Algorithms: Q Learning and Its Variants 65
Q Learning 65
Temporal Difference (TD) Learning 67
Epsilon-Greedy Algorithm 69
Frozen Lake Solved with Q Learning 70
Deep Q Learning 75
Playing Doom with Deep Q Learning 76
Simple Doom Level 81
Training and Performance 83
Limitations of Deep Q Learning 84
Double Q Learning and Double Deep Q Networks 84
Conclusion 85
Chapter 4: Market Making via Reinforcement Learning 87
What Is Market Making? 87
Trading Gym 91
Why Reinforcement Learning for This Problem? 92
Synthesizing Order Book Data with Trading Gym 94
Generating Order Book Data with Trading Gym 95
Experimental Design 97
RL Approach 1: Policy Gradients 100
RL Approach 2: Deep Q Network 101
Results and Discussion 103
Conclusion 104
Chapter 5: Custom OpenAI Reinforcement Learning Environments 105
Overview of Sonic the Hedgehog 105
Downloading the Game 106
Writing the Code for the Environment 108
A3C Actor-Critic 113
Conclusion 121
Appendix A: Source Code 123
Market Making Model Utilities 123
Policy Gradient Utilities 125
Models 126
Chapter 1 135
OpenAI Example 135
Chapter 2 135
Cart Pole Example 135
Super Mario Example 140
Chapter 3 144
Frozen Lake Example 144
Doom Example 149
Chapter 4 156
Market Making Example 156
Chapter 5 168
Sonic Example 168
Index 174