Bayesian Optimization and Data Science (eBook)

Preface 5
Who should read this book? 8
Contents 11
1 Automated Machine Learning and Bayesian Optimization 14
1.1 Automated Machine Learning 14
1.1.1 Motivation 14
1.1.2 Model Selection 16
1.1.3 Hyperparameter Optimization 17
1.1.4 Combined Algorithm Selection and Hyperparameter Optimization 19
1.1.5 Why Hyperparameter Optimization Is Important? 19
1.2 The Basic Structure of Bayesian Optimization 21
1.2.1 Sequential Model-Based Optimization 21
1.2.2 Surrogate Model 23
1.2.3 Acquisition Function 25
1.3 Automated Machine Learning for Predictive Analytics 27
References 30
2 From Global Optimization to Optimal Learning 32
2.1 A Priori Analysis of Global Optimization Strategies 33
2.2 Lipschitz Global Optimization (LGO) 34
2.3 Random Search 37
2.3.1 General Properties of Uniform Sampling 39
2.3.2 Cluster Analysis 39
2.3.3 Stopping Rules 39
2.4 Bandits, Active Learning and Bayesian Optimization 41
References 45
3 The Surrogate Model 49
3.1 Gaussian Processes 49
3.1.1 Gaussian Processes Regression 49
3.1.2 Kernel: The Data Geometry of Bayesian Optimization 52
3.1.3 Embedding Derivative Observations in the Gaussian Process 55
3.1.4 Numerical Instability 58
3.2 Thompson Sampling 59
3.3 Alternative Models 62
3.3.1 Random Forest 63
3.3.2 Neural Networks: Feedforward, Deep and Bayesian 65
References 67
4 The Acquisition Function 69
4.1 Traditional Acquisition Functions 69
4.1.1 Probability of Improvement 69
4.1.2 Expected Improvement 70
4.1.3 Upper/Lower Confidence Bound 72
4.2 New Acquisition Functions 74
4.2.1 Scaled Expected Improvement 74
4.2.2 Portfolio Allocation 74
4.2.3 Thompson Sampling 75
4.2.4 Entropy-Based Acquisition Functions 77
4.2.5 Knowledge Gradient 78
4.2.6 Look-Ahead 80
4.2.7 K-Optimality 81
4.3 Optimizing the Acquisition Function 82
References 83
5 Exotic Bayesian Optimization 85
5.1 Constrained Global Optimization 85
5.2 Support Vector Machine—Constrained Bayesian Optimization 88
5.3 Safe Bayesian Optimization 98
5.4 Parallel Bayesian Optimization 102
5.5 Multi-objective Bayesian Optimization 103
5.6 Multi-source and Multi-fidelity Bayesian Optimization 105
References 106
6 Software Resources 109
6.1 Open Source Software 109
6.2 Bayesian Optimization as a Service 113
6.3 Bayesian Optimization-Based Services for Hyperparameters Optimization 114
6.4 Test Functions and Generators 115
6.4.1 Survey and Site/Repository of Test Functions 115
6.4.2 Test Functions Generators 119
6.5 Non-Bayesian Global Optimization Software 119
References 120
7 Selected Applications 122
7.1 Overview of Applications 122
7.2 Smart Water 127
7.2.1 Leakage Localization 127
7.2.2 Pump Scheduling Optimization 129
References 133