Predictive Analytics, Data Mining and Big Data (eBook)

Steven Finlay is one of the UK's leading experts on predictive analytics and its application within Big Data environments. He has extensive experience of developing predictive analytics solutions within Financial Services, Retailing and Government organisations. Steven is currently Head of Analytics at HML, the UK's largest provider of mortgage administration services. Previously he has worked as a data scientist, consultant and project manager for a variety of organizations in both the public and private sectors. Steven has a PhD in predictive analytics and is an Honorary Research Fellow in the Management Science Department at Lancaster University in the UK.

Cover 1
Half-Title 2
Title 4
Copyright 5
Dedication 6
Contents 8
Figures and Tables 11
Acknowledgments 13
1 Introduction 14
1.1 What are data mining and predictive analytics? 15
1.2 How good are models at predicting behavior? 19
1.3 What are the benefits of predictive models? 20
1.4 Applications of predictive analytics 22
1.5 Reaping the benefits, avoiding the pitfalls 24
1.6 What is Big Data? 26
1.7 How much value does Big Data add? 29
1.8 The rest of the book 32
2 Using Predictive Models 34
2.1 What are your objectives? 35
2.2 Decision making 36
2.3 The next challenge 44
2.4 Discussion 47
2.5 Override rules (business rules) 49
3 Analytics, Organization and Culture 52
3.1 Embedded analytics 53
3.2 Learning from failure 55
3.3 A lack of motivation 56
3.4 A slight misunderstanding 58
3.5 Predictive, but not precise 63
3.6 Great expectations 65
3.7 Understanding cultural resistance to predictive analytics 67
3.8 The impact of predictive analytics 73
3.9 Combining model-based predictions and human judgment 75
4 The Value of Data 78
4.1 What type of data is predictive of behavior? 79
4.2 Added value is what's important 83
4.3 Where does the data to build predictive models come from? 86
4.4 The right data at the right time 89
4.5 How much data do I need to build a predictive model? 92
5 Ethics and Legislation 98
5.1 A brief introduction to ethics 99
5.2 Ethics in practice 102
5.3 The relevance of ethics in a Big Data world 103
5.4 Privacy and data ownership 105
5.5 Data security 109
5.6 Anonymity 110
5.7 Decision making 112
6 Types of Predictive Models 117
6.1 Linear models 119
6.2 Decision trees (classification and regression trees) 125
6.3 (Artificial) neural networks 127
6.4 Support vector machines (SVMs) 131
6.5 Clustering 133
6.6 Expert systems (knowledge-based systems) 135
6.7 What type of model is best? 137
6.8 Ensemble (fusion or combination) systems 141
6.9 How much benefit can I expect to get from using an ensemble? 143
6.10 The prospects for better types of predictive models in the future 144
7 The Predictive Analytics Process 147
7.1 Project initiation 148
7.2 Project requirements 151
7.3 Is predictive analytics the right tool for the job? 155
7.4 Model building and business evaluation 156
7.5 Implementation 158
7.6 Monitoring and redevelopment 162
7.7 How long should a predictive analytics project take? 167
8 How to Build a Predictive Model 170
8.1 Exploring the data landscape 171
8.2 Sampling and shaping the development sample 172
8.3 Data preparation (data cleaning) 175
8.4 Creating derived data 176
8.5 Understanding the data 177
8.6 Preliminary variable selection (data reduction) 178
8.7 Pre-processing (data transformation) 179
8.8 Model construction (modeling) 183
8.9 Validation 184
8.10 Selling models into the business 185
8.11 The rise of the regulator 189
9 Text Mining and Social Network Analysis 192
9.1 Text mining 192
9.2 Using text analytics to create predictor variables 194
9.3 Within document predictors 194
9.4 Sentiment analysis 197
9.5 Across document predictors 198
9.6 Social network analysis 199
9.7 Mapping a social network 204
10 Hardware, Software and All that Jazz 207
10.1 Relational databases 210
10.2 Hadoop 213
10.3 The limitations of Hadoop 215
10.4 Do I need a Big Data solution to do predictive analytics? 216
10.5 Software for predictive analytics 219
Appendix A. Glossary of Terms 222
Appendix B. Further Sources of Information 231
Appendix C. Lift Charts and Gain Charts 236
Notes 240
Index 259