Practical Machine Learning with Python (eBook)

Dipanjan Sarkar is a Data Scientist at Intel, on a mission to make the world more connected and productive. He primarily works on data science, analytics, business intelligence, application development, and building large-scale intelligent systems. He holds a master of technology degree in Information Technology with specializations in Data Science and Software Engineering from the International Institute of Information Technology, Bangalore. He is also an avid supporter of self-learning, especially Massive Open Online Courses and also holds a Data Science Specialization from Johns Hopkins University on Coursera. Dipanjan has been an analytics practitioner for several years, specializing in statistical, predictive, and text analytics. Having a passion for data science and education, he is a Data Science Mentor at Springboard, helping people up-skill on areas like Data Science and Machine Learning. Dipanjan has also authored several books on R, Python, Machine Learning and Analytics, including Text Analytics with Python, Apress 2016. Besides this, he occasionally reviews technical books and acts as a course beta tester for Coursera. Dipanjan's interests include learning about new technology, financial markets, disruptive start-ups, data science and more recently, artificial intelligence and deep learning.  Raghav Bali has a master's degree (gold medalist) in Information Technology from International Institute of Information Technology, Bangalore. He is a Data Scientist at Intel, where he works on analytics, business intelligence, and application development to develop scalable machine learning-based solutions. He has also worked as an analyst and developer in domains such as ERP, finance, and BI with some of the leading organizations in the world. Raghav is a technology enthusiast who loves reading and playing around with new gadgets and technologies. He has also authored several books on R, Machine Learning and Analytics. He is a shutterbug, capturing moments when he isn't busy solving problems. Tushar Sharma has a master’s degree from International Institute of Information Technology, Bangalore. He works as a Data Scientist with Intel. His work involves developing analytical solutions at scale using enormous volumes of infrastructure data. In his previous role, he has worked in the financial domain developing scalable machine learning solutions for major financial organizations. He is proficient in Python, R and Big Data frameworks like Spark and Hadoop. Apart from work Tushar enjoys watching movies, playing badminton and is an avid reader. He has also authored a book on R and social media analytics.

Contents 5
About the Authors 16
About the Technical Reviewer 18
Acknowledgments 19
Foreword 21
Introduction 22
Part I: Understanding Machine Learning 23
Chapter 1: Machine Learning Basics 24
The Need for Machine Learning 25
Making Data-Driven Decisions 25
Efficiency and Scale 26
Traditional Programming Paradigm 26
Why Machine Learning? 27
Understanding Machine Learning 29
Why Make Machines Learn? 29
Formal Definition 30
Defining the Task, T 31
Defining the Experience, E 33
Defining the Performance, P 33
A Multi-Disciplinary Field 34
Computer Science 35
Theoretical Computer Science 36
Practical Computer Science 36
Important Concepts 36
Algorithms 36
Programming Languages 37
Code 37
Data Structures 37
Data Science 37
Mathematics 39
Important Concepts 40
Scalar 40
Vector 40
Matrix 40
Tensor 42
Norm 42
Eigen Decomposition 42
Singular Value Decomposition 43
Random Variable 44
Probability Distribution 44
Probability Mass Function 44
Probability Density Function 44
Marginal Probability 44
Conditional Probability 44
Bayes Theorem 45
Statistics 45
Data Mining 46
Artificial Intelligence 46
Natural Language Processing 47
Deep Learning 49
Important Concepts 52
Artificial Neural Networks 52
Backpropagation 53
Multilayer Perceptrons 53
Convolutional Neural Networks 53
Recurrent Neural Networks 54
Long Short-Term Memory Networks 55
Autoencoders 55
Machine Learning Methods 55
Supervised Learning 56
Classification 57
Regression 58
Unsupervised Learning 59
Clustering 60
Dimensionality Reduction 61
Anomaly Detection 62
Association Rule-Mining 62
Semi-Supervised Learning 63
Reinforcement Learning 63
Batch Learning 64
Online Learning 65
Instance Based Learning 65
Model Based Learning 66
The CRISP-DM Process Model 66
Business Understanding 67
Define Business Problem 68
Assess and Analyze Scenarios 68
Define Data Mining Problem 69
Project Plan 69
Data Understanding 69
Data Collection 69
Data Description 70
Exploratory Data Analysis 70
Data Quality Analysis 70
Data Preparation 71
Data Integration 71
Data Wrangling 71
Attribute Generation and Selection 71
Modeling 72
Selecting Modeling Techniques 72
Model Building 72
Model Evaluation and Tuning 72
Model Assessment 72
Evaluation 73
Deployment 73
Building Machine Intelligence 73
Machine Learning Pipelines 73
Supervised Machine Learning Pipeline 75
Unsupervised Machine Learning Pipeline 76
Real-World Case Study: Predicting Student Grant Recommendations 76
Objective 77
Data Retrieval 77
Data Preparation 78
Feature Extraction and Engineering 78
Modeling 81
Model Evaluation 82
Model Deployment 82
Prediction in Action 83
Challenges in Machine Learning 85
Real-World Applications of Machine Learning 85
Summary 86
Chapter 2: The Python Machine Learning Ecosystem 87
Python: An Introduction 87
Strengths 88
Pitfalls 88
Setting Up a Python Environment 89
Set Up Anaconda Python Environment 89
Installing Libraries 91
Why Python for Data Science? 91
Powerful Set of Packages 91
Easy and Rapid Prototyping 91
Easy to Collaborate 92
One-Stop Solution 92
Large and Active Community Support 92
Introducing the Python Machine Learning Ecosystem 92
Jupyter Notebooks 92
Installation and Execution 93
NumPy 95
Numpy ndarray 95
Creating Arrays 96
Accessing Array Elements 97
Basic Indexing and Slicing 97
Advanced Indexing 99
Operations on Arrays 100
Linear Algebra Using numpy 102
Pandas 104
Data Structures of Pandas 104
Series 104
Dataframe 104
Data Retrieval 105
List of Dictionaries to Dataframe 105
CSV Files to Dataframe 105
Databases to Dataframe 107
Data Access 107
Head and Tail 107
Slicing and Dicing 108
Data Operations 111
Values Attribute 111
Missing Data and the fillna Function 111
Descriptive Statistics Functions 112
Concatenating Dataframes 114
Concatenating Using the concat Method 114
Database Style Concatenations Using the merge Command 115
Scikit-learn 116
Core APIs 117
Advanced APIs 118
Scikit-learn Example: Regression Models 119
The Dataset 119
Neural Networks and Deep Learning 122
Artificial Neural Networks 122
Deep Neural Networks 124
Number of Layers 124
Diverse Architectures 124
Computation Power 124
Python Libraries for Deep Learning 124
Theano 125
Installation 125
Theano Basics (Barebones Version) 125
Tensorflow 127
Installation 127
Keras 128
Installation 128
Keras Basics 128
Model Building 129
Learning an Example Neural Network 129
The Power of Deep Learning 131
Text Analytics and Natural Language Processing 132
The Natural Language Tool Kit 133
Installation and Introduction 133
Corpora 134
Tokenization 134
Tagging 134
Stemming and Lemmatization 134
Chunking 135
Sentiment 135
Classification/Clustering 135
Other Text Analytics Frameworks 135
Statsmodels 136
Installation 136
Modules 136
Distributions 136
Linear Regression 137
Generalized Linear Models 137
ANOVA 137
Time Series Analysis 137
Statistical Inference 137
Nonparametric Methods 137
Summary 138
Part II: The Machine Learning Pipeline 139
Chapter 3: Processing, Wrangling, and Visualizing Data 140
Data Collection 141
CSV 141
JSON 143
XML 147
HTML and Scraping 150
HTML 150
Web Scraping 151
SQL 155
Data Description 156
Numeric 156
Text 156
Categorical 156
Data Wrangling 157
Understanding Data 157
Filtering Data 160
Typecasting 163
Transformations 163
Imputing Missing Values 164
Handling Duplicates 166
Handling Categorical Data 166
Normalizing Values 167
String Manipulations 168
Data Summarization 168
Data Visualization 170
Visualizing with Pandas 171
Line Charts 171
Bar Plots 173
Histograms 174
Pie Charts 175
Box Plots 176
Scatter Plots 177
Visualizing with Matplotlib 180
Figures and Subplots 181
Plot Formatting 186
Legends 189
Axis Controls 191
Annotations 194
Global Parameters 195
Python Visualization Ecosystem 195
Summary 195
Chapter 4: Feature Engineering and Selection 196
Features: Understand Your Data Better 197
Data and Datasets 197
Features 198
Models 198
Revisiting the Machine Learning Pipeline 198
Feature Extraction and Engineering 200
What Is Feature Engineering? 200
Why Feature Engineering? 202
How Do You Engineer Features? 203
Feature Engineering on Numeric Data 204
Raw Measures 204
Values 205
Counts 206
Binarization 206
Rounding 207
Interactions 208
Binning 210
Fixed-Width Binning 211
Adaptive Binning 213
Statistical Transformations 216
Log Transform 216
Box-Cox Transform 217
Feature Engineering on Categorical Data 219
Transforming Nominal Features 220
Transforming Ordinal Features 221
Encoding Categorical Features 222
One Hot Encoding Scheme 222
Dummy Coding Scheme 225
Effect Coding Scheme 226
Bin-Counting Scheme 227
Feature Hashing Scheme 227
Feature Engineering on Text Data 228
Text Pre-Processing 229
Bag of Words Model 230
Bag of N-Grams Model 231
TF-IDF Model 232
Document Similarity 233
Topic Models 235
Word Embeddings 236
Feature Engineering on Temporal Data 239
Date-Based Features 240
Time-Based Features 241
Feature Engineering on Image Data 243
Image Metadata Features 244
Raw Image and Channel Pixels 244
Grayscale Image Pixels 246
Binning Image Intensity Distribution 246
Image Aggregation Statistics 247
Edge Detection 248
Object Detection 249
Localized Feature Extraction 250
Visual Bag of Words Model 252
Automated Feature Engineering with Deep Learning 255
Feature Scaling 258
Standardized Scaling 259
Min-Max Scaling 259
Robust Scaling 260
Feature Selection 261
Threshold-Based Methods 262
Statistical Methods 263
Recursive Feature Elimination 266
Model-Based Selection 267
Dimensionality Reduction 268
Feature Extraction with Principal Component Analysis 269
Summary 271
Chapter 5: Building, Tuning, and Deploying Models 273
Building Models 274
Model Types 275
Classification Models 275
Regression Models 276
Clustering Models 277
Learning a Model 278
Three Stages of Machine Learning 279
Representation 279
Evaluation 279
Optimization 279
The Three Stages of Logistic Regression 280
Representation 280
Evaluation 280
Optimization 281
Model Building Examples 281
Classification 282
Clustering 284
Partition Based Clustering 285
Hierarchical Clustering 287
Model Evaluation 289
Evaluating Classification Models 289
Confusion Matrix 290
Understanding the Confusion Matrix 290
Performance Metrics 292
Receiver Operating Characteristic Curve 294
Evaluating Clustering Models 296
External Validation 296
Internal Validation 297
Silhouette Coefficient 298
Calinski-Harabaz Index 298
Evaluating Regression Models 299
Coefficient of Determination or R2 299
Mean Squared Error 300
Model Tuning 300
Introduction to Hyperparameters 301
Decision Trees 301
The Bias-Variance Tradeoff 302
Extreme Cases of Bias-Variance 305
Underfitting 305
Overfitting 305
The Tradeoff 305
Cross Validation 306
Cross-Validation Strategies 308
Leave One Out CV 309
K-Fold CV 309
Hyperparameter Tuning Strategies 309
Grid Search 309
Randomized Search 312
Model Interpretation 313
Understanding Skater 315
Model Interpretation in Action 316
Model Deployment 320
Model Persistence 320
Custom Development 321
In-House Model Deployment 321
Model Deployment as a Service 322
Summary 322
Part III: Real-World Case Studies 323
Chapter 6: Analyzing Bike Sharing Trends 324
The Bike Sharing Dataset 324
Problem Statement 325
Exploratory Data Analysis 325
Preprocessing 325
Distribution and Trends 327
Outliers 329
Correlations 331
Regression Analysis 332
Types of Regression 332
Assumptions 333
Evaluation Criteria 333
Residual Analysis 333
Normality Test (Q-Q Plot) 333
R-Squared: Goodness of Fit 334
Cross Validation 334
Modeling 334
Linear Regression 336
Training 337
Testing 338
Decision Tree Based Regression 340
Node Splitting 341
Stopping Criteria 342
Hyperparameters 342
Decision Tree Algorithms 342
Training 343
Testing 346
Next Steps 347
Summary 347
Chapter 7: Analyzing Movie Reviews Sentiment 348
Problem Statement 349
Setting Up Dependencies 349
Getting the Data 350
Text Pre-Processing and Normalization 350
Unsupervised Lexicon-Based Models 353
Bing Liu’s Lexicon 354
MPQA Subjectivity Lexicon 354
Pattern Lexicon 355
AFINN Lexicon 355
SentiWordNet Lexicon 357
VADER Lexicon 359
Classifying Sentiment with Supervised Learning 362
Traditional Supervised Machine Learning Models 363
Newer Supervised Deep Learning Models 366
Advanced Supervised Deep Learning Models 372
Analyzing Sentiment Causation 380
Interpreting Predictive Models 380
Analyzing Topic Models 385
Summary 389
Chapter 8: Customer Segmentation and Effective Cross Selling 390
Online Retail Transactions Dataset 391
Exploratory Data Analysis 391
Customer Segmentation 395
Objectives 395
Customer Understanding 395
Target Marketing 395
Optimal Product Placement 395
Finding Latent Customer Segments 396
Higher Revenue 396
Strategies 396
Clustering 396
Exploratory Data Analysis 396
Clustering vs. Customer Segmentation 396
Clustering Strategy 397
RFM Model for Customer Value 397
Data Cleaning 397
Recency 398
Frequency and Monetary Value 399
Data Preprocessing 400
Clustering for Segments 403
K-Means Clustering 403
Cluster Analysis 404
Cluster Descriptions 406
Cross Selling 409
Market Basket Analysis with Association Rule-Mining 410
Association Rule-Mining Basics 411
FP Growth 412
Association Rule-Mining in Action 413
Exploratory Data Analysis 413
Mining Rules 417
Orange Table Data Structure 417
Using the FP Growth Algorithm 418
Summary 422
Chapter 9: Analyzing Wine Types and Quality 423
Problem Statement 423
Setting Up Dependencies 424
Getting the Data 424
Exploratory Data Analysis 425
Process and Merge Datasets 425
Understanding Dataset Features 426
Descriptive Statistics 429
Inferential Statistics 430
Univariate Analysis 432
Multivariate Analysis 435
Predictive Modeling 442
Predicting Wine Types 443
Predicting Wine Quality 449
Summary 462
Chapter 10: Analyzing Music Trends and Recommendations 463
The Million Song Dataset Taste Profile 464
Exploratory Data Analysis 464
Loading and Trimming Data 464
Enhancing the Data 467
Visual Analysis 468
Most Popular Songs 468
Most Popular Artist 470
User vs. Songs Distribution 471
Recommendation Engines 472
Types of Recommendation Engines 473
Utility of Recommendation Engines 473
Popularity-Based Recommendation Engine 474
Item Similarity Based Recommendation Engine 475
Matrix Factorization Based Recommendation Engine 477
Matrix Factorization and Singular Value Decomposition 479
Building a Matrix Factorization Based Recommendation Engine 479
A Note on Recommendation Engine Libraries 482
Summary 482
Chapter 11: Forecasting Stock and Commodity Prices 483
Time Series Data and Analysis 483
Time Series Components 485
Smoothing Techniques 487
Moving Average 487
Exponential Smoothing 488
Forecasting Gold Price 490
Problem Statement 490
Dataset 490
Traditional Approaches 490
Key Concepts 491
ARIMA 491
Modeling 492
Stock Price Prediction 499
Problem Statement 500
Dataset 500
Recurrent Neural Networks: LSTM 501
Regression Modeling 502
Sequence Modeling 507
Upcoming Techniques: Prophet 511
Summary 513
Chapter 12: Deep Learning for Computer Vision 514
Convolutional Neural Networks 514
Image Classification with CNNs 516
Problem Statement 516
Dataset 516
CNN Based Deep Learning Classifier from Scratch 517
CNN Based Deep Learning Classifier with Pretrained Models 520
Artistic Style Transfer with CNNs 524
Background 525
Preprocessing 526
Loss Functions 528
Content Loss 528
Style Loss 528
Total Variation Loss 529
Overall Loss Function 529
Custom Optimizer 530
Style Transfer in Action 531
Summary 535
Index 536