Pro Deep Learning with TensorFlow (eBook)

Santanu Pattanayak currently works at GE, Digital as a Senior Data Scientist. He has 10 years of overall work experience with six of years of experience in the data analytics/data science field and also has a background in development and database technologies. Prior to joining GE, Santanu worked in companies such as RBS, Capgemini, and IBM. He graduated with a degree in electrical engineering from Jadavpur University, Kolkata and is an avid math enthusiast. Santanu is currently pursuing a master's degree in data science from Indian Institute of Technology (IIT), Hyderabad. He also devotes his time to data science hackathons and Kaggle competitions where he ranks within the top 500 across the globe. Santanu was born and brought up in West Bengal, India and currently resides in Bangalore, India with his wife.

Contents 5
About the Author 12
About the Technical Reviewer 13
Acknowledgments 14
Introduction 15
Chapter 1: Mathematical Foundations 18
Linear Algebra 19
Vector 20
Scalar 21
Matrix 21
Tensor 22
Matrix Operations and Manipulations 22
Addition of Two Matrices 23
Subtraction of Two Matrices 23
Product of Two Matrices 23
Transpose of a Matrix 24
Dot Product of Two Vectors 24
Matrix Working on a Vector 25
Linear Independence of Vectors 26
Rank of a Matrix 27
Identity Matrix or Operator 28
Determinant of a Matrix 29
Interpretation of Determinant 30
Inverse of a Matrix 31
Norm of a Vector 32
Pseudo Inverse of a Matrix 33
Unit Vector in the Direction of a Specific Vector 34
Projection of a Vector in the Direction of Another Vector 34
Eigen Vectors 35
Characteristic Equation of a Matrix 36
Power Iteration Method for Computing Eigen Vector 39
Calculus 40
Differentiation 40
Gradient of a Function 41
Successive Partial Derivatives 42
Hessian Matrix of a Function 42
Maxima and Minima of Functions 43
Rules for Maxima and Minima for a Univariate Function 43
Local Minima and Global Minima 45
Positive Semi-Definite and Positive Definite 46
Convex Set 46
Convex Function 47
Non-convex Function 48
Multivariate Convex and Non-convex Functions Examples 48
Taylor Series 51
Probability 51
Unions, Intersection, and Conditional Probability 52
Chain Rule of Probability for Intersection of Event 54
Mutually Exclusive Events 54
Independence of Events 54
Conditional Independence of Events 55
Bayes Rule 55
Probability Mass Function 55
Probability Density Function 56
Expectation of a Random Variable 56
Variance of a Random Variable 56
Skewness and Kurtosis 57
Covariance 61
Correlation Coefficient 61
Some Common Probability Distribution 62
Uniform Distribution 62
Normal Distribution 63
Multivariate Normal Distribution 65
Bernoulli Distribution 65
Binomial Distribution 66
Poisson Distribution 67
Likelihood Function 68
Maximum Likelihood Estimate 69
Hypothesis Testing and p Value 70
Formulation of Machine-Learning Algorithm and Optimization Techniques 72
Supervised Learning 73
Linear Regression as a Supervised Learning Method 73
Linear Regression Through Vector Space Approach 76
Classification 78
Hyperplanes and Linear Classifiers 81
Unsupervised Learning 82
Optimization Techniques for Machine Learning 83
Gradient Descent 83
Gradient Descent for a Multivariate Cost Function 84
Contour Plot and Contour Lines 85
Steepest Descent 87
Stochastic Gradient Descent 88
Newton’s Method 91
Linear Curve 91
Negative Curvature 92
Positive Curvature 93
Constrained Optimization Problem 94
A Few Important Topics in Machine Learning 96
Dimensionality Reduction Methods 96
Principal Component Analysis 97
When Will PCA Be Useful in Data Reduction? 99
How Do You Know How Much Variance Is Retained by the Selected Principal Components? 100
Singular Value Decomposition 100
Regularization 101
Regularization Viewed as a Constraint Optimization Problem 103
Summary 104
Chapter 2: Introduction to Deep-Learning Concepts and TensorFlow 105
Deep Learning and Its Evolution 105
Perceptrons and Perceptron Learning Algorithm 108
Geometrical Interpretation of Perceptron Learning 112
Limitations of Perceptron Learning 113
Need for Non-linearity 115
Hidden Layer Perceptrons’ Activation Function for Non-linearity 116
Different Activation Functions for a Neuron/Perceptron 118
Linear Activation Function 118
Binary Threshold Activation Function 118
Sigmoid Activation Function 119
SoftMax Activation Function 120
Rectified Linear Unit(ReLU) Activation Function 122
Tanh Activation Function 123
Learning Rule for Multi-Layer Perceptrons Network 124
Backpropagation for Gradient Computation 125
Generalizing the Backpropagation Method for Gradient Computation 127
Deep Learning Versus Traditional Methods 132
TensorFlow 134
Common Deep-Learning Packages 134
TensorFlow Installation 135
TensorFlow Basics for Development 135
Gradient-Descent Optimization Methods from a Deep-Learning Perspective 139
Elliptical Contours 139
Non-convexity of Cost Functions 142
Saddle Points in the High-Dimensional Cost Functions 143
Learning Rate in Mini-batch Approach to Stochastic Gradient Descent 145
Optimizers in TensorFlow 146
GradientDescentOptimizer 146
Usage 146
AdagradOptimizer 146
Usage 147
RMSprop 147
Usage 149
AdadeltaOptimizer 149
Usage 150
AdamOptimizer 151
Usage 151
MomentumOptimizer and Nesterov Algorithm 152
Usage 153
Epoch, Number of Batches, and Batch Size 154
XOR Implementation Using TensorFlow 154
TensorFlow Computation Graph for XOR network 156
Linear Regression in TensorFlow 159
Multi-class Classification with SoftMax Function Using Full-Batch Gradient Descent 162
Multi-class Classification with SoftMax Function Using Stochastic Gradient Descent 165
GPU 168
Summary 168
Chapter 3: Convolutional Neural Networks 169
Convolution Operation 169
Linear Time Invariant (LTI) / Linear Shift Invariant (LSI) Systems 169
Convolution for Signals in One Dimension 171
Analog and Digital Signals 174
2D and 3D signals 176
2D Convolution 177
Two-dimensional Unit Step Function 177
2D Convolution of a Signal with an LSI System Unit Step Response 179
2D Convolution of an Image to Different LSI System Responses 181
Common Image-Processing Filters 185
Mean Filter 185
Median Filter 187
Gaussian Filter 189
Gradient-based Filters 190
Sobel Edge-Detection Filter 191
Identity Transform 193
Convolution Neural Networks 194
Components of Convolution Neural Networks 195
Input Layer 196
Convolution Layer 196
TensorFlow Usage 197
Pooling Layer 198
TensorFlow Usage 198
Backpropagation Through the Convolutional Layer 198
Backpropagation Through the Pooling Layers 202
Weight Sharing Through Convolution and Its Advantages 203
Translation Equivariance 204
Translation Invariance Due to Pooling 205
Dropout Layers and Regularization 206
Convolutional Neural Network for Digit Recognition on the MNIST Dataset 208
Convolutional Neural Network for Solving Real-World Problems 212
Batch Normalization 220
Different Architectures in Convolutional Neural Networks 222
LeNet 222
AlexNet 224
VGG16 225
ResNet 226
Transfer Learning 227
Guidelines for Using Transfer Learning 228
Transfer Learning with Google’s InceptionV3 229
Transfer Learning with Pre-trained VGG16 232
Summary 237
Chapter 4: Natural Language Processing Using Recurrent Neural Networks 238
Vector Space Model (VSM) 238
Vector Representation of Words 242
Word2Vec 243
Continuous Bag of Words (CBOW) 243
Continuous Bag of Words Implementation in TensorFlow 246
Skip-Gram Model for Word Embedding 250
Skip-gram Implementation in TensorFlow 252
Global Co-occurrence Statistics–based Word Vectors 255
GloVe 260
Word Analogy with Word Vectors 264
Introduction to Recurrent Neural Networks 267
Language Modeling 269
Predicting the Next Word in a Sentence Through RNN Versus Traditional Methods 270
Backpropagation Through Time (BPTT) 271
Vanishing and Exploding Gradient Problem in RNN 274
Solution to Vanishing and Exploding Gradients Problem in RNNs 275
Gradient Clipping 276
Smart Initialization of the Memory-to-Memory Weight Connection Matrix and ReLU units 276
Long Short-Term Memory (LSTM) 277
LSTM in Reducing Exploding- and Vanishing -Gradient Problems 278
MNIST Digit Identification in TensorFlow Using Recurrent Neural Networks 280
Next-Word Prediction and Sentence Completion in TensorFlow Using Recurrent Neural Networks 283
Gated Recurrent Unit (GRU) 289
Bidirectional RNN 291
Summary 293
Chapter 5: Unsupervised Learning with Restricted Boltzmann Machines and Auto-encoders 294
Boltzmann Distribution 294
Bayesian Inference: Likelihood, Priors, and Posterior Probability Distribution 296
Markov Chain Monte Carlo Methods for Sampling 301
Metropolis Algorithm 304
Restricted Boltzmann Machines 309
Training a Restricted Boltzmann Machine 314
Gibbs Sampling 319
Block Gibbs Sampling 320
Burn-in Period and Generating Samples in Gibbs Sampling 321
Using Gibbs Sampling in Restricted Boltzmann Machines 321
Contrastive Divergence 323
A Restricted Boltzmann Implementation in TensorFlow 324
Collaborative Filtering Using Restricted Boltzmann Machines 328
Deep Belief Networks (DBNs) 332
Auto-encoders 337
Feature Learning Through Auto-encoders for Supervised Learning 340
Kullback-Leibler (KL) Divergence 342
Sparse Auto-encoders 343
Sparse Auto-Encoder Implementation in TensorFlow 344
Denoising Auto-Encoder 348
A Denoising Auto-Encoder Implementation in TensorFlow 348
PCA and ZCA Whitening 355
Summary 358
Chapter 6: Advanced Neural Networks 359
Image Segmentation 359
Binary Thresholding Method Based on Histogram of Pixel Intensities 359
Otsu’s Method 360
Watershed Algorithm for Image Segmentation 363
Image Segmentation Using K-means Clustering 366
Semantic Segmentation 369
Sliding-Window Approach 369
Fully Convolutional Network (FCN) 370
Fully Convolutional Network with Downsampling and Upsampling 372
Unpooling 373
Max Unpooling 374
Transpose Convolution 375
U-Net 378
Semantic Segmentation in TensorFlow with Fully Connected Neural Networks 379
Image Classification and Localization Network 387
Object Detection 389
R-CNN 390
Fast and Faster R-CNN 391
Generative Adversarial Networks 392
Maximin and Minimax Problem 393
Zero-sum Game 395
Minimax and Saddle Points 396
GAN Cost Function and Training 397
Vanishing Gradient for the Generator 400
TensorFlow Implementation of a GAN Network 400
TensorFlow Models’ Deployment in Production 403
Summary 406
Index 407