Deep Learning and its Applications using Python

Niha Kamal Basha is an assistant professor in the Department of Information Security, School of Computer Science and Engineering, Vellore Institute of Science and Technology, India. She has received a number of awards and published numerous research articles in peer-reviewed journals. Surbi Bhatia, PhD, is an assistant professor in the Department of Information Systems, College of Computer Sciences and Information Technology, King Faisal University, Saudi Arabia. She has more than 10 years of teaching experience in different universities in India and Saudi Arabia. She has published many articles in peer-reviewed journals, authored or edited 9 books, and has been granted 8 national and international patents. Abhishek Kumar gained his PhD in computer science from the University of Madras, India in 2019. He is assistant director /associate professor in the Computer Science & Engineering Department, Chandigarh University, Punjab, India. He has more than 100 publications in peer-reviewed international and national journals, books & conferences His research interests include artificial intelligence, image processing, computer vision, data mining and machine learning. Arwa Mashat, gained her PhD in Instructional Design and Technology from Old Dominion University, Virginia, USA in 2017. She has a rich 14 years of teaching and academic experience. She is currently an assistant professor in the College of Computing and Information Technology, King Abdulaziz University, Saudi Arabia. She is currently the Vice Dean for two colleges; the College of Computing and Information Technology and the Applied College at King Abdulaziz University. She has published many research papers in reputed journals.

Preface ix

1 Introduction to Deep Learning 1

1.1 History of Deep Learning 1

1.2 A Probabilistic Theory of Deep Learning 4

1.3 Back Propagation and Regularization 14

1.4 Batch Normalization and VC Dimension 17

1.5 Neural Nets--Deep and Shallow Networks 18

1.6 Supervised and Semi-Supervised Learning 19

1.7 Deep Learning and Reinforcement Learning 21

2 Basics of TensorFlow 25

2.1 Tensors 25

2.2 Computational Graph and Session 27

2.3 Constants, Placeholders, and Variables 28

2.4 Creating Tensor 32

2.5 Working on Matrices 35

2.6 Activation Functions 36

2.7 Loss Functions 39

2.8 Common Loss Function 39

2.9 Optimizers 40

2.10 Metrics 41

3 Understanding and Working with Keras 45

3.1 Major Steps to Deep Learning Models 45

3.2 Load Data 47

3.3 Pre-Process Data 48

3.4 Define the Model 48

3.5 Compile the Model 49

3.6 Fit and Evaluate the Mode 51

3.7 Prediction 52

3.8 Save and Reload the Model 52

3.9 Additional Steps to Improve Keras Models 53

3.10 Keras with TensorFlow 55

4 Multilayer Perceptron 57

4.1 Artificial Neural Network 57

4.2 Single-Layer Perceptron 60

4.3 Multilayer Perceptron 61

4.4 Logistic Regression Model 61

4.5 Regression to MLP in TensorFlow 63

4.6 TensorFlow Steps to Build Models 63

4.7 Linear Regression in TensorFlow 63

4.8 Logistic Regression Mode in TensorFlow 67

4.9 Multilayer Perceptron in TensorFlow 69

4.10 Regression to MLP in Keras 72

4.11 Log-Linear Model 72

4.12 Keras Neural Network for Linear Regression 73

4.13 Keras Neural Network for Logistic Regression 73

4.14 MLPs on the Iris Data 75

4.15 MLPs on MNIST Data (Digit Classification) 76

4.16 MLPs on Randomly Generated Data 78

5 Convolutional Neural Networks in Tensorflow 81

5.1 CNN Architectures 81

5.2 Properties of CNN Representations 82

5.3 Convolution Layers, Pooling Layers - Strides - Padding and Fully Connected Layer 82

5.4 Why TensorFlow for CNN Models? 84

5.5 TensorFlow Code for Building an Image Classifier for MNIST Data 84

5.6 Using a High-Level API for Building CNN Models 88

5.7 CNN in Keras 88

5.8 Building an Image Classifier for MNIST Data in Keras 88

5.9 Building an Image Classifier with CIFAR-10 Data 89

5.10 Define the Model Architecture 90

5.11 Pre-Trained Models 91

6 RNN and LSTM 95

6.1 Concept of RNN 95

6.2 Concept of LSTM 96

6.3 Modes of LSTM 97

6.4 Sequence Prediction 98

6.5 Time-Series Forecasting with the LSTM Model 99

6.6 Speech to Text 100

6.7 Examples Using Each API 102

6.8 Text-to-Speech Conversion 105

6.9 Cognitive Service Providers 106

6.10 The Future of Speech Analytics 107

7 Developing Chatbot's Face Detection and Recognition 109

7.1 Why Chatbots? 109

7.2 Designs and Functions of Chatbot's 109

7.3 Steps for Building a Chatbot's 110

7.4 Best Practices of Chatbot Development 116

7.5 Face Detection 116

7.6 Face Recognition 117

7.7 Face Analysis 117

7.8 OpenCV--Detecting a Face, Recognition and Face Analysis 117

7.8.1 Face Detection 117

7.8.2 Face Recognition 120

7.9 Deep Learning-Based Face Recognition 124

7.10 Transfer Learning 127

7.11 API's 131

8 Advanced Deep Learning 133

8.1 Deep Convolutional Neural Networks (AlexNet) 133

8.2 Networks Using Blocks (VGG) 137

8.3 Network in Network (NiN) 140

8.4 Networks with Parallel Concatenations (GoogLeNet) 144

8.5 Residual Networks (ResNet) 148

8.6 Densely Connected Networks (DenseNet) 151

8.7 Gated Recurrent Units (GRU) 154

8.8 Long Short-Term Memory (LSTM) 156

8.9 Deep Recurrent Neural Networks (D-RNN) 158

8.10 Bidirectional Recurrent Neural Networks (Bi-RNN) 159

8.11 Machine Translation and the Dataset 160

8.12 Sequence to Sequence Learning 161

9 Enhanced Convolutional Neural Network 167

9.1 Introduction 167

9.2 Deep Learning-Based Architecture for Absence Seizure Detection 178

9.3 EEG Signal Pre-Processing Strategy and Channel Selection 180

9.4 Input Formulation and Augmentation of EEG Signal for Deep Learning Model 188

9.5 Deep Learning Based Feature Extraction and Classification 196

9.6 Performance Analysis 200

9.7 Summary 201

10 Conclusion 205

10.1 Introduction 205

10.2 Future Research Direction and Prospects 205

10.3 Research Challenges in Deep Learning 210

10.4 Practical Deep Learning Case Studies 210

10.4.1 Medicine: Epilepsy Seizure Onset Prediction 219

10.4.2 Using Data from Test Drills to Predict where to Drill for Oil 232

10.5 Summary 235

References 235

Index 239