Deep Learning in Natural Language Processing (eBook)

Li Deng is the Chief Artificial Intelligence Officer of Citadel since May 2017. Prior to Citadel, he was the Chief Scientist of AI, the founder of Deep Learning Technology Center, and Partner Research Manager at Microsoft. Prior to Microsoft, he was a tenured full professor at the University of Waterloo in Ontario, Canada as well as teaching and conducting research at MIT (Cambridge), ATR (Kyoto, Japan) and HKUST (Hong Kong). He is a Fellow of the IEEE, a Fellow of the Acoustical Society of America, and a Fellow of the ISCA. He has also been an Affiliate Professor at University of Washington since 2000. He was an elected member of Board of Governors of the IEEE Signal Processing Society, and was Editors-in-Chief of IEEE Signal Processing Magazine and of IEEE/ACM Transactions on Audio, Speech, and Language Processing (2008-2014), for which he received the IEEE SPS Meritorious Service Award. In recognition of the pioneering work on disrupting speech recognition industry using large-scale deep learning, he received the 2015 IEEE SPS Technical Achievement Award for “Outstanding Contributions to Deep Learning and to Automatic Speech Recognition." He also received numerous best paper and patent awards for the contributions to artificial intelligence, machine learning, natural language processing, information retrieval, multimedia signal processing, and speech processing. He is an author or co-author of six technical books. Yang Liu is an associate professor at the Department of Computer Science and Technology, Tsinghua University. He received his PhD degree from the Chinese Academy of Sciences Institute of Computing Technology in 2007. His research focuses on natural language processing and machine translation. He has published over 50 papers in leading NLP/AI journals and conferences such as Computational Linguistics, ACL, AAAI, EMNLP, and COLING. He won the COLING/ACL 2006 Meritorious Asian NLP Paper Award and the National Science and Technology Progress Award second prize. He served as Associate Editor of ACM TALLIP, ACL 2014 tutorial co-chair, ACL 2015 local arrangement co-chair, IJCAI 2016 senior PC, ACL 2017 area co-chair, EMNLP 2016 area co-chair, SIGHAN information officer, and the general secretary of the Computational Linguistics Technical Committee of Chinese Information Processing Society. 

Foreword 5
Preface 7
Contents 10
Contributors 11
Acronyms 12
1 A Joint Introduction to Natural Language Processing and to Deep Learning 15
1.1 Natural Language Processing: The Basics 15
1.2 The First Wave: Rationalism 16
1.3 The Second Wave: Empiricism 18
1.4 The Third Wave: Deep Learning 20
1.5 Transitions from Now to the Future 24
1.5.1 From Empiricism to Deep Learning: A Revolution 25
1.5.2 Limitations of Current Deep Learning Technology 25
1.6 Future Directions of NLP 26
1.6.1 Neural-Symbolic Integration 26
1.6.2 Structure, Memory, and Knowledge 28
1.6.3 Unsupervised and Generative Deep Learning 28
1.6.4 Multimodal and Multitask Deep Learning 29
1.6.5 Meta-learning 30
1.7 Summary 31
References 32
2 Deep Learning in Conversational Language Understanding 37
2.1 Introduction 37
2.2 A Historical Perspective 39
2.3 Major Language Understanding Tasks 40
2.3.1 Domain Detection and Intent Determination 41
2.3.2 Slot Filling 41
2.4 Elevating State of the Art: From Statistical Modeling to Deep Learning 42
2.4.1 Domain Detection and Intent Determination 42
2.4.2 Slot Filling 45
2.4.3 Joint Multitask Multi-domain Modeling 51
2.4.4 Understanding in Context 54
2.5 Summary 57
References 59
3 Deep Learning in Spoken and Text-Based Dialog Systems 63
3.1 Introduction 63
3.2 Learning Methodology for Components of a Dialog System 66
3.2.1 Discriminative Methods 66
3.2.2 Generative Methods 68
3.2.3 Decision-Making 68
3.3 Goal-Oriented Neural Dialog Systems 69
3.3.1 Neural Language Understanding 69
3.3.2 Dialog State Tracker 70
3.3.3 Deep Dialog Manager 70
3.4 Model-Based User Simulators 73
3.5 Natural Language Generation 74
3.6 End-to-End Deep Learning Approaches to Building Dialog Systems 77
3.7 Deep Learning for Open Dialog Systems 78
3.8 Datasets for Dialog Modeling 79
3.8.1 The Carnegie Mellon Communicator Corpus 79
3.8.2 ATIS—Air Travel Information System Pilot Corpus 79
3.8.3 Dialog State Tracking Challenge Dataset 80
3.8.4 Maluuba Frames Dataset 80
3.8.5 Facebook's Dialog Datasets 81
3.8.6 Ubuntu Dialog Corpus 81
3.9 Open Source Dialog Software 81
3.10 Dialog System Evaluation 84
3.11 Summary 86
References 87
4 Deep Learning in Lexical Analysis and Parsing 93
4.1 Background 93
4.2 Typical Lexical Analysis and Parsing Tasks 94
4.2.1 Word Segmentation 94
4.2.2 POS Tagging 95
4.2.3 Syntactic Parsing 95
4.2.4 Structured Predication 98
4.3 Structured Prediction Methods 100
4.3.1 Graph-Based Methods 100
4.3.2 Transition-Based Methods 102
4.4 Neural Graph-Based Methods 107
4.4.1 Neural Conditional Random Fields 107
4.4.2 Neural Graph-Based Dependency Parsing 108
4.5 Neural Transition-Based Methods 111
4.5.1 Greedy Shift-Reduce Dependency Parsing 111
4.5.2 Greedy Sequence Labeling 114
4.5.3 Globally Optimized Models 117
4.6 Summary 124
References 125
5 Deep Learning in Knowledge Graph 131
5.1 Introduction 131
5.1.1 Basic Concepts 132
5.1.2 Typical Knowledge Graphs 132
5.2 Knowledge Representation Learning 137
5.3 Neural Relation Extraction 138
5.3.1 Sentence-Level NRE 139
5.3.2 Document-Level NRE 144
5.4 Bridging Knowledge with Text: Entity Linking 146
5.4.1 The Entity Linking Framework 147
5.4.2 Deep Learning for Entity Linking 149
5.5 Summary 156
References 157
6 Deep Learning in Machine Translation 160
6.1 Introduction 160
6.2 Statistical Machine Translation and Its Challenges 161
6.2.1 Basics 161
6.2.2 Challenges in Statistical Machine Translation 164
6.3 Component-Wise Deep Learning for Machine Translation 165
6.3.1 Deep Learning for Word Alignment 165
6.3.2 Deep Learning for Translation Rule Probability Estimation 168
6.3.3 Deep Learning for Reordering Phrases 172
6.3.4 Deep Learning for Language Modeling 174
6.3.5 Deep Learning for Feature Combination 175
6.4 End-to-End Deep Learning for Machine Translation 177
6.4.1 The Encoder–Decoder Framework 177
6.4.2 Neural Attention in Machine Translation 180
6.4.3 Addressing Technical Challenges of Large Vocabulary 181
6.4.4 End-to-End Training to Optimize Evaluation Metric Directly 183
6.4.5 Incorporating Prior Knowledge 185
6.4.6 Low-Resource Language Translation 187
6.4.7 Network Structures in Neural Machine Translation 190
6.4.8 Combination of SMT and NMT 191
6.5 Summary 192
References 193
7 Deep Learning in Question Answering 197
7.1 Introduction 197
7.2 Deep Learning in Question Answering over Knowledge Base 198
7.2.1 The Information Extraction Style 199
7.2.2 The Semantic Parsing Style 203
7.2.3 The Information Extraction Style Versus the Semantic Parsing Style 207
7.2.4 Datasets 208
7.2.5 Challenges 209
7.3 Deep Learning in Machine Comprehension 210
7.3.1 Task Description 210
7.3.2 Feature Engineering-Based Methods in Machine Comprehension 214
7.3.3 Deep Learning Methods in Machine Comprehension 219
7.4 Summary 224
References 227
8 Deep Learning in Sentiment Analysis 230
8.1 Introduction 230
8.2 Sentiment-Specific Word Embedding 232
8.3 Sentence-Level Sentiment Classification 236
8.3.1 Convolutional Neural Networks 237
8.3.2 Recurrent Neural Networks 240
8.3.3 Recursive Neural Networks 242
8.3.4 Integration of External Resources 245
8.4 Document-Level Sentiment Classification 246
8.5 Fine-Grained Sentiment Analysis 249
8.5.1 Opinion Mining 249
8.5.2 Targeted Sentiment Analysis 251
8.5.3 Aspect-Level Sentiment Analysis 254
8.5.4 Stance Detection 255
8.5.5 Sarcasm Recognition 258
8.6 Summary 259
References 259
9 Deep Learning in Social Computing 265
9.1 Introduction to Social Computing 265
9.2 Modeling User-Generated Content with Deep Learning 268
9.2.1 Traditional Semantic Representation Approaches 269
9.2.2 Semantic Representation with Shallow Embedding 270
9.2.3 Semantic Representation with Deep Neural Networks 272
9.2.4 Enhancing Semantic Representation with Attention Mechanism 276
9.3 Modeling Social Connections with Deep Learning 278
9.3.1 Social Connections on Social Media 278
9.3.2 A Network Representation Learning Approach to Modeling Social Connections 278
9.3.3 Shallow Embedding Based Models 280
9.3.4 Deep Neural Network Based Models 283
9.3.5 Applications of Network Embedding 285
9.4 Recommendation with Deep Learning 285
9.4.1 Recommendation on Social Media 285
9.4.2 Traditional Recommendation Algorithms 286
9.4.3 Shallow Embedding Based Models 287
9.4.4 Deep Neural Network Based Models 289
9.5 Summary 294
References 294
10 Deep Learning in Natural Language Generation from Images 299
10.1 Introduction 299
10.2 Background 300
10.3 Deep Learning Frameworks to Generate Natural Language from an Image 301
10.3.1 The End-to-End Framework 301
10.3.2 The compositional framework 304
10.3.3 Other Frameworks 306
10.4 Evaluation Metrics and Benchmarks 306
10.5 Industrial Deployment of Image Captioning 307
10.6 Examples: Natural Language Descriptions of Images 308
10.7 Recent Research on Generating Stylistic Natural Language from Images 308
10.8 Summary 314
References 314
11 Epilogue: Frontiers of NLP in the Deep Learning Era 318
11.1 Introduction 318
11.2 Two New Perspectives 319
11.2.1 The Task-Centric Perspective 320
11.2.2 The Representation-Centric Perspective 321
11.3 Major Recent Advances in Deep Learning for NLP and Research Frontiers 323
11.3.1 Compositionality for Generalization 323
11.3.2 Unsupervised Learning for NLP 324
11.3.3 Reinforcement Learning for NLP 325
11.3.4 Meta-Learning for NLP 326
11.3.5 Interpretability: Weak-Sense and Strong-Sense 328
11.4 Summary 331
References 333
Appendix Glossary 336