Language Intelligence

Akshi Kumar is a Senior Lecturer and Director for Post-Graduate Research with the Department of Computing at Goldsmiths, University of London, UK. Dr. Kumar earned her PhD in Computer Science & Engineering from the University of Delhi, India, in 2011, and her research interests include sentiment analysis, affective computing, cyber-informatics, psychometric NLP, and more. She has been ranked #8 globally for Sentiment Analysis over the past 5 years by ScholarGPS. Her name has been included in the “Top 2% scientist of the world” list by Stanford University, USA in 2023, 2022 and 2021.

List of Figures xiii

List of Tables xv

About the Author xvii

Preface xix

Acknowledgements xxi

1 Foundations of Natural Language Processing 1

1.1 History of NLP 3

1.2 Approaches to NLP 5

1.3 Understanding NLP through NLU and NLG: Examples and Case Studies 9

1.3.1 Practical Case Studies 9

1.4 NLP Pipeline 10

1.5 NLP’s Transformative Impact on Business and Society 12

2 Navigating the Data Landscape for NLP 15

2.1 Types of Data in NLP 16

2.2 Data Acquisition 17

2.3 Challenges in NLP Data Acquisition and Management 21

2.4 Data Quality Check in NLP 22

2.5 Ethical Considerations in NLP Data Management 25

3 Fundamental Text Processing 31

3.1 Text Cleaning 32

3.2 Sentence Splitting 34

3.3 Tokenization 36

3.4 Lemmatization and Stemming 44

3.5 StopWord Removal 48

3.6 Part-of-Speech Tagging 49

3.7 Parsing and Syntactic Analysis 50

3.8 Tools and Libraries for Text Processing 56

4 Linguistic Features in NLP 63

4.1 Levels of Linguistic Analysis 64

4.2 Features in NLP 73

4.3 Vector Space Representation in NLP 75

4.4 Semantic Features in NLP 81

4.5 Feature Generation in NLP: Manual versus Automatic Approaches 89

5 Computational and Cognitive Approaches in Natural Language Processing 95

5.1 Machine Learning for NLP 97

5.2 Memory and Recall Models 100

5.3 Attention Mechanisms 105

5.4 Human-Like Reasoning 112

5.5 Transfer Learning in NLP 114

5.6 Learning with Minimal Examples 122

5.7 Neuro-Symbolic Approaches 123

6 Fundamental Language Processing Techniques 129

6.1 Topic Modelling and Subject Identification 129

6.2 Named Entity Recognition 136

6.3 Text Coherence and Cohesion 144

6.4 Stylistic Analysis 151

6.5 Semantic Role Labelling 154

7 Natural Language Processing for Affective, Psychological, and Content Analysis 159

7.1 Sentiment Analysis: Dissecting Text for Opinion Mining 159

7.2 Emotion Recognition: Beyond Polarity 168

7.3 Irony and Sarcasm Detection: Between the Lines 175

7.4 Humor Identification in Text: Tapping into Textual Tickle 180

7.5 Psychometric NLP 184

7.6 Learning Disabilities Detection 190

7.7 Textual Indicators of Distress: Addressing Depression, Anxiety, and Beyond 194

7.8 Digital Content Moderation using NLP 198

8 Multilingual Natural Language Processing 223

8.1 Translation and Transliteration 223

8.2 Cross-Lingual Models and Embeddings 228

8.3 Low-Resource Language Processing 235

8.4 Cultural Nuance and Idiom Recognition in Natural Language Processing 240

9 Domain-Specific Natural Language Processing 243

9.1 Healthcare Natural Language Processing 243

9.2 Legal Natural Language Processing 250

9.3 Finance Natural Language Processing 255

9.4 NLP in Education 262

10 Measuring Success in Natural Language Processing Evaluation and Metrics 269

10.1 Intrinsic versus Extrinsic Evaluation Techniques 269

10.2 Extrinsic Evaluation Techniques 270

10.3 Metrics for Text Classification 272

10.4 Evaluating Machine Translation and Text Summarization 276

10.5 Metrics for Question-Answering and Conversational AI 278

10.5.1 Evaluation in Ranking and Information Retrieval 279

10.6 Metrics for Text-Based Forecasting and Prediction 280

Knowledge Checkpoint Answers 285

A Text Representation Techniques: A Unified Overview 305

B Step-by-Step Guide to NLP Processing on E-Commerce Customer Feedback 307

C Harnessing Python Libraries for NLP 311

Further Reading 313

Index 315