Advancing Natural Language Processing in Educational Assessment

Victoria Yaneva is Senior NLP Scientist at the National Board of Medical Examiners, USA. Matthias von Davier is Monan Professor of Education in the Lynch School of Education and Executive Director of TIMSS & PIRLS International Study Center at Boston College, USA.

Preface

by Victoria Yaneva and Matthias von Davier

Section I: Automated Scoring

Chapter 1: The Role of Robust Software in Automated Scoring

by Nitin Madnani, Aoife Cahill, and Anastassia Loukina

Chapter 2: Psychometric Considerations when Using Deep Learning for Automated Scoring

by Susan Lottridge, Chris Ormerod, and Amir Jafari

Chapter 3: Speech Analysis in Assessment

by Jared C. Bernstein and Jian Cheng

Chapter 4: Assessment of Clinical Skills: A Case Study in Constructing an NLP-Based Scoring System for Patient Notes

by Polina Harik, Janet Mee, Christopher Runyon, and Brian E. Clauser

Section II: Item Development

Chapter 5: Automatic Generation of Multiple-Choice Test Items from Paragraphs Using Deep Neural Networks

by Ruslan Mitkov, Le An Ha, Halyna Maslak, Tharindu Ranasinghe, and Vilelmini Sosoni

Chapter 6: Training Optimus Prime, M.D.: A Case Study of Automated Item Generation using Artificial Intelligence – From Fine-Tuned GPT2 to GPT3 and Beyond

by Matthias von Davier

Chapter 7: Computational Psychometrics for Digital-first Assessments: A Blend of ML and Psychometrics for Item Generation and Scoring

by Geoff LaFlair, Kevin Yancey, Burr Settles, Alina A von Davier

Section III: Validity and Fairness

Chapter 8: Validity, Fairness, and Technology-based Assessment

by Suzanne Lane

Chapter 9: Evaluating Fairness of Automated Scoring in Educational Measurement

by Matthew S. Johnson and Daniel F. McCaffrey

Section IV: Emerging Technologies

Chapter 10: Extracting Linguistic Signal from Item Text and Its Application to Modeling Item Characteristics

by Victoria Yaneva, Peter Baldwin, Le An Ha, and Christopher Runyon

Chapter 11: Stealth Literacy Assessment: Leveraging Games and NLP in iSTART

by Ying Fang, Laura K. Allen, Rod D. Roscoe, and Danielle S. McNamara

Chapter 12: Measuring Scientific Understanding Across International Samples: The Promise of Machine Translation and NLP-based Machine Learning Technologies

by Minsu Ha and Ross H. Nehm

Chapter 13: Making Sense of College Students’ Writing Achievement and Retention with Automated Writing Evaluation

by Jill Burstein, Daniel McCaffrey, Steven Holtzman & Beata Beigman Klebanov

Contributor Biographies