Statistical Significance Testing for Natural Language Processing

Rotem Dror is a Ph.D. student in the Natural Language Processing Research Group under the supervision of Professor Roi Reichart at the Technion, Israel Institute of Technology. Rotem's research interests lie in the intersection of Machine Learning, Statistics, Optimization, and Natural Language Processing. In her Ph.D., she focuses mainly on developing statistical methods for evaluating results of NLP tasks and on novel algorithms for structured prediction in NLP. Rotem's papers have been published in the top-tier conferences and journals of the NLP community. Rotem is a recipient of the Google Ph.D. Fellowship 2018.Lotem Peled-Cohen holds an M.Sc. in Natural Language Processing (cum laude) from the Technion, under the supervision of Professor Roi Reichart. Lotem's research revolved around textual sarcasm, and her work about sarcasm interpretation using monolingual Machine Translation was published in the ACL 2017 Proceedings and appeared in multiple media channels. After her studies, Lotem worked as a Data Scientist, focusing mostly on Conversational AI. She later became an independent consultant and lecturer at ML, NLP, and Deep Learning. Lotem was a lecturer at multiple colleges and presented in conferences worldwide. Nowadays, Lotem brings her ML & NLP expertise to the world of product management at Samsung Next, as part of the Whisk product department. Lotem works as an ML Product Manager who leadsa collaboration between Samsung offices worldwide (Korea, Russia, US, and Israel) responsible for building innovative, intelligent, and trustworthy ML products.Segev Shlomov is a Ph.D. student under the supervision of Associate Professor Yakov Babichenko at the Technion, Israel Institute of Technology. Segev's research interests lie at the intersection of Statistics, Social Learning, and Information Theory. Segev holds an M.Sc. in Operations Research (summa cum laude) from the Technion. He was three-time a summer intern at the Artificial Intelligence department of the IBM research labs, Haifa, Israel, and he is one of the main contributors to IBM's Lambada AI service. Segev's papers have been published in top-tier conferences and journals of both the NLP community and the Economics and Computation communities. Segev is a recipient of the Jacobs Outstanding Ph.D. Scholarship for the year 2020.Roi Reichart is an Associate Professor at the Technion, Israel Institute of Technology. Before joining the Technion, on July 2013, he was a post-doctoral researcher at the Computer Laboratory of the University of Cambridge, UK, and at the Computer Science and Artificial Intelligence Laboratory (CSAIL) of MIT. Prior to this, he was a Ph.D. student under the supervision of Professor Ari Rappoport at the Interdisciplinary Center for Neural Computation (ICNC) of Hebrew University of Jerusalem. His main research interest is NLP, with a focus on language learning in its context and designing models that integrate domain and world knowledge with data-driven methods. He has hence worked on problems such as domain adaptation, learning with minimal human annotation (and involvement), language transfer and multilingual learning, multi-modal (text and vision) processing, and NLP of Web data. He has focused on structured aspects of language and has developed effective algorithms for inference across linguistic structures. Finally, he is interested in proper evaluation of NLP algorithms and has worked on problems such as measuring statistical significance in NLP, word embedding evaluation, and unsupervised learning (particularly clustering) evaluation.

Preface.- Acknowledgments.- Introduction.- Statistical Hypothesis Testing.- Statistical Significance Tests.- Statistical Significance in NLP.- Deep Significance.- Replicability Analysis.- Open Questions and Challenges.- Conclusions.- Bibliography.- Authors' Biographies.