Dirty Data Processing for Machine Learning (eBook)

In both the database and machine learning communities, data quality has become a serious issue which cannot be ignored. In this context, we refer to data with quality problems as 'dirty data.' Clearly, for a given data mining or machine learning task, dirty data in both training and test datasets can affect the accuracy of results. Accordingly, this book analyzes the impacts of dirty data and explores effective methods for dirty data processing.

Although existing data cleaning methods improve data quality dramatically, the cleaning costs are still high. If we knew how dirty data affected the accuracy of machine learning models, we could clean data selectively according to the accuracy requirements instead of cleaning all dirty data, which entails substantial costs. However, no book to date has studied the impacts of dirty data on machine learning models in terms of data quality. Filling precisely this gap, the book is intended for a broad audience ranging from researchers in the database and machine learning communities to industry practitioners.

Readers will find valuable takeaway suggestions on: model selection and data cleaning; incomplete data classification with view-based decision trees; density-based clustering for incomplete data; the feature selection method, which reduces the time costs and guarantees the accuracy of machine learning models; and cost-sensitive decision tree induction approaches under different scenarios. Further, the book opens many promising avenues for the further study of dirty data processing, such as data cleaning on demand, constructing a model to predict dirty-data impacts, and integrating data quality issues into other machine learning models. Readers will be introduced to state-of-the-art dirty data processing techniques, and the latest research advances, while also finding new inspirations in this field.

Zhixin Qi is an assistant professor in the School of Transportation Science and Engineering at Harbin Institute of Technology. She received her PhD from the School of Computer Science and Technology at Harbin Institute of Technology. Her research interests include knowledge graph, AI4DB, and graph data management. She has published more than 10 papers in international journals and conferences, including TKDE, KAIS, KBS, Neurocomputing, WWWJ, JCST, CIKM, and DASFAA.

Hongzhi Wang is a professor and doctoral supervisor at the School of Computer Science and Technology, Harbin Institute of Technology. His research interests include big data management and analysis, data quality, graph data management, and web data management. He has published more than 150 papers, and he is the primary investigator of more than 10 projects including three NSFC projects, and co-PI of 973, 863, and NSFC key projects. He was awarded as Microsoft fellowship, China Excellent Database Engineer, and IBM PhD fellowship.

Zejiao Dong is a full professor in the School of Transportation Science and Engineering of Harbin Institute of Technology. He has served as School Dean and Deputy Director of MOT Key Laboratory of Specialized Material and Intelligent Control for Traffic Safety as well as MIIT Key Laboratory of Intellectualization and Safety Assurance for Cold Regional Transportation Infrastructure. His research interests include intelligent monitoring, massive data analysis, and dynamic mechanics of transportation infrastructures. He has published over 60 research papers in top-quality international journals as well as three academic monographs in national press.

---

In both the database and machine learning communities, data quality has become a serious issue which cannot be ignored. In this context, we refer to data with quality problems as "e;dirty data."e; Clearly, for a given data mining or machine learning task, dirty data in both training and test datasets can affect the accuracy of results. Accordingly, this book analyzes the impacts of dirty data and explores effective methods for dirty data processing.Although existing data cleaning methods improve data quality dramatically, the cleaning costs are still high. If we knew how dirty data affected the accuracy of machine learning models, we could clean data selectively according to the accuracy requirements instead of cleaning all dirty data, which entails substantial costs. However, no book to date has studied the impacts of dirty data on machine learning models in terms of data quality. Filling precisely this gap, the book is intended for a broad audience ranging from researchers inthe database and machine learning communities to industry practitioners.Readers will find valuable takeaway suggestions on: model selection and data cleaning; incomplete data classification with view-based decision trees; density-based clustering for incomplete data; the feature selection method, which reduces the time costs and guarantees the accuracy of machine learning models; and cost-sensitive decision tree induction approaches under different scenarios. Further, the book opens many promising avenues for the further study of dirty data processing, such as data cleaning on demand, constructing a model to predict dirty-data impacts, and integrating data quality issues into other machine learning models. Readers will be introduced to state-of-the-art dirty data processing techniques, and the latest research advances, while also finding new inspirations in this field.