Making Sense of Data I (eBook)

Glenn J. Myatt, PhD, is Chief Scientific Officer and Cofounder of Leadscope, Inc. The author of numerous journal articles, Dr. Myatt, is also the coauthor of Making Sense of Data II: A Practical Guide to Data Visualization, Advanced Data Mining Methods, and Applications and Making Sense of Data III: A Practical Guide to Designing Interactive Data Visualizations, both of which are published by Wiley. Wayne P. Johnson, MSc, is Cofounder of Leadscope, Inc., as well as a partner of Myatt & Johnson, Inc. He has over 35 years of experience in software engineering related to operating systems, telecommunications, and artificial intelligence at various companies including IBM, AT&T Bell Laboratories, and Ford Motor Company. He has led research projects related to informatics, and in addition to authoring numerous journal articles, Mr. Johnson is the coauthor of Making Sense of Data II: A Practical Guide to Data Visualization, Advanced Data Mining Methods, and Applications and Making Sense of Data III: A Practical Guide to Designing Interactive Data Visualizations, both of which are published by Wiley.

Preface ix

1 Introduction 1

1.1 Overview 1

1.2 Sources of Data 2

1.3 Process for Making Sense of Data 3

1.4 Overview of Book 13

1.5 Summary 16

Further Reading 16

2 Describing Data 17

2.1 Overview 17

2.2 Observations and Variables 18

2.3 Types of Variables 20

2.4 Central Tendency 22

2.5 Distribution of the Data 24

2.6 Confidence Intervals 36

2.7 Hypothesis Tests 40

Exercises 42

Further Reading 45

3 Preparing Data Tables 47

3.1 Overview 47

3.2 Cleaning the Data 48

3.3 Removing Observations and Variables 49

3.4 Generating Consistent Scales Across Variables 49

3.5 New Frequency Distribution 51

3.6 Converting Text to Numbers 52

3.7 Converting Continuous Data to Categories 53

3.8 Combining Variables 54

3.9 Generating Groups 54

3.10 Preparing Unstructured Data 55

Exercises 57

Further Reading 57

4 Understanding Relationships 59

4.1 Overview 59

4.2 Visualizing Relationships Between Variables 60

4.3 Calculating Metrics About Relationships 69

Exercises 81

Further Reading 82

5 Identifying and Understanding Groups 83

5.1 Overview 83

5.2 Clustering 88

5.3 Association Rules 111

5.4 Learning Decision Trees from Data 122

Exercises 137

Further Reading 140

6 Building Models From Data 141

6.1 Overview 141

6.2 Linear Regression 149

6.3 Logistic Regression 161

6.4 k-Nearest Neighbors 167

6.5 Classification and Regression Trees 172

6.6 Other Approaches 178

Exercises 179

Further Reading 182

Appendix A Answers to Exercises 185

Appendix B Hands-on Tutorials 191

B. 1 Tutorial Overview 191

B. 2 Access and Installation 191

B. 3 Software Overview 192

B. 4 Reading in Data 193

B. 5 Preparation Tools 195

B. 6 Tables and Graph Tools 199

B. 7 Statistics Tools 202

B. 8 Grouping Tools 204

B. 9 Models Tools 207

B. 10 Apply Model 211

B. 11 Exercises 211

Bibliography 227

Index 231