Statistics for Censored Environmental Data Using Minitab and R, Second Edition

Dennis R. Helsel, PhD, is owner and Principal Scientist of Practical Stats, where he designs and conducts training courses in environmental statistics for scientists. He has over thirty years of experience working with the U.S. Geological Survey and is the author of numerous published articles on nondetect data and statistical methods in the environmental sciences. Dr. Helsel is the recipient of the Distinguished Service Award from the U.S. Department of the Interior (2007) as well as the Distinguished Achievement Award from the American Statistical Association (2003).

Preface ix Acknowledgments xi Introduction to the First Edition: An Accident Waiting To Happen xiii Introduction to the Second Edition: Invasive Data xvii 1 Things People Do with Censored Data that Are Just Wrong 1 Why Not Substitute-Missing the Signals that Are Present in the Data 3 Why Not Substitute?-Finding Signals that Are Not There 8 So Why Not Substitute? 10 Other Common Misuses of Censored Data 10 2 Three Approaches for Censored Data 12 Approach 1: Nonparametric Methods after Censoring at the Highest Reporting Limit 13 Approach 2: Maximum Likelihood Estimation 14 Approach 3: Nonparametric Survival Analysis Methods 17 Application of Survival Analysis Methods to Environmental Data 17 Parallels to Uncensored Methods 21 3 Reporting Limits 22 Limits When the Standard Deviation is Considered Constant 23 Insider Censoring-Biasing Interpretations 29 Reporting the Machine Readings of all Measurements 33 Limits When the Standard Deviation Changes with Concentration 34 For Further Study 36 4 Reporting, Storing, and Using Censored Data 37 Reporting and Storing Censored Data 37 Using Interval-Censored Data 41 Exercises 42 5 Plotting Censored Data 44 Boxplots 44 Histograms 46 Empirical Distribution Function 47 Survival Function Plots 49 Probability Plot 52 X-Y Scatterplots 59 Exercises 61 6 Computing Summary Statistics and Totals 62 Nonparametric Methods after Censoring at the Highest Reporting Limit 62 Maximum Likelihood Estimation 64 The Nonparametric Kaplan-Meier and Turnbull Methods 70 ROS: A "Robust" Imputation Method 79 Methods in Excel 86 Handling Data with High Reporting Limits 86 A Review of Comparison Studies 87 Summing Data with Censored Observations 94 Exercises 98 7 Computing Interval Estimates 99 Parametric Intervals 100 Nonparametric Intervals 103 Intervals for Censored Data by Substitution 103 Intervals for Censored Data by Maximum Likelihood 104 Intervals for the Lognormal Distribution 112 Intervals Using "Robust" Parametric Methods 125 Nonparametric Intervals for Censored Data 126 Bootstrapped Intervals 136 For Further Study 140 Exercises 141 8 What Can be Done When All Data Are Below the Reporting Limit? 142 Point Estimates 143 Probability of Exceeding the Reporting Limit 144 Exceedance Probability for a Standard Higher than the Reporting Limit 148 Hypothesis Tests Between Groups 151 Summary 152 Exercises 152 9 Comparing Two Groups 153 Why Not Use Substitution? 154 Simple Nonparametric Methods After Censoring at the Highest Reporting Limit 156 Maximum Likelihood Estimation 161 Nonparametric Methods 167 Value of the Information in Censored Observations 178 Interval-Censored Score Tests: Testing Data that Include (DL to RL) Values 180 Paired Observations 183 Summary of Two-Sample Tests for Censored Data 192 Exercises 192 10 Comparing Three or More Groups 194 Substitution Does Not Work-Invasive Data 195 Nonparametric Methods after Censoring at the Highest Reporting Limit 196 Maximum Likelihood Estimation 199 Nonparametric Method-The Generalized Wilcoxon Test 209 Summary 215 Exercises 216 11 Correlation 218 Types of Correlation Coefficients 218 Nonparametric Methods after Censoring at the Highest Reporting Limit 219 Maximum Likelihood Correlation Coefficient 224 Nonparametric Correlation Coefficient-Kendall's Tau 227 Interval-Censored Score Tests: Testing Correlation with (DL to RL) Values 230 Summary: A Comparison Among Methods 232 For Further Study 234 Exercises 235 12 Regression and Trends 236 Why Not Substitute? 237 Nonparametric Methods After Censoring at the Highest Reporting Limit 239 Maximum Likelihood Estimation 249 Akritas-Theil-Sen Nonparametric Regression 258 Additional Methods for Censored Regression 264 Exercises 266 13 Multivariate Methods for Censored Data 268 A Brief Overview of Multivariate Procedures 269 Nonparametric Methods After Censoring at the Highest Reporting Limit 273 Multivariate Methods for Data with Multiple Reporting Limits 288 Summary of Multivariate Methods for Censored Data 296 14 The NADA for R Software 297 A Brief Overview of R and the NADA Software 297 Summary of the Commands Available in NADA 300 Appendix: Datasets 303 References 309 Index 321