Statistical Modeling of the National Assessment of Educational Progress (eBook)

Preface 6
Acknowledgements 8
Contents 10
Chapter 1 14
Theories of Data Analysis and Statistical Inference 14
1.1 Introduction 14
1.2 Example 15
1.3 Statistical models 15
1.4 The likelihood function 18
1.5 Theories 19
1.5.1 Likelihood-based repeated sampling theory 19
1.5.2 Bayes theory 19
1.5.3 “Model assisted” survey sampling theory 21
1.6 Weighting 26
1.6.1 Stratified random sampling 26
1.6.2 Design-based analysis 27
1.6.3 Model-based analysis 28
1.6.4 Weighted likelihoods 29
1.7 Missing data and non-response 31
1.7.1 Weighting adjustments for nonresponse 32
1.7.2 Incomplete data in regression 33
1.7.3 Multiple imputation 33
Chapter 2 35
The Current Design and Analysis 35
2.1 NCES and NAEP 35
2.2 Design 36
2.2.1 PSUs 36
2.2.2 Schools 37
2.2.3 Students 37
2.2.4 Test items 37
2.2.5 Important design issues 38
2.3 NAEP state sample design 2002+ 38
2.4 Weighting 38
2.4.1 Design effect corrections 39
2.5 Analysis 40
2.5.1 Item models 40
2.5.2 Multidimensional ability 42
2.5.3 Inference and the likelihood function 46
2.5.4 The ability regression model 47
2.5.5 Current model parameter estimation 48
2.5.6 Plausible value imputation 48
Chapter 3 51
Psychometric and Survey Models 51
3.1 The Rasch model 51
3.2 The 2PL and MIMIC models 52
3.3 Three-parameter models 54
3.4 Partial credit model 56
3.5 The HYBRID model 56
3.6 Extensions of the guessing model 57
3.6.1 A four-parameter guessing model 57
3.6.2 The “2-guess” model 58
3.6.3 The “2-mix” model – a five-parameter general mixture of logits model 58
3.7 Modeling the component membership probability 60
3.8 Multidimensional ability 60
3.9 Clustering and variance component models 62
3.9.1 Three-level models 63
3.9.2 Four-level models 64
3.10 Summary of the full model for NAEP analysis 65
Chapter 4 67
Technical Reports – Data Analyses and Simulation Studies 67
4.1 Research reports 67
Chapter 5 74
Model-Based Analysis of the 1986 NAEP Math Survey 74
5.1 Data and model specification – subscale 74
5.2 Model aspects 76
5.2.1 Maximised log-likelihoods 76
5.2.2 Twoand three-level models 76
5.2.3 Four-level models 77
5.2.4 The 3PL model 77
5.3 Reporting group differences 78
5.3.1 Comparison with NAEP subscale estimates 80
5.4 Mixture models 82
5.4.1 2-guess model 83
5.4.2 2-guess-prob model 83
5.4.3 Two-dimensional model 84
5.4.4 2-mix model 85
5.4.5 2-mix-regressions model 85
5.4.6 2-mix-prob model 85
5.4.7 Conclusions from the 30-item analysis 86
Chapter 6 88
Analysis of All 1986 Math Iems 88
6.1 The full math test 88
6.2 2PL models 89
6.3 Results 90
6.3.1 Mixed 2PL models 91
6.3.2 Three-component membership models 92
6.3.3 Multidimensional ability model 93
6.4 MIMIC models 94
6.5 Results 94
6.5.1 Two-parameter MIMIC model 94
6.5.2 Mixed MIMIC models 95
6.6 Comparison with published NAEP results 96
6.7 Discussion 97
Chapter 7 98
Analysis of the 2005 NAEP Math Survey – Texas 98
7.1 Population, sample, and test 98
7.2 Variable names and codes 99
7.3 Models fitted 99
7.4 Results – limited teacher data 101
7.4.1 Three-parameter interpretation 101
7.4.2 Mixture models 102
7.5 Boundary values in logistic regression 104
7.6 Results – extensive teacher data 104
7.6.1 Mixture models 106
7.7 Comparison with official NCES analysis 107
7.8 Conclusion 109
Chapter 8 110
Analysis of the 2005 NAEP Math Survey – California 110
8.1 Population, sample, and test 110
8.2 Models 111
8.3 Results 111
8.3.1 Limited teacher data 111
8.3.2 3PL interpretation 112
8.4 Mixture models 113
8.5 Extensive teacher data 114
8.5.1 3PL interpretation 115
8.6 Mixture models 116
8.7 Comparison with official NCES analysis 118
8.8 Conclusion 120
Chapter 9 121
Conclusions 121
9.1 The nature and structure of models 121
9.2 Our modeling results 122
9.2.1 Comparisons with published NAEP tables 122
9.2.2 Main effects and interactions 122
9.2.3 Mixtures and latent subpopulations 123
9.3 Current analysis 126
9.3.1 Dependence of design on analysis 126
9.3.2 Multilevel modeling 127
9.3.3 The limitations of NAEP data for large-scale modeling 127
9.4 The reporting of NAEP data 128
9.5 The future analysis and use of NAEP data 129
9.6 Resolution of the model-comparison difficulties 130
9.7 Resolution of the problems with incomplete data 130
Appendix A 131
Appendix B 143
Appendix C 151
Model Parameter Estimates and SEs, 2005 Texas Survey 151
1. Variable names and definitions, national NAEP sample 151
C.1 Parameter estimates and SEs – limited teacher data 156
C.2 Parameter estimates and SEs – extensive teacher data 158
Appendix D 159
Model Parameter Estimates and SEs, 2005 California survey 159
D.1 Parameter estimates for MIMIC models – limited teacher data 159
D.2 Parameter estimates for MIMIC models – extensive teacher data 163
References 167
Author Index 170