Applying Test Equating Methods (eBook)

Foreword 7
References 9
Preface 10
References 12
Contents 13
Acronyms 18
List of Symbols 20
1 General Equating Theory Background 24
1.1 Introduction 24
1.1.1 A Conceptual Description of Equating 25
1.1.2 A Statistical Model View of Equating 25
1.2 Statistical Models 26
1.2.1 General Definition, Notation, and Examples 26
1.2.2 Types of Statistical Models 27
1.2.3 Mathematical Statistics Formulation of the Equating Problem 29
1.2.4 Mathematical Form of the Equating Transformation 30
1.2.5 Continuization 31
1.2.6 Requirements for Comparability of Scores 32
1.2.7 Assessing the Uncertainty of Equating Results 32
1.3 Collecting Data in Equating 33
1.3.1 Data Collection Designs in Equating 34
1.3.1.1 Single Group Design 34
1.3.1.2 Equivalent Groups Design 34
1.3.1.3 Counterbalanced Design 34
1.3.1.4 Non Equivalent Groups with Anchor Test Design 35
1.3.1.5 Non Equivalent Groups with Covariates Design 35
1.4 Some Examples of Equating Transformations 36
1.4.1 The Equipercentile Equating Function 36
1.4.2 The Linear Equating Function 37
1.4.3 The Kernel Equating Function 37
1.5 R Packages That Are Used in This Book 38
1.6 Summary and Overview of the Book 38
References 39
2 Preparing Score Distributions 42
2.1 Data 42
2.1.1 Data from Ch2:kolenbrennan2014 42
2.1.2 Data from Ch2:vondavieretal2004 43
2.1.3 The ADM Admissions Test Data 43
2.1.4 The SEPA Test Data 44
2.2 Preparing the Score Data 44
2.2.1 Functions to Create Score Frequency Distributions 45
2.2.2 Score Data in the EG Design 45
2.2.3 Score Data in the SG Design 50
2.2.4 Score Data in the NEAT Design 53
2.3 Presmoothing the Score Distributions 56
2.3.1 Polynomial Log-Linear Models for Presmoothing 56
2.3.2 Polynomial Log-Linear Smoothing in equate 58
2.3.3 Examples 59
2.3.3.1 Smoothing Univariate Distributions 59
2.3.3.2 Smoothing a Bivariate Distribution 60
2.3.4 Choosing the Best Log-Linear Model 61
2.4 Using Other Arguments, Packages and Functions 64
2.5 Summary 65
References 65
3 Traditional Equating Methods 66
3.1 Equipercentile, Linear, and Mean Equating Transformations 66
3.2 Assumptions in the Different Designs 67
3.2.1 Assumptions in EG, SG, and CB Designs 67
3.2.2 Assumptions in the NEAT Design 68
3.3 Traditional Equating Methods for the EG, SG and CB Designs 69
3.4 Traditional Equating Methods for the NEAT Design 69
3.4.1 Linear Equating Methods for the NEAT Design 70
3.4.1.1 Tucker Equating 70
3.4.1.2 Nominal Weights Equating 71
3.4.1.3 Levine Observed-Score Equating 71
3.4.1.4 Levine True-Score Equating 72
3.4.1.5 Chained Linear Equating 73
3.4.2 Equipercentile Equating Methods for the NEAT Design 73
3.4.2.1 Frequency Estimation 73
3.4.2.2 Chained Equipercentile Equating 74
3.4.2.3 Braun-Holland Equating 74
3.5 Examples with the equate Function 75
3.5.1 The equate Function 75
3.5.2 Examples Under the EG and SG Designs 76
3.5.3 Examples Under the NEAT Design 83
3.5.3.1 Linear Methods 83
3.5.3.2 Equipercentile Methods 83
3.5.3.3 Comparison Between Linear and Equipercentile Methods 84
3.5.4 Examples Using the ADM Data Under the NEAT Design 86
3.6 Additional Features in equate 86
3.7 Performing Traditional Equating Methods with SNSequate 87
3.8 Comparing Traditional Test Equating Methods 88
3.8.1 Bootstrap Standard Errors of Equating 88
3.8.2 Bias and RMSE 89
3.8.3 Examples Using equate 90
3.8.4 Additional Example: A Comparison of Traditional Equating Methods 91
3.9 Summary 94
References 94
4 Kernel Equating 96
4.1 A Quick Overview of Kernel Equating 96
4.2 Step 1: Presmoothing 97
4.2.1 Presmoothing with SNSequate 97
4.2.1.1 Presmoothing Under the EG Design 98
4.2.1.2 Presmoothing Under the SG Design 99
4.2.1.3 Presmoothing Under the CB Design 101
4.2.1.4 Presmoothing Under the NEAT Design 101
4.2.1.5 Modeling Complexities in the Data 102
4.2.2 Presmoothing with kequate 104
4.2.2.1 Presmoothing Under the EG Design 104
4.2.2.2 Presmoothing Under the SG Design 105
4.2.2.3 Presmoothing Under the CB Design 106
4.2.2.4 Presmoothing Under the NEAT Design 106
4.2.2.5 Modeling Complexities in the Data 107
4.2.2.6 Presmoothing Under the NEC Design 108
4.2.3 Assessing Log-Linear Model Fit 109
4.2.3.1 Assessing Log-Linear Model Fit in SNSequate 110
4.2.3.2 Assessing Log-Linear Model Fit in kequate 111
4.3 Step 2: Estimation of Score Probabilities 113
4.3.1 Estimation of Score Probabilities with SNSequate 113
4.3.2 Estimation of Score Probabilities with kequate 114
4.4 Step 3: Continuization 115
4.4.1 Bandwidth Selection 116
4.4.2 Choosing the Kernel 116
4.4.3 Continuization Choices in SNSequate 117
4.4.4 Continuization Choices in kequate 117
4.5 Step 4: Equating 118
4.5.1 Equating in SNSequate 118
4.5.2 Equating in kequate 122
4.6 Step 5: Computation of Accuracy Measures 125
4.6.1 Calculating the Standard Error of Equating 126
4.6.2 Standard Error of Equating Difference 126
4.6.3 Percent Relative Error 126
4.6.4 Obtaining SEE, SEED, and PRE in SNSequate 127
4.6.5 Obtaining SEE, SEED, and PRE in kequate 129
4.7 Different Features in kequate and SNSequate 132
4.8 Summary 132
References 132
5 Item Response Theory Equating 134
5.1 IRT Models 134
5.1.1 Scoring Using IRT Models 135
5.2 Equating IRT Scores 136
5.2.1 Parameter Linking 136
5.2.1.1 Moments Methods to Estimate Equating Coefficients 137
5.2.1.2 Characteristic Curves Methods to Estimate Equating Coefficients 138
5.2.1.3 IRT Parameter Linking Using SNSequate 138
5.2.1.4 IRT Parameter Linking Using equateIRT 139
5.3 Equating Observed Scores Under the IRT Framework 142
5.3.1 IRT True-Score Equating 143
5.3.2 IRT Observed-Score Equating 143
5.3.3 IRT True-Score and Observed-Score Equating Using SNSequate 144
5.3.4 IRT True-Score and Observed-Score Equating Using equateIRT 149
5.4 Other Equating Methods for IRT Scores 151
5.4.1 Concurrent Calibration 151
5.4.1.1 Concurrent Calibration Using ltm 152
5.4.2 Fixed Item Parameter Calibration 154
5.4.2.1 Fixed Item Parameter Calibration Using mirt 154
5.5 Other R Packages for IRT Analysis 156
5.6 Summary 157
References 157
6 Local Equating 160
6.1 The Concept of Local Equating 160
6.1.1 True Equating Transformation 161
6.2 Performing Local Equating 162
6.3 Local Linear Equating Transformations 162
6.3.1 Local Linear Equating Conditioning on Anchor Test Scores: NEAT Design 163
6.3.2 Local Linear Equating Method of Conditional Means: SG Design 163
6.3.3 Local Linear Equating Examples in R 163
6.3.3.1 Implementing Local Linear Equating Conditioning on Anchor Test Scores 164
6.3.3.2 Implementing the Local Linear Equating Method of Conditional Means 167
6.4 Local Equipercentile Equating Transformations 167
6.4.1 Local IRT Observed-Score Equating 168
6.4.2 Local Observed-Score Kernel Equating Conditioning on Anchor Test Scores 169
6.4.3 Local IRT Observed-Score Kernel Equating 169
6.4.4 Local Equipercentile Equating Examples in R 170
6.4.4.1 Local IRT Observed-Score Equating Using SNSequate 170
6.4.4.2 Local Observed-Score Kernel Equating Using kequate 172
6.4.4.3 Local IRT Observed-Score Kernel Equating Using kequate 174
6.5 Other Local Equating Methods 177
6.6 Summary 177
References 177
7 Recent Developments in Equating 179
7.1 Alternative Kernel Equating Transformations 179
7.1.1 Epanechnikov Kernel 179
7.1.2 Adaptive Kernels 180
7.1.3 Examples of Epanechnikov and Adaptive Kernel Equating in SNSequate 181
7.2 Bandwidth Selection in Kernel Equating 183
7.2.1 Rule-Based Bandwidth Selection 183
7.2.2 Bandwidth Selection with Double Smoothing 184
7.2.3 Examples of the Rule-Based and Double Smoothing Bandwidth Selection Methods Using kequate 184
7.3 Item Response Theory Kernel Equating 185
7.3.1 Two Polytomous IRT Models 185
7.3.2 Performing IRT Kernel Equating with kequate 186
7.3.3 Examples of IRT Kernel Equating for Binary Scored Items Using kequate 187
7.3.4 Examples of IRT Kernel Equating for Polytomous Scored Items Using kequate 189
7.4 Bayesian Nonparametric Approach to Equating 190
7.4.1 Bayesian Nonparametric Modeling 190
7.4.2 BNP Model for Equating 191
7.4.3 An Illustration of the BNP Model for Equating in SNSequate 192
7.5 Assessing the Equating Transformation 194
7.5.1 An Illustration of Assessing (x) in Kernel Equating Using SNSequate 196
7.6 Summary 199
References 199
Appendix A Installing and Reading Data in R 201
A.1 Installing R 201
A.1.1 R Studio 201
A.2 Installing and Loading R Packages 202
A.3 Working Directory and Accessing Data 202
A.4 Loading Data of Different File Formats 203
Reference 204
Appendix B Additional Material 205
B.1 Design Functions 205
B.2 C C C C Matrices 207
B.3 Calculation of the SEE 207
B.4 Score Distributions Under the NEAT Design 208
B.5 The Lord-Wingersky Algorithm 209
B.6 Other Justifications for Local Equating 210
B.7 Epanechnikov Kernel Density Estimate and Derivatives 211
B.8 The Double Smoothing Bandwidth Selection Method in Kernel Equating 212
B.9 The DBPP Model 213
B.10 Measures of Statistical Assessment When Equating Test Scores 213
References 214
Index 216