Elements of Adaptive Testing (eBook)

Preface 5
Contributors 9
Contents 11
Part I Item Selection and Ability Estimation 14
1 Item Selection and Ability Estimation in Adaptive Testing 15
1.1 Introduction 15
1.2 Classical Procedures 17
1.2.1 Notation and Some Statistical Concepts 17
1.2.2 Ability Estimators 19
1.2.3 Choice of Estimator 20
1.2.4 Classical Item-Selection Criteria 23
1.3 Modern Procedures 24
1.3.1 Maximum Global-Information Criterion 25
1.3.2 Likelihood-Weighted Information Criterion 27
1.3.3 Fully Bayesian Criteria 28
1.3.4 Bayesian Criteria with Collateral Information 30
1.3.5 Bayesian Criteria with Random Item Parameters 33
1.3.6 Miscellaneous Criteria 35
1.3.7 Evaluation of Item-Selection Criteria and Ability Estimators 36
1.4 Concluding Remarks 39
References 40
2 Constrained Adaptive Testing with Shadow Tests 43
2.1 Introduction 43
2.2 Review of Existing Methods for Constrained CAT 45
2.2.1 Item-Pool Partitioning 45
2.2.2 Weighted-Deviation Method 45
2.2.3 Maximum Priority Index Method 45
2.2.4 Testlet-Based Adaptive Testing 46
2.2.5 Multistage Testing 46
2.2.6 Evaluation of Existing Approaches 47
2.3 Constrained CAT with Shadow Tests 48
2.4 Technical Implementation 49
2.4.1 Basic Notation and Definitions 50
2.4.2 IP Model for Shadow Test 51
2.4.3 Numerical Aspects 53
2.5 Four Applications to Adaptive Testing Problems 54
2.5.1 CAT with Large Numbers of Nonstatistical Constraints 55
2.5.2 CAT with Response-Time Constraints 55
2.5.3 CAT with Item-Exposure Control 59
2.5.4 CAT with Equated Number-Correct Scores 62
2.6 Concluding Remarks 65
References 65
3 Principles of Multidimensional Adaptive Testing 68
3.1 Introduction 68
3.2 Literature Review 69
3.3 Multidimensional Item Selection and Scoring 70
3.3.1 Prior Density 71
3.3.2 Likelihood Function 72
3.3.3 Posterior Density 74
3.3.4 Item Selection 76
3.3.5 Posterior Inference 79
3.4 Example 80
3.4.1 Initialization 80
3.4.2 Item Selection 81
3.4.3 Provisional Ability Estimation 82
3.4.4 Item Selection and Scoring Cycle 82
3.5 Discussion 84
3.6 Appendix: Computational Formulas 84
References 85
4 Multidimensional Adaptive Testing with Kullback–Leibler Information Item Selection 87
4.1 Multidimensional IRT model 88
4.2 Bayesian Estimation of bold0mu mumu * 89
4.3 Kullback–Leibler Information 91
4.3.1 Mutual Information 94
4.4 Item Selection Using KL Information 94
4.4.1 Posterior Expected Kullback–Leibler Information 95
4.4.2 KL Distance between Subsequent Posteriors 97
4.4.3 Mutual Information 98
4.5 Relationship between Selection Criteria 99
4.6 Special Status of Some of the Ability Parameters 101
4.6.1 Nuisance Abilities 101
4.6.2 Composite Ability 104
4.7 Posterior Covariance 107
4.8 Conclusion 109
References 110
5 Sequencing an Adaptive Test Battery 112
5.1 Introduction 112
5.2 Multilevel Model 114
5.3 Empirical Bayes Approach 115
5.3.1 Selection of Initial Pool 115
5.3.2 Selection of First Test 117
5.3.3 Administration of First Test 118
5.3.4 Selection of Subsequent Tests 119
5.3.5 Administration of Subsequent Tests 120
5.4 Simulation Study 120
5.4.1 Design of Study 121
5.4.2 Results 122
5.5 Concluding Remarks 125
5.6 Appendix: Computational Approach 126
References 127
Part II Applications in Large-Scale Testing Programs 129
6 Adaptive Tests for Measuring Anxiety and Depression 130
6.1 Introduction 130
6.2 Development of CAT Systems 132
6.2.1 Patient Samples for Empirical Item Analyses 132
6.2.2 Definition of Target Construct 133
6.2.3 Initial Item Pool 133
6.2.4 Test Dimensionality 134
6.2.5 Nonparametric Analyses 134
6.2.6 DIF Analysis 135
6.2.7 Item Calibration 136
6.2.8 Investigation of Model Fit 136
6.2.9 Item Banks 137
6.2.10 CAT Algorithm 137
6.2.11 Delivery System 138
6.3 Evaluation Studies 138
6.4 Discussion 140
References 141
7 MATHCAT: A Flexible Testing System in Mathematics Education for Adults 144
7.1 Introduction 144
7.2 The Item Bank for Numerical and Mathematical Skills 145
7.3 Item-Selection Algorithm 146
7.3.1 Placement Testing 148
7.3.2 Achievement Testing 151
7.4 MATHCAT Software 154
7.5 Conclusions 155
References 156
8 Implementing the Graduate Management Admission Test Computerized Adaptive Test 157
8.1 Overview of the GMAT 157
8.1.1 Content 158
8.2 Becoming a Computerized Adaptive Test 159
8.3 Implementation Issues 162
8.3.1 Meeting Content Specifications 162
8.3.2 Item Exposure, Item Use, and the CAT Algorithm 163
8.3.3 Item Pool Characteristics 165
8.3.4 Item Bias 168
8.3.5 Item Parameter Shift 168
8.4 Conclusion 169
References 170
9 Designing and Implementing a Multistage Adaptive Test: The Uniform CPA Exam 172
9.1 Introduction 172
9.2 Decision Making 173
9.2.1 Content 173
9.2.2 Administration Models 175
9.2.3 Evaluation Criteria 178
9.2.4 The AICPA MST Model 179
9.3 Implementation 180
9.3.1 Test Assembly 180
9.3.2 Calibration and Statistical Analyses of Multiple-Choice Items 186
9.3.3 Controls for Security and Score Accuracy 188
9.3.4 Standard Setting 191
9.4 Conclusion 193
References 193
10 A Japanese Adaptive Test of English as a Foreign Language: Developmental and Operational Aspects 195
10.1 Introduction 195
10.2 Overview of the Test 195
10.3 CASEC Development Flow 199
10.3.1 Measurement Target 199
10.3.2 Item Format 200
10.3.3 Item Response Model 201
10.3.4 Item Construction and Pretesting 201
10.3.5 Description of the Item Bank 201
10.3.6 Testing Algorithms 203
10.3.7 Stopping Rule 206
10.3.8 Score Scale 206
10.4 Validity Research 207
10.4.1 Comparison of Measurement Accuracies Between CAT and P& P Versions of the Test
10.4.2 Examining CASEC Reliability and Validity 209
10.5 Current Challenges 211
10.5.1 Psychometric Issues 211
10.5.2 Operational Issues 214
10.6 Conclusion 215
References 215
Part III Item Pool Development and Maintenance 216
11 Innovative Items for Computerized Testing 217
11.1 Introduction 217
11.2 A Taxonomy for Innovative Items 218
11.2.1 Assessment Structure 219
11.2.2 Complexity 221
11.2.3 Fidelity 222
11.2.4 Interactivity 223
11.2.5 Media Inclusion 224
11.2.6 Response Action 227
11.2.7 Scoring Methods 228
11.3 Conclusion 229
References 230
12 Designing Item Pools for Adaptive Testing 233
12.1 Introduction 233
12.2 Review of Item-Pool-Design Literature 234
12.3 Designing a Blueprint for the Item Pool 236
12.3.1 Identifying the Design Space 236
12.3.2 Simulation of Adaptive Test Administrations 237
12.4 Empirical Example 242
12.4.1 Design Space 242
12.5 Some Related Issues 244
12.5.1 Exposure Control 244
12.5.2 Rotating Item Pools (Calculating the Blueprint) 245
12.5.3 Multidimensionality 245
12.6 Concluding Remarks 245
References 246
13 Assembling an Inventory of Multistage Adaptive Testing Systems 248
13.1 Introduction 248
13.2 Mixed-Integer Programming Concepts 250
13.3 The Ideal Bank (Steady-State Model) 253
13.3.1 Assumptions Required 254
13.3.2 Considerations for Linear and Adaptive Testing Formats 255
13.3.3 Inventory-Scheduling Problem 256
13.3.4 Analysis Methods 257
13.4 Illustration from the Uniform CPA Licensing Exam 258
13.4.1 Ideal Bank and Inventory-Supply Schedule 259
13.4.2 Supply-Scheduling Model 261
13.5 Discussion 264
References 266
Part IV Item Calibration and Model Fit 268
14 Item Parameter Estimation and Item Fit Analysis 269
14.1 Introduction 269
14.2 Item Parameter Estimation 270
14.2.1 MML Estimation 270
14.2.2 Impact of Violations of Ignorability on Item Parameter Estimation 274
14.2.3 Simulated Examples 276
14.3 Item Fit Analysis 277
14.3.1 Lagrange Multiplier Tests 277
14.3.2 An LM Test for the Fit of Item-Characteristic Curves 279
14.3.3 An LM Test for Parameter Drift 280
14.3.4 A CUSUM Test for Parameter Drift 281
14.4 Examples 282
14.5 Discussion 286
References 287
15 Estimation of the Parameters in an Item-Cloning Model for Adaptive Testing 289
15.1 Introduction 289
15.2 The Model 291
15.2.1 Level 1 Model 292
15.2.2 Level 2 Model 292
15.2.3 Prior for Hyperparameters 293
15.2.4 Likelihood Function 294
15.3 Sampling Design 294
15.3.1 Type of Design 295
15.3.2 Structural and Incidental Parameters 295
15.3.3 Estimation Methods 296
15.4 Fully Bayesian Estimation (Gibbs Sampler) 297
15.4.1 Data Augmentation 297
15.4.2 Posterior Distribution 298
15.4.3 Steps in the Gibbs Sampler 298
15.4.4 Identifiability Problems 301
15.5 Bayes Modal Estimation (Nip=1) 301
15.6 Bayes Modal Estimation (Nip2) 302
15.6.1 Discussion 304
15.7 Some Numerical Examples 304
15.8 Final Remarks 309
References 313
16 Detecting Person Misfit in Adaptive Testing 315
16.1 Introduction 315
16.2 Practical and Theoretical Relevance of Person Fit Analysis in a CAT 316
16.3 Review of Existing Literature 318
16.3.1 Person Fit in Paper-and-Pencil Testing 318
16.3.2 Person Fit in Computerized Adaptive Testing 319
16.3.3 Person Fit Statistics Designed for a CAT 320
16.4 Discussion 325
References 327
17 The Investigation of Differential Item Functioning in Adaptive Tests 330
17.1 Introduction 330
17.2 Methods for Assessing DIF in CATs 331
17.2.1 A Review of the Mantel–Haenszel, Standardization, and SIBTEST Procedures 332
17.2.2 A Modification of the MH and Standardization Approaches for CATs (ZTW) 335
17.2.3 Correlations among CAT DIF Statistics, Nonadaptive DIF Statistics, and Generating DIF 336
17.2.4 An Empirical Bayes (EB) Enhancement of the MH Approach (ZTL) 338
17.2.5 LSAT Simulation Study 339
17.2.6 Properties of EB DIF Estimates 340
17.2.7 RMSRs of EB and MH Point Estimatesin the No-DIF Case 341
17.2.8 RMSRs of EB and MH Point Estimatesin the DIF-present Case 342
17.2.9 Bias of EB and MH Point Estimates in the DIF Case. 343
17.2.10 Probabilistic Classification of DIF Results 344
17.2.11 CATSIB: A Modification of the SIBTEST Method of Shealy and Stout 345
17.3 Recent Developments 347
17.4 Future Research 348
References 349
Part V Multistage and Mastery Testing 352
18 Multistage Testing: Issues, Designs, and Research 353
18.1 Introduction 353
18.2 Fundamentals of the MST Design 354
18.3 Structuring a Multistage Test 355
18.4 Automated Test Assembly 360
18.5 Comparative Studies of MST 363
18.6 Evaluating MST Relative to Other Test Designs 363
18.7 Modules with Common-Stem Items 365
18.8 Conclusions 366
References 367
19 Three-Category Adaptive Classification Testing 371
19.1 Introduction 371
19.2 Overview of Approaches to Classification Testing 372
19.3 Basic Elements of Adaptive Testing 373
19.4 The SPRT in CAT 375
19.4.1 Classification in Two Categories 375
19.4.2 Classification in Three Categories 376
19.4.3 Evaluation of the Test Statistics 378
19.5 Item Selection in CAT with the SPRT 379
19.5.1 Kullback-Leibler Information 379
19.5.2 K-L Information in the Three-Category Problem 380
19.6 Performance of Three-Category Classification CAT with the SPRT 381
19.6.1 Simulation Example 382
19.6.2 Results 383
19.7 Concluding Remarks 384
References 384
20 Testlet-Based Adaptive Mastery Testing 386
20.1 Introduction 386
20.2 Earlier Approaches to the Variable-Length Mastery Problem 387
20.2.1 Contributions of SPRT to Variable-Length Mastery Testing 388
20.2.2 IRT-Based Item Selection Strategies Applied to Adaptive Mastery Testing 388
20.2.3 Sequential Mastery Testing Based on Bayesian Decision Theory 388
20.3 Bayesian Sequential Decision Theory Applied to Adaptive Mastery Testing 389
20.3.1 Formalization of the Variable-Length Mastery Problem 390
20.3.2 Linear Loss 390
20.3.3 The Rasch Model 392
20.3.4 The 3PL Model and the 3PL Testlet Model 394
20.3.5 Adaptive Sequential Mastery Testing 396
20.4 Performance of Sequential and Adaptive Sequential Mastery Testing 397
20.4.1 The 1PL Model 397
20.4.2 The 3PL Model and the 3PL Testlet Model 399
20.5 Discussion 401
References 402
21 Adaptive Mastery Testing Using a Multidimensional IRT Model 405
21.1 Introduction 405
21.2 Definition of the Decision Problem 406
21.3 Multidimensional IRT Models 406
21.4 Compensatory and Conjunctive-Disjunctive Loss Functions 407
21.4.1 Compensatory Loss Functions 408
21.4.2 Conjunctive Loss Functions 410
21.5 Computation of Expected Loss and Risk Using Backward Induction 412
21.6 The Compound Multidimensional Rasch Model 413
21.7 Simulation Studies 416
21.7.1 Compensatory Loss Functions 416
21.7.2 Conjunctive Loss Functions 420
21.8 An Empirical Example 421
21.9 Conclusions and Further Research 422
References 425
Index 428