Bayesian Item Response Modeling (eBook)

Jean-Paul Fox is Associate Professor of Measurement and Data Analysis, University of Twente, The Netherlands. His main research activities are in several areas of Bayesian response modeling. Dr. Fox has published numerous articles in the areas of Bayesian item response analysis, statistical methods for analyzing multivariate categorical response data, and nonlinear mixed effects models.

Preface 8
Contents 12
1 Introduction to Bayesian Response Modeling 16
1.1 Introduction 16
1.1.1 Item Response Data Structures 18
Hierarchically Structured Data 18
1.1.2 Latent Variables 20
1.2 Traditional Item Response Models 21
1.2.1 Binary Item Response Models 22
The Rasch Model 22
Two-Parameter Model 24
Three-Parameter Model 26
1.2.2 Polytomous Item Response Models 27
1.2.3 Multidimensional Item Response Models 29
1.3 The Bayesian Approach 30
1.3.1 Bayes' Theorem 31
Constructing the Posterior 33
Updating the Posterior 33
1.3.2 Posterior Inference 35
The Role of Prior Information 30
1.4 A Motivating Example Using WinBUGS 36
1.4.1 Modeling Examinees' Test Results 36
WinBUGS 37
1.5 Computation and Software 39
Computer Code Developed for This Book 41
1.6 Exercises 42
2 Bayesian Hierarchical Response Modeling 45
2.1 Pooling Strength 45
2.2 From Beliefs to Prior Distributions 47
A Hierarchical Prior for Item Parameters 48
A Hierarchical Prior for Person Parameters 52
2.2.1 Improper Priors 52
2.2.2 A Hierarchical Bayes Response Model 53
Posterior Computation 55
2.3 Further Reading 56
2.4 Exercises 57
3 Basic Elements of Bayesian Statistics 59
3.1 Bayesian Computational Methods 59
3.1.1 Markov Chain Monte Carlo Methods 60
Gibbs Sampling 60
Metropolis-Hastings 61
Issues in MCMC 62
Single Chain Analysis 63
Multiple Chain Analysis 64
3.2 Bayesian Hypothesis Testing 65
3.2.1 Computing the Bayes Factor 68
Importance Sampling 69
Using Identities and MCMC Output 70
Bayes Factor for Item Response Models 71
3.2.2 HPD Region Testing 72
3.2.3 Bayesian Model Choice 73
3.3 Discussion and Further Reading 75
3.4 Exercises 76
4 Estimation of Bayesian Item Response Models 81
4.1 Marginal Estimation and Integrals 81
4.2 MCMC Estimation 85
4.3 Exploiting Data Augmentation Techniques 87
4.3.1 Latent Variables and Latent Responses 88
4.3.2 Binary Data Augmentation 89
4.3.3 TIMMS 2007: Dutch Sixth-Graders' Math Achievement 95
4.3.4 Ordinal Data Augmentation 97
4.4 Identification of Item Response Models 100
4.4.1 Data Augmentation and Identifying Assumptions 101
4.4.2 Rescaling and Priors with Identifying Restrictions 102
4.5 Performance MCMC Schemes 103
4.5.1 Item Parameter Recovery 103
4.5.2 Hierarchical Priors and Shrinkage 106
4.6 European Social Survey: Measuring Political Interest 109
4.7 Discussion and Further Reading 112
4.8 Exercises 113
5 Assessment of Bayesian Item Response Models 121
5.1 Bayesian Model Investigation 121
5.2 Bayesian Residual Analysis 122
5.2.1 Bayesian Latent Residuals 123
5.2.2 Computation of Bayesian Latent Residuals 123
5.2.3 Detection of Outliers 124
5.2.4 Residual Analysis: Dutch Primary School Mathematics Test 125
5.3 HPD Region Testing and Bayesian Residuals 126
5.3.1 Measuring Alcohol Dependence: Graded Response Analysis 130
Item and Person Fit 126
Detecting Discriminating Items 128
5.4 Predictive Assessment 131
5.4.1 Prior Predictive Assessment 133
5.4.2 Posterior Predictive Assessment 136
Overview of Posterior Predictive Model Checks 138
5.5 Illustrations of Predictive Assessment 140
5.5.1 The Observed Score Distribution 140
5.5.2 Detecting Testlet E ects 141
A Short Introduction to Testlet E ects 141
Simulation Study 142
5.6 Model Comparison and Information Criteria 144
5.6.1 Dutch Math Data: Model Comparison 145
5.7 Summary and Conclusions 145
5.8 Exercises 147
5.9 Appendix: CAPS Questionnaire 153
6 Multilevel Item Response Theory Models 154
6.1 Introduction: School Effectiveness Research 154
6.2 Nonlinear Mixed Effects Models 155
6.3 The Multilevel IRT Model 158
6.3.1 A Structural Multilevel Model 158
6.3.2 The Synthesis of IRT and Structural Multilevel Models 161
6.4 Estimating Level-3 Residuals: School Effects 166
6.5 Simultaneous Parameter Estimation of MLIRT 171
6.6 Applications of MLIRT Modeling 175
6.6.1 Dutch Primary School Mathematics Test 175
6.6.2 PISA 2003: Dutch Math Data 178
Multiple Imputation 178
6.6.3 School Effects in the West Bank: Covariate Error 185
School Leadership and Math Achievement 185
6.6.4 MMSE: Individual Trajectories of Cognitive Impairment 187
Mixture MLIRT Modeling 189
Identifiability 191
OPTIMA Cohort: Modeling Individual Trajectories 191
6.7 Summary and Further Reading 194
6.8 Exercises 196
6.9 Appendix: The Expected School Effect 201
6.10 Appendix: Likelihood MLIRT Model 203
7 Random Item Effects Models 205
7.1 Random Item Parameters 205
7.1.1 Measurement Invariance 206
7.1.2 Random Item Effects Prior 207
7.2 A Random Item Effects Response Model 210
Binary Response Data 210
Polytomous Response Data 212
7.2.1 Handling the Clustering of Respondents 215
7.2.2 Explaining Cross-national Variation 215
7.2.3 The Likelihood for the Random Item Effects Model 216
7.3 Identification: Linkage Between Countries 217
7.3.1 Identification Without (Designated) Anchor Items 218
7.3.2 Concluding Remarks 220
7.4 MCMC: Handling Order Restrictions 221
7.4.1 Sampling Threshold Values via an M-H Algorithm 221
Nation-Specific Threshold Parameters 221
International Threshold Parameters 222
7.4.2 Sampling Threshold Values via Gibbs Sampling 223
7.4.3 Simultaneous Estimation via MCMC 224
7.5 Tests for Invariance 226
7.6 International Comparisons of Student Achievement 228
7.7 Discussion 233
7.8 Exercises 234
8 Response Time Item Response Models 238
8.1 Mixed Multivariate Response Data 238
8.2 Measurement Models for Ability and Speed 239
8.3 Joint Modeling of Responses and Response Times 242
8.3.1 A Structural Multivariate Multilevel Model 243
8.3.2 The RTIRT Likelihood Model 245
8.4 RTIRT Model Prior Speci cations 246
8.4.1 Multivariate Prior Model for the Item Parameters 246
8.4.2 Prior for P with Identifying Restrictions 247
8.5 Exploring the Multivariate Normal Structure 249
8.6 Model Selection Using the DIC 252
8.7 Model Fit via Residual Analysis 253
8.8 Simultaneous Estimation of RTIRT 254
8.9 Natural World Assessment Test 257
8.10 Discussion 259
8.11 Exercises 261
8.12 Appendix: DIC RTIRT Model 265
9 Randomized Item Response Models 266
9.1 Surveys about Sensitive Topics 266
9.2 The Randomized Response Technique 267
9.2.1 Related and Unrelated Randomized Response Designs 268
9.3 Extending Randomized Response Models 269
9.4 A Mixed Effects Randomized Item Response Model 270
9.4.1 Individual Response Probabilities 270
Univariate Response Data 270
Multivariate Response Data 271
9.4.2 A Structural Mixed Effects Model 272
9.5 Inferences from Randomized Item Response Data 273
9.5.1 MCMC Estimation 276
9.5.2 Detecting Noncompliance Behavior 278
9.5.3 Testing for Fixed-Group Differences 279
9.5.4 Model Choice and Fit 281
9.6 Simulation Study 283
9.6.1 Different Randomized Response Sampling Designs 283
9.6.2 Varying Randomized Response Design Properties 285
9.7 Cheating Behavior and Alcohol Dependence 286
9.7.1 Cheating Behavior at a Dutch University 286
Differences in Attitudes 287
Item Level Analysis 288
9.7.2 College Alcohol Problem Scale 290
Fixed Versus Random Effects 292
Explaining Item Response Variation 293
9.8 Discussion 295
9.9 Exercises 296
References 300
Index 319

"8 Response Time Item Response Models (p. 227-228)

Response times and responses can be collected via computer adaptive testing or computer-assisted questioning. Inferences about test takers and test items can therefore be based on the response time and response accuracy information. Response times and responses are used to measure a respondents speed of working and ability using a multivariate hierarchical item response model. A multivariate multilevel structural population model is de ned for the person parameters to explain individual and group di erences given background information. An application is presented that illustrates novel features of the model.

8.1 Mixed Multivariate Response Data

Nowadays, response times (RTs) are easily collected via computer adaptive testing or computer-assisted questioning. The RTs can be a valuable source of information on test takers and test items. The RT information can help to improve routine operations in testing such as item calibration, test design, detection of cheating, and adaptive item selection. The collection of multiple item responses and RTs leads to a set of mixed multivariate response data since the individual item responses are often observed on an ordinal scale, whereas the RTs are observed on a continuous scale.

The observed responses are imperfect indicators of a respondents ability. When measuring a construct such as ability, attention is focused on the accuracy of the test results. The observed RTs are indicators of a respondents speed of working, and speed is considered to be a di erent construct. As a result, mixed responses are used to measure the two constructs ability and speed. Although response speed and response accuracy measure di erent con- structs (Schnipke and Scrams, 2002, and references therein), the reaction-time research in psychology indicates that there is a relationship between response speed and response accuracy (Luce, 1986).

This relationship is often characterized as a speed{accuracy trade-o . A person can decide to work faster, but this will lead to a lower accuracy. The trade-o is considered to be a withinperson relationship: a respondent controls the speed of working and accepts the related level of accuracy. It will be assumed that each respondent chooses a xed level of speed, which is related to a xed accuracy. A hierarchical measurement model was proposed by van der Linden (2007) to model RTs and dichotomous responses simultaneously that accounts for di erent levels of dependency.

The di erent stages of the model capture the dependency structure of observations nested within persons at the observational level and the relationship between speed and ability at the individual level. Klein Entink, Fox and van der Linden (2009a), and Fox, Klein Entink and van der Linden (2007) extended the model for measuring accuracy and speed (1) to allow time-discriminating items, (2) to handle individual and/or group characteristics, and (3) to handle the nesting of individuals in groups.

This extension has a multivariate multilevel structural population model for the ability and the speed parameters that can be considered a multivariate extension of the structural part of the MLIRT model of Chapter 6. In this chapter, the complete modeling framework will be discussed, and an extension is made to handle polytomous response data."