Psychometrics (eBook)

Front cover 1
Title page 4
Copyright page 5
Preface 6
Table of contents 8
Contributors 18
Ch. 1. A History and Overview of Psychometrics 22
1. Introduction 22
2. The origins of psychometrics (circa 1800-1960) 22
3. Psychological scaling 29
4. Psychological measurement (test theory) 31
5. Factor analysis 36
6. Psychological statistics 39
7. Conclusion 42
References 43
Ch. 2. Selected Topics in Classical Test Theory 50
1. A modern introduction 52
2. Restriction of range 57
3. Nonlinear classical test theory 60
References 63
Ch. 3. Validity: Foundational Issues and Statistical Methodology 66
1. Introductory remarks 66
2. Foundational issues 68
3. Statistical methods 86
4. Closing remarks 92
References 97
Ch. 4. Reliability Coefficients and Generalizability Theory 102
1. Introduction 102
2. Reliability Coefficients in Classical Test Theory 102
3. Generalizability theory 114
4. Concluding remarks 141
References 142
Ch. 5. Differential Item Functioning and Item Bias 146
1. Introduction 146
2. General definition of DIF 148
3. Item response theory approaches for dichotomous items 149
4. Proportion-difference approaches for dichotomous items 153
5. Common odds ratio approaches for dichotomous items 156
6. Logistic regression approaches for dichotomous items 160
7. Classification schemes for dichotomous items 162
8. Item response theory approaches for polytomous items 164
9. Mean-difference approaches for polytomous items 167
10. Multivariate hypergeometric distribution approaches for polytomous items 168
11. Common odds ratio approaches for polytomous items 170
12. Logistic regression approaches for polytomous items 171
13. Differential test functioning and DIF effect variance 173
14. Explaining the sources of DIF 176
15. Steps to conducting DIF analyses: An applied example 178
16. Cautions and limitations 180
References 184
Ch. 6. Equating Test Scores 190
1. Introduction and definitions 190
2. Data collection designs used in test score equating 195
3. Procedures for equating scores 204
4. Best practices and challenges to best practices 218
References 222
Ch. 7. Electronic Essay Grading 226
1. Regression analysis 227
2. Composite scales 236
3. Content analysis 239
4. Analysis of discrete responses 247
5. Bayesian analysis 251
6. Conclusions 252
References 253
Ch. 8. Some Matrix Results Useful in Psychometric Research 256
1. Notation and Basic Matrix Results 256
2. Decomposition of matrices 260
3. Matrix approximations 264
4. Procrustean transformation 268
5. Correspondence analysis 271
6. Metric and multidimensional scaling 273
Acknowledgements 275
References 275
Ch. 9. Factor Analysis 278
1. Introduction 278
2. Historical development of factor analysis 278
3. The factor analysis model 282
4. Statistical inference in factor analysis 292
5. Factor rotation and estimation of factor scores 302
Acknowledgements 314
References 314
Ch. 10. Structural Equation Modeling 318
1. Introduction 318
2. Model identification 324
3. Estimation and evaluation 326
4. Missing data 348
5. Multiple groups 355
6. Multilevel models with hierarchical data 359
7. Examples 365
References 369
Ch. 11. Applications of Multidimensional Scaling in Psychometrics 380
1. Introduction 380
2. MDS with the simple Euclidean model 382
3. Individual differences MDS 396
4. Unfolding analysis 406
5. Concluding remarks 416
References 418
Ch. 12. Multilevel Models in Psychometrics 422
1. Introduction 422
2. Basic Models for Two-level Hierarchical Data Structures 423
3. Models for repeated measures 427
4. Models for multivariate response data 430
5. Models for non-hierarchical structures 433
6. Further extensions 437
7. Estimation procedures and software 438
8. Resources 439
References 439
Ch. 13. Latent Class Analysis in Psychometrics 442
1. Introduction 442
2. The model for LCA 443
3. Estimation and model fit 444
4. Analyses with score data 449
5. Unconstrained latent class models 453
6. Scaling models 454
7. Models incorporating grouping of respondents 457
8. Covariate latent class models 459
9. Other models 462
References 465
Ch. 14. Random-Effects Models for Preference Data 468
1. Introduction 468
2. Thurstonian random utility models 470
3. Identifiability 476
4. Identifying the scale origin 479
5. Estimation 480
6. Applications 481
7. Concluding remarks 487
References 487
Ch. 15. Item Response Theory in a General Framework 490
1. Introduction 490
2. The general IRT framework 491
3. Item response models 493
4. Estimation of item and group parameters 511
5. Estimation of respondent scores 520
6. Item factor analysis 526
7. Response relations with external variables 529
References 530
Ch. 16. Rasch Models 536
1. Some history of the Rasch model 536
2. Some basic concepts and properties of the RM 538
3. Characterizations and scale properties of the RM 543
4. Item parameter estimation 553
5. Person parameter estimation 567
6. Testing of fit 570
7. The linear logistic test model 585
8. Longitudinal linear logistic models 592
9. Some remarks on applications and extensions of the RM 597
References 599
Ch. 17. Hierarchical Item Response Theory Models 608
1. Introduction 608
2. Developing the hierarchical IRT model 610
3. Estimation 613
4. Examples 616
5. Conclusions 625
References 625
Ch. 18. Multidimensional Item Response Theory 628
1. Introduction 628
2. General forms of MIRT models 630
3. Common forms of MIRT models 633
4. Descriptions of item characteristics 640
5. Descriptions of test characteristics 646
6. The estimation of model parameters 650
7. Applications 652
8. Discussion and conclusions 661
References 662
Ch. 19. Mixture Distribution Item Response Models 664
1. Introduction 664
2. Different perspectives on mixture IRT 665
3. Applications of mixture IRT models 676
4. Mixture IRT for large-scale survey assessments 677
5. Conclusion 679
References 680
Ch. 20. Scoring Open Ended Questions 684
1. Introduction 684
2. Motivating example 684
3. Theory 686
4. A measurement model for scoring open ended questions 694
5. Motivating example revisited 698
6. Discussion 700
References 701
Ch. 21. Assessing the Fit of Item Response Theory Models 704
1. Introduction: Models and assumptions 704
2. Checking the assumption of unidimensionality 707
3. Assessing model data fit: Checking model predictions 712
4. Empirical example 728
5. Conclusions 735
Acknowledgements 736
References 736
Ch. 22. Nonparametric Item Response Theory and Special Topics 740
1. Place of nonparametric models in item response theory 740
2. Analyzing test data using nonparametric IRT 748
3. Special topics in NIRT 760
References 764
Ch. 23. Automatic Item Generation and Cognitive Psychology 768
1. Introduction 768
2. Item development and automatic item generation 769
3. Item response models for automatic item generation 774
4. Calibration of the cognitive IRT models 777
5. An application: Cognitive models for algorithmically generated spatial ability items 782
6. Overall discussion 786
References 787
Ch. 24. Statistical Inference for Causal Effects, with Emphasis on Applications in Psychometrics and Education 790
1. Causal inference primitives 790
2. The assignment mechanism 797
3. Randomization-based modes of causal inference 802
4. Posterior predictive causal inference 807
5. Complications 815
References 817
Ch. 25. Statistical Aspects of Adaptive Testing 822
1. Introduction 822
2. Response models 823
3. Item calibration 825
4. Ability estimation 832
5. Empirical examples 833
6. Rules for adaptive item selection 835
7. Other statistical issues in adaptive testing 843
8. Concluding comment 854
References 856
Ch. 26. Bayesian Psychometric Modeling From An Evidence-Centered Design Perspective 860
1. Introduction and overview 860
2. An evidentiary perspective 861
3. Evidence-centered design 868
4. Bayesian psychometric modeling 877
5. Discussion 882
6. Concluding remarks 884
References 884
Ch. 27. Value-Added Modeling 888
1. Introduction 888
2. General issues 890
3. Models for estimating teacher effects 896
4. Example 901
5. Current research 905
6. Summing up 909
References 910
Ch. 28. Three Statistical Paradoxes in the Interpretation of Group Differences: Illustrated with Medical School Admission and Licensing Data 914
1. Introduction 914
2. The data 915
3. Simpson's Paradox 916
4. Kelley's Paradox 922
5. Lord's Paradox 931
6. Conclusion 937
Acknowledgements 938
References 939
Ch. 29. Meta-Analysis 940
1. Introduction 940
2. Effect sizes 941
3. The problem of dependent estimates 944
4. Fixed effects analyses 945
5. Mixed models 963
6. Conclusions 973
References 974
Ch. 30. Vertical Scaling: Statistical Models for Measuring Growth and Achievement 976
1. Introduction 976
2.Vertical scaling of achievement tests: Recent and past statistical practice 977
3. Statistical modeling approaches for vertical scaling 981
4. Exploring grade-to-grade growth in achievement: An application of hierarchical modeling 985
5. Discussion 993
References 994
Ch. 31. Cognitive Diagnosis 998
Ch. 31A. Review of Cognitively Diagnostic Assessment and a Summary of Psychometric Models 1000
1. Preliminaries 1000
2. Implementation framework for diagnostic assessment 1001
3. Reliability, validity, and granularity 1014
4. Summary of cognitively diagnostic psychometric models 1016
5. Summary and conclusions: Issues and future directions 1044
References 1048
Ch. 31B. Some Notes on Models for Cognitively Based Skills Diagnosis 1052
Introduction 1052
1. Retrofitting a test with a hypothesized skill structure 1054
2. Dimensionality 1055
3. Utility and use of skills classifications 1056
4. Latent structure models, latent responses, conjunctive versus compensatory skills 1057
5. Outlook 1058
References 1059
Ch. 32. The Statistical Procedures Used in National Assessment of Educational Progress: Recent Developments and Future Directions 1060
1. Introduction 1060
2. The current NAEP model and estimation method 1062
3. Example: NAEP data and results 1068
4. Alternatives approaches 1070
Conclusions 1073
Acknowledgements 1074
Appendix: Sampling students in NAEP 1074
References 1075
Ch 33. Statistical Procedures Used in College Admissions Testing 1078
1. Introduction 1078
2. Test design and assembly 1080
3. Item pretesting 1085
4. Item analysis 1087
5. Scaling and equating 1095
6. Reliability and validity 1098
7. Other issues 1107
8. Summary 1109
References 1110
Ch. 34. Future Challenges in Psychometrics 1114
Ch. 34A. Integration of Models 1116
What constitutes a replication? 1116
What are true scores? 1117
What is error? 1118
No ``right'' model 1118
References 1118
Ch. 34B. Linking Scores Across Computer and Paper-Based Modes of Test Administration 1120
References 1123
Ch. 34C. Linking Cognitively-Based Models and Psychometric Methods 1124
Introduction 1124
Developing cognitive models of task performance 1125
Incorporating cognitive models into psychometric methods 1126
Conclusion 1127
References 1127
Ch. 34D. Technical Considerations in Equating Complex Assessments 1128
References 1130
Ch. 34E. Future Challenges to Psychometrics: Validity, Validity, Validity 1132
References 1133
Ch. 34F. Testing with and without Computers 1134
Ch. 34G. Practical Challenges to Psychometrics Driven by Increased Visibility of Assessment 1138
Growth models 1138
Diagnosis, prescription, and instruction 1139
Dual-platform testing 1139
Subject Index 1142
Handbook of Statistics Contents of Previous Volumes 1164