User-Centered Assessment Design

Madhabi Chatterji, PhD, is Professor Emerita of Measurement, Evaluation, and Education at Teachers College, Columbia University, where she founded and directs the Assessment and Evaluation Research Initiative (AERI). AERI is dedicated to promoting meaningful use of assessment and evaluation information to improve equity and the quality of practices and policies in education, psychology, and the health professions. An award-winning, internationally recognized methodologist and educationist, Dr. Chatterji has taught and mentored numerous doctoral students and postdoctoral researchers over her 30-plus-year career. She is author or editor of more than 100 publications and is a Fellow of the National Education Policy Center. A public intellectual, Dr. Chatterji has spoken out frequently on the limitations of large-scale tests and the adverse social consequences of misused high-stakes educational assessments. Her longstanding scholarly interests lie in instrument design, validation, validity, and test use issues; improving program and policy evaluation designs to support evidence-based practices; and closing learning gaps with proximal diagnostic assessments.

I. Foundations
1. Foundational Concepts in Assessment Design
1.1 Chapter Overview
1.2 Assessments: Old and Emerging Traditions, a Starting Definition, and Some Distinctions
1.3 Viewpoints on Assessment, Measurement, Testing, and Evaluation
1.4 Role of Assessment in Scientific, Professional, and Practical Endeavors
1.5 Evaluating the Quality of Assessments and Construct Measures: Validity, Reliability, and Utility
1.6 Integrating Assessment Design, Validation, and Use: A User-Centered Process
1.7 Summary
2. Why Assess?: Measure-Based Inferences, Uses, Users, and Consequences
2.1 Chapter Overview
2.2 Back to the Future: Early Drivers, Milestones, and Consequences of Assessment Use
2.3 Modern Drivers and Consequences of Assessment Uses in Education
2.4 Modern Drivers and Consequences of Assessment Uses in Psychology, Health, Business, and Other Fields
2.5 Applying User-Centered Principles to Improve Practices
2.6 Summary
3. Whom to Assess? and How?: Specifying the Population and the Assessment Operations
3.1 Chapter Overview
3.2 Why Population Characteristics and the Socioecological Contexts of Assessments Matter
3.3 What Is Measurement Bias?: Case Studies and Hypothetical Illustrations
3.4 Selecting Assessment Operations for Diverse Populations and Multidisciplinary Constructs
3.5 Steps and Actions: Specifying Whom to Assess? and How? with the Process Model
3.6 Summary
II. Assessment Design
4. What to Assess?: Specifying the Domains for Constructs
4.1 Chapter Overview
4.2 Domain Sampling and Domain Specification: Functional Theory and Applied Illustrations
4.3 Construct Types, Domain Conceptualizations, and Structures
4.4 Domain Specification as a Part of the Process Model: Steps, Techniques, Guidelines, and Conventions
4.5 Content-Validating Specified Domains
4.6. Summary
5. Designing Assessments with Structured and Constructed-Response Items
5.1 Chapter Overview
5.2 Why the Mechanics of Item Construction Matter
5.3 Cognitive Constructs Measured Best with Structured- or Constructed-Response Items
5.4 Writing Structured-Response Items: Principles, Guidelines, and Applied Examples
5.5 Guidelines for Designing Constructed-Response and Essay Tasks
5.6. Instrument Assembly
5.7 An Application with the Process Model: A Case Study of Cognitively Based Item and Assessment Design to Foster Learning in Long Division
5.8 Summary
6. Designing Behavior-Based, Product-Based, and Portfolio-Based Assessments
6.1 Chapter Overview
6.2 Behavior-, Product-, and Portfolio-Based Assessments: Definitions, Examples, and Origins
6.3 Advantages of the Performance Assessment Format
6.4 Disadvantages of Performance Assessments: Human Vulnerabilities, Errors, and Biases
6.5 Three Case Studies: Applying the Process Model to Design and Validate Performance Assessments
6.6 Summary
7. Designing Survey-Based and Interview-Based Assessment Tools
7.1 Chapter Overview
7.2 Self-Report Instruments: Their Defining Properties and Common Applications
7.3 Historical Origins of Questionnaires and Attitude Surveys
7.4 Measurement Issues with the Self-Report Modality
7.5 General Design Guidelines for Self-Report Instruments
7.6 Ten More Guidelines for Writing Closed-Ended Survey Items
7.7 A Case Study: Applying the Process Model to Design Two Complementary Self-Report Tools
7.8 Summary
III. Validation and Use of Assessments
8. Analyzing Data from Assessments: A Statistics Refresher
8.1 Chapter Overview
8.2 Preparing for Data Analysis
8.3 Organizing the Data
8.4 Measures of Central Tendency
8.5 Measures of Variability
8.6 Graphical Displays of Data
8.7 The Standard Normal Distribution and Its Applications
8.8 Correlation Coefficients and Their Applications
8.9 Related Statistical Techniques
8.10 Summary
9. Improving the Inferential Utility of Assessment Results: Methods and Limitations
9.1 Chapter Overview
9.2 Frames of Reference and Derived Scores
9.3 Using Norms as the Frame of Reference
9.4 Using Criterion Scores or Standards as the Frame of Reference
9.5 Using Self as the Frame of Reference
9.6 Composite Scores
9.7 Grouped Scores, Equated Scales, and Linked Tests
9.8 Summary
10. A Unified Approach to Construct Validity and Validation: Theory to Evidence
10.1 Chapter Overview
10.2 Construct Validity: An Evolving Concept
10.3 Theoretical Foundations of the Unitarian View of Validation
10.4 Main Clusters and Types of Validity Evidence
10.5 Random Errors of Measurement and Types of Reliability Evidence
10.6 Utility of Measures, Assessments, and Assessment Systems
10.7 Unified Validation Plans
10.8 Chapter Summary
11. Empirical Methods of Validation
11.1 Chapter Overview
11.2 Planning Empirical Validation Studies
11.3 Evaluating Item Performance
11.4 Examining Fairness and Measurement Bias
11.5 Gathering Evidence of Content-Based Validity
11.6 Validating Response Processes: The Cognitive Interview
11.7 Gathering Correlational Evidence of Validity
11.8 Empirical Estimation of Reliability
11.9 Methods to Examine Utility
11.10 Evaluating the Evidence: The PSQI Case Revisited
11.11 Summary
12. User-Centered Assessment Design: Revisiting the Principles, Comparisons, and Conclusions
12.1 Chapter Overview
12.2 Applying the Principles Undergirding the Process Model: A Summary by Section
12.3. A User-Centered Design Process: Comparing the Old with the New
12.4 Extended Applications of the Process Model
12.5 The Process Model Compared to Existing Models of Assessment Design
12.6 Connecting the Process Model with the 2014 Standards
12.7 Summary
Glossary
References
Author Index
Subject Index
About the Author