Handbook of Accessible Achievement Tests for All Students (eBook)

Stephen N. Elliott, PhD is the founding Director of the Learning Sciences Institute, a trans-university research enterprise at Arizona State University, and is the Mickelson Foundation Professor of Education. He received his doctorate at Arizona State University in 1980 and has been on the faculty at several major research universities, including the University of Wisconsin-Madison and Vanderbilt University. At Wisconsin (1987-2004), Steve was a professor of educational psychology and served as the Associate Director of the Wisconsin Center for Education Research. At Vanderbilt (2004-2010), he was the Dunn Family Professor of Educational and Psychological Assessment in the Special Education Department and directed the Learning Sciences Institute and Dunn Family Scholars Program. His research focuses on scale development and educational assessment practices. In particular, he has published articles on (a) the assessment of children's social skills and academic competence, (b) the use of testing accommodations and alternate assessment methods for evaluating the academic performance of students with disabilities for educational accountability, and (c) students’ opportunities to learn the intended curriculum. Steve's scholarly and professional contributions have been recognized by his colleagues in education and psychology research as evidenced by being selected as an American Psychological Association Senior Scientist in 2009. Steve consults with state assessment leaders on the assessment and instruction of PreK-12 students, and serves on ETS’s Visiting Research Panel, and is the Director of Research and Scientific Practice for the Society of the Study of School Psychology. Ryan J. Kettler, PhD is a Research Assistant Professor in Special Education at Peabody College of Vanderbilt University. He received his doctorate in Educational Psychology, with a specialization in School Psychology, from the University of Wisconsin-Madison in 2005. Ryan’s dissertation, Identifying students who need help early: Validation of the Brief Academic Competence Evaluation Screening System, won the 2006 Outstanding Dissertation award from the Wisconsin School Psychologists Association. In 2007, he was named an Early Career Scholar by the Society for the Study of School Psychology. Prior to joining Vanderbilt University, Ryan was an assistant professor at California State University, Los Angeles, and completed an APA-accredited internship at Ethan Allen School in Wales, Wisconsin. He has worked on multiple federally funded grants examining the effectiveness of alternate assessments, academic and behavioral screening systems, and testing accommodations. Ryan is the author of peer reviewed publications and presentations within the broader area of data-based assessment for intervention, representing specific interests in academic and behavioral screening, inclusive assessment, reliability and validity issues, and rating scale technology. He currently serves as a consultant to College Board and to the Wisconsin Center for Education Research, providing expertise in the area of inclusive assessment.Peter A. Beddow, PhD received his doctorate in Special Education and Educational Psychology at Vanderbilt University in 2011. His research focuses on test accessibility and item-writing for assessments of student achievement. He is the senior author of the Test Accessibility and Modification Inventory (TAMI) and the Accessibility Rating Matrix, a set of tools for evaluating the accessibility of test items for learners with a broad range of abilities and needs. Based on his work on accessibility theory, Peter was awarded the Bonsal Education Research Entrepreneurship Award in 2009 and the Melvyn R. Semmel Dissertation Research Award in 2010. Prior to beginning his academic career, Peter taught for seven years in Los Angeles County, including five years teaching Special Education for students with emotional and behavior problems at Five Acres School, part of a residential treatment facility for children who are wards-of-the-court for reasons of abuse and neglect. Peter’s primary goal is to help children realize their infinite value and achieve their ultimate potential. Pete lives in Nashville, Tennessee.Alexander Kurz, MEd is a doctoral student in Special Education and the Interdisciplinary Program in Educational Psychology at Vanderbilt University. He has studied in Germany and the U.S. earning degrees in Special Education and Philosophy. Upon moving to the U.S., he worked as a special education teacher in Tennessee and California, designed and implemented curricula for reading intervention classes, and participated in school reform activities through the Bill and Melinda Gates Foundation. Prior to beginning his doctoral studies, Alex worked as behavior analyst for children with autism and as an educational consultant to Discovery Education Assessment. During his graduate work at Vanderbilt, he collaborated with the Wisconsin Center for Education Research and Discovery Education Assessment leading research efforts to examine curricular alignment and its relation to student achievement for students with and without disabilities. Alex has coauthored several peer-reviewed publications on alignment and alternate assessment. His latest scholarly contributions have reexamined the concepts of opportunity-to-learn (OTL), alignment, and access to the general curriculum in the context of curricular frameworks for general and special education. Alex is the senior author of My Instructional Learning Opportunities Guidance System, a teacher-oriented OTL measurement tool. His current interest in educational technology and innovation is aimed at identifying and creating pragmatic solutions to the problems of practice.

Preface 6
Contents 7
Contributors 9
About the Editors 11
1 Creating Access to Instruction and Tests of Achievement: Challenges and Solutions 13
Legislative Context and Key Concepts 14
Providing Access to Overcome Barriers 17
Access via Opportunity to Learn 17
Access via Testing Accommodations 20
Access via Well-Designed Test Items 22
Actions and Innovations Needed to Keep Moving Forward 23
Conclusions 24
References 25
Part I Government Policies and Legal Considerations 29
2 U.S. Policies Supporting Inclusive Assessments for Students with Disabilities 30
U.S. Policies Supporting Inclusive Assessment of Students with Disabilities 30
Assessment Policies in the 1960s and 1970s: Inclusion and 'Equal Terms' 30
The 1980s and 1990s: IEP as Curriculum 31
IDEA 97 and Options for Alternate Assessment 35
2001 No Child Left Behind Act 37
2002--2003 Title I Regulations Permitting Alternate Achievement Standards in Accountability 38
IDEA 2004 and Assessments Measuring Responsiveness to Intervention 39
2007 Joint Title I IDEA Regulations Permitting Modified Academic Achievement Standards in Accountability 40
'Race To The Top' Assessment Initiatives 41
References 41
3 U.S. Legal Issues in Educational Testing of Special Populations 44
Introduction 44
Nonstandard Test Administrations 44
Federal Legislation 46
Section 504 of the Rehabilitation Act 46
Americans with Disabilities Act (ADA) 47
Individuals with Disabilities Education Act (IDEA) 47
Professional Standards 48
Terminology 49
Tension Between Accessibility and Construct Preservation/Score Comparability 49
Labeling Nonstandard Test Administrations 50
Skill Substitution 51
Construct Fragmentation 54
Construct Shift 54
Public Policy Exceptions 55
Leveling the Playing Field: Access Versus Success 55
Oregon Case Settlement 56
No Child Left Behind (NCLB) Modified Tests 57
Eligibility for Nonstandard Test Administrations 58
Undue Burdens 59
Graduation Testing Challenges 60
The Debra P. and GI Forum Cases 60
Notice 61
Curricular Validity 61
Retests and Remediation 61
Notice and Curricular Validity for Special Education Students 62
The Brookhart Case 62
The Ambach Case 63
Recent Challenges to Graduation Tests by Students with Disabilities 64
The Indiana Case 64
The California Case 65
ELL Challenges to Graduation Tests 67
The Texas ELL Case 68
The California ELL Case 69
Accountability Testing Challenges 70
No Child Left Behind (NCLB) Act 70
NCLB ELL Provisions 71
ELL ''Accommodations'' and Modifications 71
Majority and Minority ELLs 71
Construct Shift 72
California State Law Case (2000) 72
NCLB Cases 73
The Reading School District Case 73
The Coachella Valley Case 73
Recommendations 74
Conclusion 77
References 77
4 IEP Team Decision-Making for More Inclusive Assessments: Policies, Percentages, and Personal Decisions 79
IEP Team Decision Making for More Inclusive Assessments 79
Historic Role of the IEP Team 82
IEP Team Decisions Regarding Participation in Statewide Assessment 82
State Guidelines for IEP Team Decision Making, 20072009 84
Recommendations to IEP Teams 84
Concluding Comments 90
References 90
5 Australian Policies to Support Inclusive Assessments 92
Introduction 92
The Australian Education Landscape 92
Australian Legislation Relevant to Students with a Disability 93
Disability Discrimination Act 93
Education Standards 94
Box 5.1. Standards for participation 95
Australian Policy Relevant to Students with a Disability 96
National Goals for Schooling in the Twenty-First Century 96
The National Assessment Program 97
Student Participation 99
Exempt Students 99
Special Provisions/Considerations 100
Issues Arising from Exemptions/Absences/Withdrawals 101
Issues Arising from Special Provisions/Considerations 102
Conclusion 103
References 104
Part II Classroom Connections 106
6 Access to What Should Be Taught and Will Be Tested: Students' Opportunity to Learn the Intended Curriculum 107
Student Access to the Intended Curriculum 108
The Intended Curriculum Model 108
The ICM for General Education 109
The ICM for Special Education 113
The Relevance of OTL 114
Conceptual Relevance 115
Substantive Relevance 115
Instructional Dimensions of OTL 116
Time on Instruction 116
Content of Instruction 118
Quality of Instruction 120
The Unfolding of Instruction 121
Measurement of OTL 122
Options for Measurement 126
The Future of OTL 132
References 133
7 Instructional Adaptations: Accommodations and Modifications That Support Accessible Instruction 138
Introduction 138
Instructional Adaptations: Important Distinctions 139
Distinguishing Between Instructional Adaptations and Differentiated Instruction 139
Distinguishing Between Instructional Accommodations and Modifications 140
Alignment of Instructional Adaptations with Grade-Level Content Standards 141
Consistency of Performance Expectations with General Education 142
Consequences of Implementing Instructional Adaptations 143
Consequences of Overuse of Instructional Adaptations 143
Consequences of Instructional Modifications 144
Interdependence Between Accessible Instruction and Accessible Assessments 144
Instructional Adaptations That Support Accessible Learning Environments 145
Instructional Accommodations 146
Changes in Presentation 146
Changes in Setting 148
Changes in Timing or Scheduling 148
Changes in Response Mode 149
Instructional Modifications 149
Changes in Presentation 149
Changes in Setting 150
Changes in Timing or Scheduling 150
Changes in Response Mode 151
Integrating Instructional Adaptations Based on Students Needs 151
Conclusions 151
References 152
8 Test-Taking Skills and Their Impact on Accessibility for All Students 154
Access and Test-Wiseness 154
Threshold Hypothesis 155
Frameworks and Findings 157
Test-Taking Skills and Other Methods of Increasing Access 160
Test-Taking Skills and Accommodations 160
Test-Taking Skills and Modifications 161
Computer-Based Test-Wiseness 161
Suggestions for Developers of Computer-Based Tests 162
A Lesson from Video Games 163
Practical Implications 164
Design Assessments to Minimize Test-Taking Skills 164
Evaluate the Influence of Test-Taking Strategies on a Case-By-Case Basis 164
Spend a Lot of Time Teaching Content and a Little Time Teaching Test-Taking Skills 164
Consider Interaction with Other Methods of Increasing Access to Tests 165
Teach for the Assessment 165
Conclusions 165
References 165
Part III Test Design and Innovative Practices 167
9 Accessibility Theory: Guiding the Science and Practice of Test Item Design with the Test-Taker in Mind 168
Universal Design: The End or the Beginning 168
Accessibility Theory: The Test-Taker, Not the Universe 169
Cognitive Load Theory: From Teaching to Testing 171
Developing Accessible Test Items: Identify, Quantify, and Modify 171
Item Stimulus 176
Item Stem 177
Visuals 177
Answer Choices 178
Page/Item Layout 178
Overall Accessibility Rating 179
Technical Evidence to Verify Intended Effects: The Accessibility Proof Paradox 182
Conclusion 183
Universal Design: The End or the Beginning? 183
Accessibility Theory: The Test-Taker, Not the Universe 183
Cognitive Load Theory: From Teaching to Testing 183
Developing Accessible Test Items: Identify, Quantify, and Modify 184
Technical Evidence to Verify Intended Effects: The Accessibility Proof Paradox 184
The Access Pathway: Accessibility Across the Educational Environment 184
References 185
10 Validity Evidence for Making Decisions About Accommodated and Modified Large-Scale Tests 188
Current Practice in Testing Approaches and Implications for Response Processes 190
Extended Time Research 193
Task Demands 193
Student Characteristics 193
Read Aloud Research 194
Task Demands 194
Student Characteristics 198
Research Designs and Quality of Research 198
Extended Time 199
Read Aloud Accommodations 200
Conclusions on Validity Evidence 201
Definitions of Constructs 202
Operationalization of Test Design 202
Training of Teachers 203
Organization of Systems (Research and Practice) 203
Validation of Outcomes 203
References 203
11 Item-Writing Practice and Evidence 206
Four Thousand Years of Limited Accessibility 206
Access and Assessment as Policy Tools 207
MC Formats 207
CR Formats 208
Item-Writing Guidelines and Taxonomies 209
MC Item-Writing Guidelines 209
Constructed-Response Item-Writing Guidelines 211
Choosing the Item Format 213
Evidence 213
Item-Writing Guidelines with Empirical Evidence 213
Three Options Are Optimal 214
Research on Item Modifications for Accessibility 215
AA-AAS 215
AA-MAS 216
Innovations and Technological Enhancements 216
Alternative Scoring Models 218
Pulling It All Together 219
References 220
12 Language Issues in the Design of Accessible Items 222
The Concept of Linguistic Accessibility in Content-Based Assessments in English 222
The Nature and Impact of Language Factors on Content-Based Assessment Outcomes 223
Creating More Linguistically Accessible Assessments 224
Unnecessary Linguistic Complexity in Assessments 224
Linguistic Features That May Affect Accessibility of Assessments 224
Procedures for Linguistic Modification of Test Items 225
Linguistic Modification: Practical Implications 228
Research and Methodological Issues Related to the Linguistic Accessibility of Assessments 228
Research on the Effectiveness of Linguistic Modification Approach for Improving Accessibility of Assessments for ELLs 229
Research Findings on Language as a Source of Measurement Error in Assessments 230
Research Findings on the Impact of Linguistic Complexity on the Validity of Assessments for ELL Students 231
Practical Steps for Improving Assessments for ELL Students 231
Formative Assessments to Help Identify Language Issues in the Assessments of ELL Students 231
Accessible Assessments at the Classroom Level Versus Accessible Assessments at the State or National Level 232
Guidelines and Recommendations for Creating More Linguistically Accessible Assessments for Students 232
Summary and Recommendations 233
References 234
13 Effects of Modification Packages to Improve Test and Item Accessibility: Less Is More 236
The AA-MAS Policy 236
State-Modified Achievement Tests 237
Kansas Assessments of Multiple Measures 238
Louisiana Educational Assessment Program Alternate Assessment, Level 2 238
Texas Assessment of Knowledge and Skills -- Modified 240
Experimental Studies of Modifications 240
Consortium for Alternate Assessment Validity and Experimental Studies 241
Consortium for Modified Alternate Assessment Development and Implementation 243
Operationalizing Alternate Assessment for Science Inquiry Skills 243
What Do We Know About Modified Achievement Tests 244
Future Needs 245
Conclusions 245
References 245
14 Including Student Voices in the Design of More Inclusive Assessments 247
Epistemological and Methodological Frameworks for Including Student Voices 248
Uses of Student Response Data in the Standards for Testing 249
Ethics and Standards for Professional Practice 250
Existing Assessment Research That Integrates Student Voice 251
Research Using Student Drawings: Two Examples 252
Research Using Student Interviews: Four Examples 253
Incorporating Student Voices in Future Assessment Research 256
Conclusions 257
References 257
15 Computerized Tests Sensitive to Individual Needs 259
Background 259
Rethinking Test Accommodations 261
Categories of Accommodations 262
Audio Access to Knowledge Representations 264
Needs Met by an Audio Presentation 265
Signed Access to Knowledge Representation 267
Alternate Language Access to Knowledge Representation 270
Tactile Access to Knowledge Representation 270
Adapted Presentation of Item Content 271
Implications for Test Validity 273
The Future Is Today 274
References 276
16 The 6D Framework: A Validity Framework for Defining Proficient Performance and Setting Cut Scores for Accessible Tests 278
ALDs and Standard Setting Terminology 279
A Validity Framework for Standard Setting for All Standards-Based Assessments 279
Define 280
Identify the Examinee Population 280
Identify the Relationship to Unmodified Grade-Level Assessments 280
Identify Desired Rigor for ALDs 281
Identify Means for Increasing Accessibility 282
Identify Intended Uses of ALDs 282
Stakeholder Involvement 282
Validity Evidence 282
Process Planning 282
Panelists and Committee Members 283
Process Evaluations 283
Describe 283
Types of ALDs 283
Defining Proficiency 284
ALD-Writing Workshop 284
Method for Writing Range and Target ALDs 285
Validity Evidence 287
Design 287
AA-MAS Test Designs 287
AA-AAS Test Designs 287
Methodologies for Recommending Cut Scores 288
Methodology for Policy Review of Cut Scores 288
Validity Evidence 289
Deploy 289
Refining Target ALDs into Reporting ALDs 289
Validity Evidence 290
Deliver 290
Using Reporting ALDs 290
Validity Evidence 291
Deconstruct 291
Technical Report 291
Outside Perspective 292
Validity Evidence 292
Discussion 292
Implementing the 6D Framework 292
What to Do When Test Development Has Already Occurred 293
Conclusions 293
References 294
Part IV Conclusions 296
17 Implementing Modified Achievement Tests: Questions, Challenges, Pretending, and Potential Negative Consequences 297
A Brief History of NCLB, Accountability, and the AA-MAS 298
The AA-AAS 298
The AA-MAS 298
Pennsylvania's GSEG Project 299
GSEG Activities 300
Survey 300
Focus Group 304
Analysis of PSSA Performance Trends for Students in Special Education 305
Recommendations from the GSEG 307
Additional Considerations and Unintended Consequences 310
Commonly Provided Rationale for the AA-MAS 310
Unintended Consequences 315
Concluding Thoughts 317
References 317
18 Accessible Tests of Student Achievement: Access and Innovations for Excellence 320
What We Know 320
Policy and Regulations About Access 321
Classroom Instruction and Access to the General Curriculum 322
Test Design That Supports Access 324
Where Are We Going 326
Challenges to Access 327
Needed Innovations to Improve Access 327
Conclusion 329
Subject Index 330