Innovative Assessment for the 21st Century (eBook)

Acknowledgements 4
Foreword 5
Contents 7
Contributors 9
1 Prelude: Assessment for the 21st Century 11
1.1 Introduction 11
1.2 The Big Problems 12
1.3 Defining Assessment Ideas and Terms 14
1.3.1 Educational Measurement 14
1.3.2 Assessment 15
1.3.3 Determining Assessment Quality 16
1.4 Kinds of Assessment 17
1.5 Discussion 18
References 20
2 Human Action and Social Groups as the Natural Home of Assessment: Thoughts on 21st Century Learning and Assessment 22
2.1 The Natural Home of Assessment: Human Action and Social Conventions 22
2.2 Becoming a SWAT Team Member: Identity and Domains 25
2.3 Lifting Assessment Out of the Basic Circuit of Human Action: The First Step 29
2.4 Sim Domains: Simplified Simulations of a Domain 30
2.5 Play 31
2.6 Five Learning Environments 33
2.7 Generic Environment Learning 36
2.8 Non-lucid Pseudo-Domain Learning in Schools 37
2.9 Pro-Am Communities 38
2.10 Twenty-First Century Skills 40
2.11 Formalizing Assessment Beyond the Basic Circuit of Human Action and Social Groups that Form Peoples Appreciative Systems 41
2.12 Conclusion 45
References 47
3 Growing Learning and Assessment in the 21st Century 49
References 54
4 Aiming at Learning: Assessment as the Critical Link 56
4.1 Part I: Current State 58
4.2 Part II: What Would Be Better 64
4.3 Part III: Practical Solutions 68
4.4 Part IV: Creating a Roadmap with a Destination 70
4.5 Conclusion 73
References 74
5 Sharpening the Aim: Making Strides to Create an Assessment Culture in Schools 76
5.1 Part I: Current State: My Perspective 76
5.2 Part II: What Would Be Better Aim Higher Using National Assessment of Educational Progress (NAEP) Assessment Tasks 78
5.3 Part III: A Culture of Assessment 78
5.4 Part IV: Evidence Focused Teaching 79
5.5 Conclusion 80
6 Using Evidence Centered Design to Think About Assessments 81
6.1 A Language for Talking About New Kinds of Assessments 81
6.2 The Mathematics of Evidence 82
6.2.1 Bayesian Model of Evidence 82
6.2.2 Weight of Evidence 83
6.2.3 Influence Diagrams 84
6.3 The Evidence-Centered Design (ECD) Models 87
6.4 Integrating Evidence from Diverse Sources 89
6.4.1 ECD as an Evidence Integration Framework 89
6.4.2 ECD Over Time 91
6.5 Assessments of Multiple Aspects of Proficiency 92
6.5.1 Direct and Indirect Evidence 93
6.5.2 Proficiency Model Revisited 94
6.5.3 Evidence Model Revisited 95
6.5.4 The ACED Experience 98
6.6 Evidence from Complex Tasks 100
6.6.1 Observables and Rubrics 100
6.6.2 Many-to-Many Mappings 102
6.7 Evidence and Argument 103
References 104
7 Thinking About Assessments in a Transitional Time 107
7.1 How Can We Organize Evidence About Student Performance Gathered from Diverse Sources Across Multiple Time Points 108
7.2 How Should We Balance Information Gathered About Multiple Aspects of Proficiency 109
7.3 How Should We Collect Evidence from Complex Tasks 110
7.4 Final Thoughts 111
References 112
8 Participatory Assessment of 21st Century Proficiencies 113
8.1 New Media the Technology Proficiencies and Schools 115
8.1.1 Why Bother? 115
8.2 Efforts to Bridge the Void Between New Proficiencies and Schools 116
8.2.1 Practice-Oriented Responses to New Media and Technology 116
8.2.1.1 New Literacy Studies 117
8.2.1.2 New Media Studies 118
8.2.1.3 New Approaches to Writing 118
8.2.1.4 Challenges for Practice-Oriented Responses 119
8.2.2 Measurement-Oriented Responses to New Media and Technology 120
8.2.2.1 Changing Standards and Tests 120
8.2.2.2 New Curriculum and Classroom Assessments 122
8.3 So, What Is the Problem 122
8.3.1 Evidential Validity 123
8.3.2 Consequential Validity 125
8.4 A Proposed Participatory Alternative 126
8.4.1 Situative and Sociocultural Considerations of Assessment and Testing 126
8.4.2 An Initial Application of Participatory Assessment to the Teachers' Strategy Guide 128
8.4.3 Key Aspects and Assumptions of Participatory Assessment 128
8.4.3.1 Multiple Levels of Assessment 128
8.4.3.2 A Focus on Assessment Orientation and Timescale 129
8.4.3.3 A Focus on Formative and Summative Functions 131
8.4.3.4 Design-Based Iterative Refinements 132
8.4.3.5 Proficiencies as Formalisms and Boundary Objects 132
8.4.4 Examples and Descriptions of Assessments for the Teachers' Strategy Guide 133
8.4.4.1 Immediate-Level Event Reflections 133
8.4.4.2 Close-Level Activity-Oriented Reflections 136
8.4.4.3 Proximal-Level Artifact-Oriented Reflections 136
8.4.4.4 Distal-Level Standards-Oriented Assessments 138
8.4.4.5 Remote-Level Achievement-Oriented Tests 138
8.4.5 Continuing and Future Efforts 139
8.5 Conclusion 140
References 140
9 Making Assessment Relevant to Students, Teachers, and Schools 145
9.1 Introduction 145
9.2 Transparency and the Failure of Relevance 146
9.3 A Proposed Balanced Assessment (and Accountability) System 148
9.4 Describing and Explaining the Proposed System 150
9.5 Entry Points 150
9.5.1 Entry Point 1: Limited Number of High-School Exit Standards 150
9.5.2 Entry Point 2: Assessment Literacy Standards for Educator Certification 153
9.6 Developing the Assessment Components of the System 155
9.6.1 Classifying Content and Process Standards 155
9.6.2 Developing Model Classroom Strategies and Materials 156
9.6.3 Developing a Clearinghouse for Classroom Assessment Strategies and Materials 158
9.6.4 Implement Classroom Assessment Strategies and Materials 158
9.6.5 Developing the Secure Assessment Components 159
9.7 Accountability 165
9.7.1 Accountability Components 165
9.7.2 Combining Achievement and Growth for Accountability 166
9.8 Conclusion 167
References 168
10 Automated Essay Scoring in a High Stakes Testing Environment 172
10.1 The Need for Change in Testing Practices 172
10.2 Automated Essay Scoring 173
10.2.1 How Automated Essay Scoring Works 174
10.2.2 IntelliMetric 
176 
10.2.3 Reliability and Validity of Automated Essay Scoring Systems 179
10.2.4 Reliability 179
10.2.5 Validity 180
10.3 Electronic Portfolio Applications 182
10.3.1 My Access® 
182 
10.3.2 Computer-based Testing and Vulnerable Populations 184
10.4 A Proposal 185
10.4.1 Potential Benefits 186
10.4.1 Conclusion 187
10.4.1 Automated Essay Scoring Vendors 188
References 188
11 Assessing Change in Learners Causal Understanding Using Sequential Analysis and Causal Maps 191
11.1 Introduction 191
11.2 Assessing Change in Causal Maps with Sequential Analysis 195
11.2.1 Method 195
11.2.2 Data for Sequential Analysis 197
11.2.3 Findings 199
11.2.4 Implications 200
11.3 Assessing Argumentation and Effects on Causal Maps 200
11.3.1 Method 201
11.3.2 Findings 205
11.3.3 Implications 207
11.4 Directions for Future Research 208
References 208
12 Development and Use of a Tool for Evaluating Teacher Effectiveness in Grades K-12 210
12.1 Introduction 210
12.1.1 Exemplary Teacher Research 212
12.1.2 Development of the Classroom AIMS Instrument 213
12.2 Method 217
12.2.1 Data Sources 217
12.2.2 Confirmatory Factor Analysis (CFA) Results 218
12.2.3 Further Evidence of Construct Validity 224
12.3 Discussion 226
12.3.1 Implications 227
12.3.2 Applications 228
12.3.3 Conclusion 229
References 229
13 Epilogue: Achieving Quality 21st Century Assessment 232
13.1 Assessment Must Capitalize on Technology 233
13.2 Assessment is a Contextualized, Social Activity 235
13.3 Assessment Must Serve Teaching and Learning 237
13.4 Conclusion 239
References 240
Subject Index 241