Competence Assessment in Education (eBook)

Contents 6
Chapter 1: Competence Assessment in Education: An Introduction 10
1.1 The German DFG-Priority Program “Competence Models for Assessing Individual Learning Outcomes and Evaluating Educational Processes” 10
1.2 Research Areas of the DFG-Priority Program 11
1.2.1 Cognitive Modeling and Assessment of Competencies 12
1.2.2 Innovations in Psychometric Models and Computer-­Based Assessment 13
1.2.3 Reception and Usage of Assessment Results 13
1.3 Conclusion 14
References 14
Part I: Modeling and Assessing Student Competencies 16
Chapter 2: Science-P I: Modeling Conceptual Understanding in Primary School 17
2.1 The Assessment of Science Competency in Primary School 18
2.2 Modeling Conceptual Understanding in Primary School Science 18
2.2.1 Model Specification and Item Construction 18
2.2.2 Conceptual Understanding: Dimensions and Levels 19
2.2.3 Validity 21
2.3 The Development of Conceptual Understanding in Primary School Science 22
2.4 Conceptual Understanding and Scientific Reasoning 23
References 24
Chapter 3: Science-P II: Modeling Scientific Reasoning in Primary School 26
3.1 Science-P 26
3.2 Development of Our Inventory 27
3.3 Convergent Validity of Paper-and-Pencil Inventory and Interviews 29
3.3.1 Method 30
3.3.1.1 Participants 30
3.3.1.2 Material 30
3.3.1.3 Procedure 31
3.3.2 Results 31
3.3.2.1 Pre-analyses 31
3.3.2.2 Convergent Validity 31
3.3.3 Discussion 32
3.4 Scientific Reasoning: Development from Grades 2 to 4 33
3.5 Competence-Structure Model of Scientific Reasoning: Hierarchical Levels of Competence 34
3.6 Outlook 35
References 35
Chapter 4: The Heidelberg Inventory of Geographic System Competency Model 37
4.1 The Role of Geographic System Competency in Geography Education 38
4.2 Study Overview 39
4.3 CogLabs 41
4.3.1 Description of the Measurement Instruments 41
4.3.2 Selected Results 43
4.4 First Quantitative Study (Q1) 45
4.4.1 Description of the Measurement Instruments 45
4.4.2 Dimensions of the Competency Model 45
4.4.3 Levels of the Competency Model 47
4.5 Second Quantitative Study (Q2) 49
4.5.1 Description of the Measurement Instruments 49
4.5.2 Dimensions of the Competency Model 49
4.5.3 Levels of the Competency Model 53
4.6 Discussion 53
4.6.1 Dimensions of GSC 53
4.6.2 GSC Levels 55
4.7 Conclusions 56
References 57
Chapter 5: An Extended Model of Literary Literacy 60
5.1 The Comprehension of Literary and Expository Texts 61
5.2 Current Research on Literary Literacy and Further Directions 61
5.2.1 The Internal Structure of Literary Literacy 62
5.2.2 The Need for an Extended Model of Literary Literacy 63
5.3 Research Objectives 65
5.4 Method 65
5.4.1 Sample 65
5.4.2 Measures 65
5.4.2.1 Semantic Literary Literacy 66
5.4.2.2 Idiolectal Literary Literacy 67
5.4.2.3 The Ability to Recognize Foregrounded Passages (Foregrounding) 67
5.4.2.4 Specific Literary Knowledge 67
5.4.2.5 The Ability to Recognize Emotions Intended by a Literary Text 68
5.4.3 Statistical Analyses 68
5.5 Results 70
5.6 Discussion 72
5.6.1 The Structure of Literary Literacy 74
5.6.2 Some Considerations on Teaching Literary Literacy 74
5.6.3 Limitations of the Study 76
References 76
Chapter 6: Self-Regulated Learning with Expository Texts as a Competence: Competence Structure and Competence Training 80
6.1 Theoretical Background 80
6.1.1 Integrated Model of Self-Regulated Learning 81
6.2 Research Questions and Hypotheses 84
6.3 Studies 1a and 1b: A “Component Perspective” on the Structure of Self-Regulated Learning Competence 85
6.4 Study 2: A “Process Perspective” on the Structure of Self-Regulated Learning Competence 89
6.5 Discussion 91
References 93
Part II: Modeling and Assessing Teacher Competencies 95
Chapter 7: Investigating Pre-service Teachers’ Professional Vision Within University-Based Teacher Education 96
7.1 Introduction 96
7.2 Modeling the Structure of Professional Vision 97
7.2.1 Noticing: Selective Attention to Important Classroom Events 98
7.2.2 Reasoning: Interpretation of Important Classroom Events 99
7.3 Testing the Structure of Professional Vision 100
7.3.1 The Assessment Tool Observer 101
7.3.2 Interrelation Between the Three Reasoning Dimensions 103
7.4 Investigating Changes in Professional Vision Within University-Based Teacher Education 103
7.4.1 The Role of Formal and Informal OTL 104
7.4.2 The Design of Formal and Informal OTL 105
7.5 Building the Bridge from Professional Vision to Teaching Action 106
7.5.1 M-Teach Events as Assessment of Teaching Action 107
7.5.2 Pre-service Teachers’ Teaching Skills in M-Teach Events 108
7.6 Conclusion and Outlook 109
References 109
Chapter 8: Teacher Knowledge Experiment: Conditions of the Development of Pedagogical Content Knowledge 113
8.1 Introduction 114
8.1.1 The Construct of Pedagogical Content Knowledge 114
8.1.2 Conditions for the Development of Pedagogical Content Knowledge: The Role of Prior Content Knowledge and Pedagogical Knowledge 115
8.1.3 The Present Study 117
8.2 Methods 118
8.2.1 Participants 119
8.2.2 Treatments 119
8.2.3 Measures 122
8.2.4 Baseline Equivalence and Treatment Implementation Checks 123
8.3 Results 124
8.3.1 Measurement of Pre-service Teachers’ Knowledge 124
8.3.2 Testing the Assumptions on PCK Development 127
8.4 Discussion 127
References 129
Chapter 9: Teachers’ School Tracking Decisions 132
9.1 Introduction 133
9.2 Dual Process Models of Decision Making 134
9.3 The Adaptive Diagnostic Competency Model (ADCM) 135
9.4 Testing the ADCM 136
9.4.1 Student Case Vignettes (Pre-studies 1 and 2) 137
9.4.2 The ADCM: Case Consistency and Expertise (Study 1) 139
9.4.3 The ADCM: Case Consistency and Accountability (Study 2) 141
9.5 Training Study 143
9.6 Discussion 144
References 146
Chapter 10: Modeling, Measuring, and Training Teachers’ Counseling and Diagnostic Competencies 149
10.1 Introduction 150
10.2 Project Goals 151
10.3 Modeling Teachers’ Counseling and Diagnostic Competencies 152
10.3.1 Theoretical Background 152
10.3.2 Method 154
10.3.3 Results 154
10.3.4 Relationship Between Teachers’ Counseling and Diagnostic Competence 155
10.4 Measuring Teachers’ Counseling and Diagnostic Competence 156
10.4.1 Scenario Tests 156
10.4.2 Situational Judgment Test 157
10.4.3 Knowledge Tests and Self-Assessment Questionnaires 157
10.5 Training Teachers’ Counseling and Diagnostic Competence 158
10.5.1 Training Program in Diagnostic Competence for In-Service Teachers 159
10.5.2 Training Programs in Counseling and Diagnostic Competence for Prospective Teachers 160
10.6 Conclusions and Outlook 162
References 163
Chapter 11: Development and Evaluation of a Competence Model for Teaching Integrative Processing of Texts and Pictures (BiTe) 167
11.1 The “BiTe-Project” 168
11.2 Theoretical Background 168
11.2.1 Challenges of Picture-Text-Integration (PTI) 168
11.2.2 Teachers’ Competencies for Teaching the Integrative Processing of Pictures and Texts 168
11.2.3 Quality of Instruction 169
11.3 Research Questions and Hypotheses 169
11.4 Methods 170
11.4.1 Sample and Study Design 170
11.4.2 Measures 171
11.5 Results 175
11.5.1 Research Question (1): Knowledge About PTI 175
11.5.2 Research Question (2): Teachers’ Attitudes, Motivation, and Self-Related Cognitions Towards PTI and Diagnostics in PTI 175
11.5.3 Research Question (3a): Teachers’ Accuracy of Judgment 176
11.5.4 Research Question (3b): Teachers’ Accuracy of Judgment, Knowledge, and Duration of Contact 177
11.5.5 Research Question (4a): Relations Between Teachers’ Competencies and Instruction 177
11.5.6 Research Question (4b): Relations Between Teachers’ Competencies and Students’ Competence and Engagement 178
11.6 Discussion 178
References 179
Part III: Modeling and Assessing Vocational Competencies and Adult Learning 181
Chapter 12: Multidimensional Competency Assessments and Structures in VET 182
12.1 Introduction 183
12.2 Theoretical Issues and Operationalization 183
12.3 Research Questions 186
12.4 Research Design 187
12.5 Hypotheses and Results 188
12.5.1 Research Question 1: Competency Structures Within the Construct of Not Directly Action-Centered Occupation-Specific Knowledge: Dimensionality and Its Development 188
12.5.2 Research Question 1: Competency Structures Within the Construct of Action-Centered Occupation-­Specific Knowledge: Dimensionality and Its Development 193
12.5.3 Preliminary Analysis for Research Question 2: Construct of Occupation-Specific Problem Solving: Validity and Reliability 195
12.5.4 Research Question 2: Competency Structures Between Different Constructs 197
12.6 Summary and Final Discussion 198
References 200
Chapter 13: Professional Competencies of Building Trade Apprentices After Their First Year of Training 202
13.1 Introduction 202
13.2 Aims and Objectives 204
13.3 Vocational Training in the Building Trades 204
13.4 Professional Competence of Building Trade Apprentices 205
13.5 Research Design and Data Collection 206
13.6 Results 207
13.6.1 Sociodemographic Factors, Cognitive Abilities and Apprentices’ Performance at the Beginning of the First Training Year 208
13.6.2 Professional Competence at the End of the First Training Year 210
13.7 Additional Findings and Prospects 215
References 216
Chapter 14: Assessing Tomorrow’s Potential: A Competence Measuring Approach in Vocational Education and Training 220
14.1 Background 221
14.1.1 Prospects and Demand for Adequate Competence Assessments in Vocational Education 221
14.1.2 The Original Conceptualization of Final Examinations in the Area of Business and Commerce 222
14.1.3 Assessment Model for Commercial Vocations 225
14.2 Method 228
14.2.1 Sample 228
14.2.2 Examination of Validity 228
14.2.3 Examination of Reliability 229
14.3 Results 230
14.3.1 Results for the Test’s Validity 230
14.3.2 Results for the Test’s Reliability 231
14.4 Discussion 232
14.5 Conclusions 234
Appendix 235
Ceraforma Keramik AG 235
Business Process 1 236
References 240
Part IV: Competency Development: Modeling of Change and Training of Competencies 243
Chapter 15: The Development of Students’ Physics Competence in Middle School 244
15.1 Introduction 245
15.2 Theoretical Background 245
15.2.1 Students’ Understanding of Energy 245
15.2.2 Students’ Learning About Energy 246
15.3 Research Questions 247
15.4 Project Design 249
15.5 Phase 1: The Cross-Sectional Study 250
15.5.1 Method 250
15.5.2 Results 251
15.6 Phase 2: Longitudinal Study 253
15.6.1 Method 253
15.6.2 Results 254
15.7 Summary and Outlook 256
References 257
Chapter 16: Modeling and Fostering Decision-Making Competencies Regarding Challenging Issues of Sustainable Development 260
16.1 Introduction 261
16.2 A Competence Model for Decision Making with Respect to Sustainable Development 261
16.3 Measurement Instruments and Competence Modeling 263
16.3.1 “Developing Solutions”: Development of the Measurement Instrument 263
16.3.1.1 Sample 263
16.3.1.2 Measures: Tasks and Items 263
16.3.1.3 Instrument Functioning 264
16.3.2 Modeling of “Developing Solutions” 266
16.3.3 Discussion 268
16.4 Experimental Validation: A Comparison of Socioscientific Decision Making with Analytical Problem Solving 268
16.4.1 Objectives and Research Design 269
16.4.2 Methods 269
16.4.2.1 Participating Students and Teachers 270
16.4.2.2 Trainings and Learning Material 270
16.4.2.3 Measures 272
16.4.3 Results of the Pilot Study 274
16.4.4 Discussion 275
16.5 Conclusions and Outlook 277
References 278
Chapter 17: Metacognitive Knowledge in Secondary School Students: Assessment, Structure, and Developmental Change 282
17.1 Theoretical Background 283
17.1.1 Methodological Issues Regarding the Assessment of Declarative Metacognitive Knowledge 285
17.1.2 Design of the EWIKO Study 286
17.1.3 Test Instruments 287
17.2 Overview of Major Results 290
17.2.1 Development of Metacognitive Knowledge: Sources of Interindividual Differences 290
17.2.2 Domain-Specificity—A Transitional Period of Metacognitive Development? 292
17.2.3 Interrelations Between Metacognitive Knowledge and Achievement 294
17.3 Discussion 295
References 296
Chapter 18: Development of Dynamic Usage of Strategies for Integrating Text and Picture Information in Secondary Schools 300
18.1 Texts Combined with Instructional Pictures 301
18.2 Theoretical Background 301
18.2.1 Taxonomies of Text-Picture-Integration 301
18.2.2 Strategies for Integrative Processing of Text and Pictures 302
18.3 Research Questions and Hypotheses 302
18.4 Method 303
18.5 Results 305
18.5.1 Reading and Observation Times 305
18.5.2 Transitions Between Texts, Pictures, and Items 306
18.6 Discussion 306
References 309
Chapter 19: Training in Components of Problem-Solving Competence: An Experimental Study of Aspects of the Cognitive Potential Exploitation Hypothesis 311
19.1 Introduction 312
19.2 Theoretical Framework 312
19.2.1 Problem Solving in PISA 2003: The Cognitive Potential Exploitation Hypothesis 313
19.2.2 Components of Problem-Solving Competence 315
19.3 Research Questions 316
19.4 Study I 316
19.4.1 Methods 316
19.4.2 Results 317
19.4.3 Discussion 318
19.5 Study II 318
19.5.1 Methods 318
19.5.1.1 Data Analysis 320
19.5.2 Results 320
19.5.2.1 Planning 320
19.5.2.2 Problem Solving 320
19.5.2.3 Mathematics 321
19.5.3 Discussion 322
19.6 General Discussion 323
19.6.1 Limitations and Future Research 323
References 324
Chapter 20: An Intensive Longitudinal Study of the Development of Student Achievement over Two Years (LUISE) 328
20.1 Introduction 329
20.2 Student Achievement in Languages 330
20.3 Student Achievement in Mathematics 331
20.4 Changes in Student Achievement in Mathematics and Native Language 331
20.5 The Present Study 333
20.5.1 Method 334
20.5.1.1 Procedure and Participants 334
20.5.1.2 Measures 335
20.5.1.2.1 German Achievement Tests 335
20.5.1.2.2 Mathematics Achievement Tests 337
20.5.2 Results 337
20.5.2.1 German Achievement 337
20.5.2.2 Mathematics Achievement 338
20.6 Summary and Discussion 344
References 346
Part V: Innovations in Psychometric Models and Computer-Based Assessment 350
Chapter 21: Multidimensional Structures of Competencies: Focusing on Text Comprehension in English as a Foreign Language 351
21.1 Introduction 352
21.2 Test Development 352
21.2.1 Item Characteristics 352
21.2.2 Item Development 355
21.2.3 Validation of Item Characteristics 356
21.3 Test and Item Analysis 356
21.3.1 Sample and Data Collection 356
21.3.2 Unidimensional Test and Item Analysis 356
21.3.3 Item Difficulties Across Domains 357
21.3.4 Local Dependencies 357
21.3.5 Multidimensional Analysis 358
21.4 Discussion 359
21.4.1 Research Perspectives 359
21.4.2 Implications for Educational Contexts 360
References 361
Chapter 22: Multidimensional Adaptive Measurement of Competencies 363
22.1 Problem 364
22.2 Consideration of Multiple Constraints in MAT 366
22.2.1 Multidimensional Maximum Priority Index 367
22.2.2 Research Objective 367
22.2.3 Method 368
22.2.4 Procedure 368
22.2.5 Results 369
22.2.6 Discussion 369
22.3 Using Prior Information for Item Selection and Ability Estimation 370
22.3.1 Research Questions 370
22.3.2 Method 370
22.3.3 Results 371
22.3.4 Discussion 373
22.4 The Multidimensional Adaptive Testing Environment (MATE) 373
22.4.1 Computerization of Items 373
22.4.2 Assignment of Item Parameters 374
22.4.3 Configuration of Tests and Test Batteries 374
22.4.4 Pre-operational Simulation Studies 375
22.4.5 Graphical User Interface, System Requirements, Availability and Manual 376
22.5 Empirical Application 376
22.6 Conclusion 378
References 380
Chapter 23: Development, Validation, and Application of a Competence Model for Mathematical Problem Solving by Using and Translating Representations of Functions 382
23.1 Introduction 383
23.2 Construction of a Theoretical Framework Model 384
23.3 Development and Empirical Validation of Psychometric Models 386
23.3.1 Basic Model: Representations and Translations Between and Within Situational, Numerical, and Graphical Representation 386
23.3.2 Extension: Inclusion of Algebraic Representation/Cognitive Action 388
23.3.3 Extension: Hierarchical Models 391
23.4 Development and Evaluation of a Computerized Adaptive Testing Procedure 392
23.4.1 Aims for the Development of the Adaptive Test 392
23.4.2 Item Pool and Calibration 393
23.4.3 Pre-operational Simulation Study 394
23.5 Discussion 396
References 397
Chapter 24: Relating Product Data to Process Data from Computer-Based Competency Assessment 400
24.1 Introduction 401
24.2 Study 1: The Effect of Time on Task Success Differs Across Persons and Tasks 402
24.2.1 Research Goal and Hypotheses 403
24.2.2 Methods 403
24.2.3 Results 404
24.2.4 Discussion 404
24.3 Study 2: Benefits for Task Completion from the Automatization of Subtasks 405
24.3.1 Research Goal and Hypotheses 406
24.3.2 Methods 406
24.3.3 Results 407
24.3.4 Discussion 408
24.4 Study 3: Number of Interactions: More Is Not Always Better 408
24.4.1 Research Goal and Hypotheses 409
24.4.2 Methods 410
24.4.3 Results 410
24.4.4 Discussion 411
24.5 Study 4: Problem Solver Types: Different Ways to Success in Information Problems 412
24.5.1 Methods 412
24.5.2 Results 412
24.5.3 Discussion 413
24.6 How to Handle Unstructured Process Data?: The Log File Data Extraction Tool 413
24.7 Conclusions and Final Remarks 415
References 416
Chapter 25: Dynamic Problem Solving: Multiple-Item Testing Based on Minimally Complex Systems 419
25.1 Introduction 420
25.2 Modeling of Problem Solving Competencies 420
25.3 Development of Computer-Based Assessment Instruments 422
25.4 Empirical Tests of the Newly Developed Instruments 427
25.5 Educational Application: PISA 2012 429
25.5.1 Two Additional Issues: Optimization and Causal Diagrams 430
25.6 Future Developments 431
References 432
Part VI: Feedback From Competency Assessment: Concepts, Conditions and Consequences 436
Chapter 26: Formative Assessment in Mathematics Instruction: Theoretical Considerations and Empirical Results of the Co2CA Project 437
26.1 Formative Assessment: A Promising Approach to Improving Teaching and Learning? 438
26.1.1 Formative Assessment: State of the Art 438
26.1.2 Components of Formative Assessment 439
26.1.3 How Formative Assessment Affects Learning 439
26.2 The Four Studies of the Co2CA Project 440
26.2.1 Survey Study 440
26.2.2 Experimental Study 441
26.2.3 Intervention Study 442
26.2.3.1 Aims and Research Questions of the Intervention Study 442
26.2.3.2 Design of the Intervention Study 443
26.2.3.3 Selected Results of the Intervention Study 449
26.2.4 Transfer Study 449
26.2.4.1 Aims and Research Questions of the Transfer Study 449
26.2.4.2 Design of the Transfer Study 450
26.2.4.3 First Results of the Transfer Study 452
26.3 Summary 453
References 455
Chapter 27: Arguing Validity in Educational Assessment 458
27.1 Introduction 459
27.2 The Validity Concept 460
27.2.1 The Place of Standard Setting in Educational Assessment 462
27.3 The Argument Approach to Evaluating Validity 463
27.3.1 A Structured Validity Argument 464
27.3.1.1 Warrant of Well-Defined Content Standards 466
27.3.1.2 Warrant of Test Alignment to the Content Domain 466
27.3.1.3 Warrant of Well-Defined Performance Level Descriptors 467
27.3.1.4 Warrant of Reliable Test-Score Measurement 468
27.3.1.5 Warrant of Defensible Cut-Score Placements 468
27.4 Discussion 469
References 471
Chapter 28: Evaluating Prerequisites for the Development of a Dynamic Test of Reading Competence: Feedback Effects on Reading Comprehension in Children 475
28.1 Introduction 476
28.2 The Idea of Dynamic Assessments 476
28.3 Dynamic Assessments of Reading Competence: Existing Approaches and Challenges 478
28.4 Experiments on the Effectiveness of Feedback on  Reading Comprehension in a Train-Within-Test Setting 481
28.4.1 Experiment 1 481
28.4.2 Experiment 2 483
28.4.3 Experiment 3 485
28.5 Effects of Feedback on Reading Comprehension Within a Computer-Delivered Test: Lessons Learned 486
28.6 Prospects of Dynamic Tests of Reading Competence 487
References 488