Computer-Based Diagnostics and Systematic Analysis of Knowledge (eBook)

Preface 6
Elicitation of Knowledge 7
Aggregation and Classification of Knowledge 7
Comparison and Empirical Testing of Strategies 7
Application of Results 7
Acknowledgements 7
Contents 10
Contributors 14
About the Authors 16
Reviewers 22
Part I Elicitation of Knowledge 23
Intermezzo 1 -- To Be Moved by Knowledge: Moving Knowledge MovesKnowledgeAboutKnowing 23
1 Essentials of Computer-Based Diagnostics of Learning and Cognition 25
1.1 Introduction 25
1.2 Diagnostics and Diagnosis in The Area Of Education and Instruction 27
1.3 Responsive Methods of Measurement and Assessment 28
1.4 Constructive Methods of Knowledge Diagnosis 29
1.5 The Role Of External Representations 30
1.6 Computer-Based and Agent-Based Knowledge Diagnosis 33
1.7 Preview 35
References 35
2 A Functional View Toward Mental Representations 37
2.1 Representation in General 37
2.1.1 Mental Representations 38
2.1.2 The Vehicle 39
2.1.3 The Representandum 39
2.1.4 The Subject 40
2.1.5 The Triple-Digit Relation 40
2.1.6 Types of Mental Representation 41
2.2 Related Notions 43
2.2.1 ''Explained by '''' 43
2.2.2 ''Structured by '''' 43
2.2.3 Misrepresentation 44
2.3 How to Operate on Representations 44
2.3.1 How Do You Know that Someone Is Able to Generate and Use Conceptual Representations? 45
References 46
3 Mental Representations and Their Analysis: An Epistemological Perspective 48
3.1 Introduction 48
3.1.1 The Nature of Learning 48
3.1.2 Learning Complex Problem-Solving Tasks and Skills 49
3.2 Mental Models 54
3.3 Mental Model Assessments and Learning Progress 55
3.4 An Epistemological Perspective 58
3.5 Concluding Remarks 59
References 60
4 Multi-decision Approaches for Eliciting Knowledge Structure 62
4.1 Knowledge and Knowledge Structure 62
4.2 Relatedness Data and Its Analysis and Representation 63
4.3 Alternative Approaches to Elicit Relatedness Data 66
4.3.1 The Role of Context 67
4.3.2 Computer-Based Listwise and Sorting Multi-decision Approaches 69
4.3.3 The Effects of Headings on Knowledge Structure 69
4.3.4 Listwise and Sorting Approaches Compared to the Pairwise Approach 71
4.3.5 Sorting and Listwise Combined Approach 75
4.4 Summary and Conclusion 78
References 79
5 The Problem of Knowledge Elicitation from the Experts Point of View 81
5.1 Introduction 81
5.2 Description of the System and Knowledge Elicitation 82
5.3 Language Skills Database 87
5.4 Adaptive Fuzzy E-Learning Subsystems 88
5.5 Supervised Learning Schema 90
5.6 Conclusions 92
References 92
Part II Aggregation and Classification of Knowledge 94
Intermezzo 2 -- Artefacts of Thought: Properties and KindsofRe-representations 94
6 Automated Knowledge Visualization and Assessment 96
6.1 Introduction 96
6.2 Applying Current Computer Technology 97
6.3 Automated Tools 99
6.3.1 Mitocar 99
6.3.1.1 Two Phases of Data Collection 99
6.3.1.2 Graphical Re-representation of the Model 102
6.3.1.3 Additional Descriptive Elicitation Modules 103
6.3.1.4 Automated Report Engine 105
6.3.2 T-MITOCAR 107
6.3.2.1 Preparing the Text 107
6.3.2.2 Tokenizing 107
6.3.2.3 Tagging 107
6.3.2.4 Stemming 108
6.3.2.5 Fetching the Most Frequent Concepts from the Text 108
6.3.2.6 Sum of Distances: Determining Pairwise Associatedness 108
6.3.2.7 Determining the Weights 109
6.3.2.8 De-stemming 109
6.3.2.9 Writing the Model to the List Form 109
6.3.2.10 Example from Wikipedia Texts (Economy) 109
6.3.2.11 How to Use T-MITOCAR 111
6.3.2.12 Applications 112
6.3.3 T-MITOCAR Artemis 113
6.3.3.1 Input Formats and Interface 113
6.3.3.2 Output Format of the Knowledge Map 114
6.3.4 SMD Technology 114
6.3.4.1 Phase 1: Input 114
6.3.4.2 Phase 2: Analysis Specification 117
6.3.4.3 Phase 3: Quantitative Analysis Output 117
6.3.4.4 Phase 4: Standardized Graphical Output 117
6.3.5 Model Comparison 118
6.3.6 Comparison Measures 120
6.3.6.1 Surface Matching 120
6.3.6.2 Graphical Matching 120
6.3.6.3 Structural Matching 121
6.3.6.4 Gamma Matching 121
6.3.6.5 Concept Matching 121
6.3.6.6 Propositional Matching 121
6.3.6.7 Balanced Semantic Matching 122
6.3.6.8 Triangulation of Types of Expertise 122
6.3.7 HIMATT 122
6.4 AKOVIA 124
6.4.1 Foundation and Design 125
6.4.2 AKOVIA Input 125
6.4.2.1 Input from Graphs (List Form) 126
6.4.2.2 Input from Text 127
6.4.2.3 Mixed Format Input 127
6.4.3 Common Model Data Frame 128
6.4.4 Analysis and Scripting 128
6.4.4.1 Visualize 129
6.4.4.2 Ganalyze 129
6.4.4.3 Compare 129
6.4.4.4 Aggregate 130
6.4.5 Upload, Feedback, and Analysis 130
6.4.6 Server Topology 130
6.4.7 Data Warehousing 131
6.5 Applications And Future Perspectives 131
6.5.1 Applications 132
6.5.2 Future Perspectives 132
References 133
7 Deriving Individual andINTtie Group Knowledge Structure fromINTtie
7.1 Introduction 135
7.2 Pathfinder Network Analysis 135
7.3 Network Diagrams and Knowledge Structure 136
7.3.1 ALA-Mapper Investigations 137
7.3.2 Rubrics and Network Diagram Scores 139
7.4 Essays and Knowledge Structure 140
7.4.1 Sentence Aggregate Approach 141
7.4.2 Linear Aggregate Approach 142
7.5 Next Steps 145
References 147
8 A Self-Organising Systems Approach to History-Enriched Digital Objects 149
8.1 Self-Organising Systems 149
8.2 Social Software 151
8.2.1 Social Software for Education 152
8.3 Text Signalling 153
8.4 History-Enriched Digital Objects 154
8.5 Summary 155
8.6 The Design of CoREAD 155
8.7 The Study 157
8.8 Method 157
8.9 Results 158
8.9.1 Descriptive Statistics 159
8.9.2 The Author's Summary 160
8.9.2.1 The Author's Summary Reproduced 160
8.9.3 Students' Highlights 161
8.9.3.1 Differences Between the Text and Author Comparisons 161
8.9.3.2 Trend over Time 161
8.9.3.3 Correlational Analyses 161
8.9.4 Students' Written Summaries 163
8.9.4.1 Differences Between the Text and Author Comparisons 163
8.9.4.2 Trend over Time 164
8.9.4.3 Correlational Analyses 164
8.9.4.4 Multiple Regression Analyses 164
8.9.5 Case Studies 165
8.9.5.1 Best Summary When Compared to the Flynn Effect Text # Participant #14 166
8.9.5.2 Worst Summary When Compared to the Flynn Effect Text # Participant #1 167
8.9.5.3 Best Summary When Compared to the Author#s Summary # Particpant #4 168
8.9.5.4 Worst Summary When Compared to the Author#s Summary # Participant #40 169
8.10 Discussion 170
8.10.1 Social Software for Assessment and Feedback 170
8.10.1.1 Assessment 170
8.10.1.2 Feedback 171
8.10.2 Limitations and Future Work 172
8.10.2.1 Limitations of Highlighting 172
8.10.2.2 No Trend over Time 173
8.10.2.3 Effects of Text Length on LSA Scores 173
8.11 Conclusion 173
References 174
9 Performance Categories: Task-Diagnostic Techniquesand Interfaces 177
9.1 Introduction 177
9.2 Tasks 177
9.2.1 Tasks, Outcomes, and Processing 178
9.2.2 Tasks at Work 178
9.2.3 A Continuum of Tasks 179
9.3 Diagnostic Environments 180
9.3.1 Task Success 181
9.4 The Diagnostic Continuum 181
9.4.1 Prescribed Task Diagnosis 182
9.4.2 Discretionary Task Diagnosis 183
9.5 Delivery Mechanisms for Diagnosis in Prescribed Tasks 184
9.5.1 Computer-Based Training Diagnosis 184
9.5.2 Work Task Diagnosis 185
9.5.3 Agent-Based Diagnosis 186
9.6 Delivery Mechanisms for Diagnosis in Discretionary Tasks 187
9.6.1 Discretionary Simulations 187
9.6.2 Consultant Agents 188
9.7 Conclusion 189
References 190
Part III Comparison and Empirical Testing Strategies 192
Intermezzo 3 -- The Inner Workings of Knowledge and Its Structure: Reasoning, Comparison, Testing, Evaluation, Decision,andAc 192
10 Graphs and Networks 194
10.1 Graphs as Representations of Binary Relations 194
10.2 Graphs and Matrices 197
10.3 Connectivity 199
10.4 Graph Isomorphism 200
10.5 Networks 201
10.6 Drawing Graphs 203
References 204
11 Abductive Reasoning and Similarity: Some ComputationalTools 206
11.1 Introduction 206
11.1.1 Abductive Reasoning 207
11.1.2 The Importance of Novelty 210
11.1.3 Approaches to Understanding the Generation of Hypotheses 211
11.1.4 Optimizing Versus Satisficing 212
11.2 Generating and Evaluating Hypotheses 213
11.2.1 Constraints on Abduction 213
11.2.2 Similarity in Abductive Inference 216
11.3 Random Vectors and Pathfinder Networks as Aids for Abduction from Text 217
11.4 Predicting Discoveries 223
11.5 Conclusions 225
References 225
12 Scope of Graphical Indices in Educational Diagnostics 229
12.1 Introduction 229
12.2 Graphs as External Knowledge Representation 231
12.2.1 Basics of Graph Theory 232
12.2.2 Measures of Graph Theory 232
12.2.3 Measures Beyond Graph Theory 235
12.2.4 Implementation of Graphical Indices for Educational Diagnostics 236
12.3 Empirical Studies 237
12.3.1 Development of Cognitive Structures 238
12.3.2 Feedback for Improving Expert Performance 241
12.3.3 Between-Domain Distinguishing Features of Cognitive Structures 244
12.4 Conclusion 245
References 247
13 Complete Structure Comparison 251
13.1 Knowledge and Structure 251
13.2 Retracing Knowledge Structure 253
13.3 Completeness and Explanatory Power 254
13.4 Simple Structures and a Preliminary Structural Notation 255
13.4.1 Complete Structural Traces 257
13.4.2 Downtrace 259
13.4.3 Structural Matching Similarity Measure 261
13.5 Studies 1 and 2: Trace-Based Structural Complexity Measure 261
13.5.1 Methods 262
13.5.2 Study 1: In the Field of Learning and Instruction 263
13.5.3 Study 2: In the Field of Economics 264
13.5.4 Discussion of Studies 1 and 2 265
13.6 Study 3: Technological Study on the Sensitivity of Structural Matching 265
13.6.1 Methods 265
13.6.2 Results 266
13.6.3 Post Hoc Analysis 267
13.6.4 Discussion of Study 3 267
13.7 Study 4: Empirical Study on the Semantic Interference with Structural Matching 268
13.7.1 Methods 268
13.7.2 Results 269
13.7.3 Post Hoc Analysis 270
13.7.4 Discussion of Study 4 271
13.8 Comparison to Heuristic Measures of Structure 271
13.9 Conclusion 272
References 273
Part IV Application of Obtained Results 275
Intermezzo 4 -- Using Knowledge to Support Knowing 275
14 Computer-Based Feedback for Computer-Based Collaborative Problem Solving 277
14.1 Introduction 277
14.1.1 Effects of Visual and Verbal Feedback 281
14.1.2 Methodology 282
14.1.3 Networked Knowledge Mapping System 282
14.1.4 Simulated World Wide Web Environment 283
14.1.5 Feedback 284
14.1.6 Participants 284
14.2 Measures 285
14.2.1 Group Outcome Measures 285
14.2.2 Information Seeking and Feedback Behavior Measures 285
14.2.3 Teamwork Process Measures 286
14.3 Procedure 286
14.3.1 Teamwork Questionnaire 287
14.3.2 Task Instructions and Search Strategies Training 287
14.3.3 Collaborative Group Task 1 287
14.3.4 After-Action Review Feedback 287
14.3.5 Collaborative Group Task 2 289
14.3.6 Debriefing 289
14.4 Data Analysis 289
14.5 Results and Discussion 289
14.5.1 The Effect of AAR on Team Map Scores 289
14.5.2 The Effect of AAR on Search Scores 290
14.5.3 The Effect of AAR on Communication Scores 291
14.6 Summary and Conclusions 292
14.7 Summary of Chapter 292
References 293
15 Modeling, Assessing, and Supporting Key Competencies Within Game Environments 296
15.1 Introduction 296
15.1.1 Purpose 297
15.1.2 Where We Are 298
15.1.2.1 Disengaged Students 298
15.1.2.2 The Shrinking World 298
15.1.3 Where We Should Be Heading 299
15.2 Assessment Methodology: Evidence-Centered Design 300
15.2.1 ECD Models 300
15.2.1.1 Competency Model 301
15.2.1.2 Evidence Model 301
15.2.1.3 Task Model 301
15.2.1.4 Design and Diagnosis 302
15.2.2 Stealth Assessment 302
15.2.3 Systems Thinking 303
15.2.3.1 Definitions of Systems Thinking 303
15.2.3.2 Systems Thinking and Its Role in Education 304
15.2.3.3 The Competency Model of Systems Thinking 304
15.3 Application of the Stealth Assessment Approach 308
15.3.1 Quest Atlantis: Taiga Park 308
15.3.2 ECD Models Applied to Taiga 310
15.3.2.1 Tools to Automatically Assess Causal Diagrams 313
15.3.2.2 Adding Stealth Assessment to Taiga 315
15.4 Summary and Discussion 319
References 322
16 A Methodology for Assessing Elicitation of Knowledge in Complex Domains: Identifying Conceptual Representations of Ill-Structured Problems in Medical Diagnosis 325
16.1 Introduction 325
16.1.1 Assessing Learning in Complex Domains 325
16.1.2 Assessing the Ability to Solve Ill-Defined Problems 327
16.1.3 Assessing Progress in Complex Problem Solving in DEEP 328
16.2 Methods 329
16.2.1 Research Design and Questions 329
16.2.2 Research Methodology 330
16.2.3 Problem Scenarios 330
16.2.4 Participants 330
16.2.5 Data Collection Process 332
16.2.6 Data Analysis 333
16.3 Results and Analysis 336
16.3.1 Comparisons of Expert and Novice Responses 339
16.4 Conclusion 342
16.4.1 Medical Domain Issues 343
16.4.2 Further Work on the DEEP Tools and Analysis Methodology 344
References 345
17 Selection ofINTtie Team Interventions Based onINTtie
17.1 Introduction 349
17.2 Team Cognition 350
17.3 Team Assessment and Diagnostic Instrument 352
17.4 Data Collection and Analysis 353
17.5 Interventions 355
17.5.1 Intervention Decision Making 356
17.5.1.0 Phase 1: Determining the Need for Consensus-Building (CB) Interventions 357
17.5.1.0 Phase 2: Determining the Need for Team Improvement Planning (TIP) Interventions 359
17.5.2 Intervention Types 360
17.5.2.0 Consensus Building (CB) Interventions 360
17.5.2.0 Team Improvement Planning (TIP) Interventions 361
17.5.3 Intervention Focusing on Consensus Building and Team Improvement Planning 362
17.6 Extension of TADI 363
17.7 Application of the TADI Measures for Selection of Team Interventions 363
17.7.1 Consider the TADI Similarity Measure 364
17.7.2 Look at the Range of the Similarity and Degree Measures 364
17.7.3 Examine All of the TADI Factors 364
17.7.4 Focus on One Team at a Time 365
References 366
Author Index 369
Subject Index 379