Analyzing Interactions in CSCL (eBook)

Analyzing Interactions in CSCL 3
Acknowledgements 7
Introduction 9
Contents 15
Contributors 19
Part I: Understanding Group Processes 23
Chapter 1: A Complexity-Grounded Model for the Emergence of Convergence in CSCL Groups 24
1.1 Introduction 24
1.1.1 Unpacking Emergent Behavior: Emergent Simplicity Versus Emergent Complexity 26
1.1.2 Purpose 27
1.2 Methodology 27
1.2.1 Research Context and Data Collection 27
1.2.2 Procedure 28
1.2.3 Hypothesizing Simple Rules 28
1.2.4 Operationalizing Convergence 29
1.3 Results 29
1.3.1 Interpreting Fitness Curves 30
1.3.2 Relationship between Convergence and Group Performance 31
1.3.3 Comparing Convergence with Other Commonly-Used Interactional Predictors 34
1.4 Discussion 35
1.4.1 Implications for Scaffolding 35
1.4.2 Implications for Methodology: The Temporal Homogeneity Assumption 36
1.4.3 Implications for Theorizing CSCL Groups as Complex Systems 37
1.4.4 Some Caveats and Limitations 38
1.4.5 Future Directions 39
1.5 Conclusion 41
References 41
Chapter 2: Analyzing the Contextual Nature of Collaborative Activity 45
2.1 Taking into Account the Context of Activity in Research on Collaboration: Theoretical Considerations 45
2.2 Evaluating Collaborative Activity in Its Context 47
2.2.1 Discursive Approach to Studying Context in Students’ Collaborative Activity: The Case of Face-To-Face Activity 47
2.2.1.1 Different Contexts in Meaning-Making 48
2.2.2 Contextual Process of Collaborative Knowledge Construction: The Case of Asynchronous Web-Based Discussion 54
2.2.2.1 Analyzing Collaborative Knowledge Construction in Its Specific Context 56
2.2.2.2 Combining Quantitative and Qualitative Analysis 60
2.3 Discussion 63
References 64
Chapter 3: Understanding Learners’ Knowledge Building Trajectory Through Visualizations of Multiple Automated Analyses 67
3.1 Introduction 68
3.2 Indicators for Knowledge Building Advancement in the Literature 69
3.2.1 ATK Indices as Indicators for Knowledge Building 70
3.2.2 Questioning and Level of Inquiry 71
3.2.3 Argumentation as Indicative of Inquiry and Engagement in Discourse 73
3.3 A Study to Identify Features of More Productive Discourse Threads 74
3.4 Methods and Tools Used in this Study 76
3.4.1 Scaffold Supports Used 78
3.4.2 Discourse Acts Related to Argumentation and Questioning 78
3.4.3 Domain-Specific Topics and Associated Keywords and Text Patterns 79
3.4.4 Information Visualization 79
3.5 The Context and Basic Quantitative Informationof the Discourse Corpora Analyzed 83
3.6 Comparing Corpora Using Discourse Markers 85
3.7 Comparing Scaffolds Used in the Discourses 87
3.8 Comparing the Discussion Content of the Discourses 89
3.9 Integrating the Different Comparisons: Towards a Methodology for Understanding Learners’ Knowledge Building Trajectory 98
References 99
Chapter 4: Representational Tools for Understanding Complex Computer-Supported Collaborative Learning Environments* 103
4.1 Visual Representations for Understanding CSCL 105
4.2 CORDTRA Diagrams 106
4.3 Example I: CORDTRA Analysis for Asynchronous CSCL 107
4.3.1 Context for Study: The STELLAR Learning Environment 108
4.3.2 CORDTRA Analysis 108
4.4 Example II: CORDTRA for Synchronous CSCL 115
4.4.1 CORDTRA Analysis 117
4.5 Discussion 123
References 124
Chapter 5: How to Study Group Cognition 127
5.1 The Need for a New Science of Group Cognition 127
5.2 Designing a Testbed for Studying Group Cognition 131
5.3 Studying Group Cognition 133
5.3.1 Group Cognition in a Virtual Math Team (Research Question) 133
5.3.2 Non-laboratory Experimental Design (External Validity) 134
5.3.3 Data Collection at the Group Level of Description (Unit of Analysis) 134
5.3.4 Instrumentation and Data Formats (Objectivity) 135
5.3.5 Collaborative Data Sessions (Reliability) 136
5.3.6 Describing Social Practices (Generalizability) 139
5.4 Conceptualizing Group Cognition 141
5.4.1 Proposal-Driven Sustained Group Activity 142
5.4.2 The Social Order of Group Cognition 144
5.5 How We Study Group Cognition 146
References 147
Chapter 6: Thinking about Methods to Capture Effective Collaborations 151
6.1 Capturing the Influence of Context 152
6.2 Representing Timing 153
6.3 Calculating When Timing Matters 155
6.4 Capturing Community Knowledge Building 157
6.5 Designing to Support (and Capture) Collaboration 158
6.6 Where Do We Go from Here? 160
References 161
Part II: Understanding Learning Within Groups 163
Chapter 7: Analyzing Collaborative Processes and Learning from Hypertext Through Hierarchical Linear Modelling 164
7.1 Introduction 164
7.2 Purpose of the Study 165
7.3 Research Context: Integrating CoMPASS in the Science Classroom 166
7.3.1 Technological Affordances of CoMPASS 166
7.3.2 Participants 167
7.3.3 Procedures 168
7.4 Providing Metanavigation Support 168
7.5 Data Sources and Measures 169
7.5.1 Individual Measures 170
7.5.2 Group Measures 171
7.5.3 Investigations and Data Analyses 171
7.6 Results 173
7.6.1 Predicting Connection Ratio in the Concept Map Test 173
7.6.2 Predicting Concept Ratio in the Concept Map Test 174
7.7 Conclusions 176
References 177
Chapter 8: Analyzing Collaborative Interactions with Data Mining Methods for the Benefit of Learning 179
8.1 Introduction 179
8.2 Temporality in Groups: Development and Change 180
8.2.1 Phases of Group Development 181
8.2.2 Adaptation to Change 182
8.2.3 Groups as Activity Systems 183
8.2.4 Learning from Experience 184
8.2.5 Conclusions 185
8.3 Methods for Process Analysis 186
8.3.1 Conceptualizing Process in Terms of Event Sequences 187
8.3.2 Accounting for Event Sequences with Patterns and Models 188
8.4 Sequential Pattern Mining in Asynchronous Interaction Data 189
8.4.1 Top-Down, Theory Driven Data Mining on Traces of Group Interaction 189
8.4.2 Bottom-up Discovery of Patterns 190
8.4.3 Synergies from Modeling Groups and Individuals 192
8.4.4 Abstraction from Data Traces to Meaningful Sequences 193
8.5 Mining for Process Models Based on Chat Data 194
8.5.1 Discrete Event Models as a Model Class for Activity Systems 195
8.5.2 Discovering Discrete Event Models 196
8.5.2.1 Heuristic Process Mining 198
8.6 Discussion 199
References 201
Chapter 9: Multilevel Analysis in CSCL Research 204
9.1 Introduction 205
9.2 Multilevel Analysis: A ‘New’ Methodological Approach in CSCL Research 205
9.3 The Problems CSCL Researchers Encounter 207
9.3.1 Hierarchically Nested Datasets 207
9.3.2 Nonindependence of Dependent Variables 207
9.3.3 Differing Units of Analysis 209
9.4 Common Analysis Strategies 210
9.4.1 Ignoring Nonindependence 210
9.4.2 Aggregating or Disaggregating Data 211
9.4.3 Multilevel Analysis 212
9.5 Illustration of Problems and Analysis Strategies 213
9.5.1 Example 1: Impact of an Awareness Tool on Online Discussion 213
9.5.2 Example 2: Influence of Representational Guidance on Student Learning 215
9.5.3 Example 3: Influence of Representational Guidance on Essay Quality 218
9.6 Conclusion and Discussion 218
References 220
Chapter 10: Sequential Analysis of Scientific Argumentation in Asynchronous Online Discussion Environments* 223
10.1 Introduction 224
10.2 Steps, Tools, and Metrics Used in Each Approach 225
10.2.1 Core Coding: Examining the Nature of Comments Found within Discourse Episodes 226
10.2.1.1 Coding the Discourse Moves of Individual Postings 226
10.2.1.2 Coding the Grounds of a Comment 227
10.2.1.3 Coding the Conceptual Quality of a Comment 227
10.2.1.4 Coding the Level of Opposition within Discourse Episodes 229
10.2.2 Using Sequential Analysis to Identify Discourse Patterns in Argumentation 229
10.3 A Sample Study and Analysis 231
10.3.1 Data Sample 232
10.3.2 Instructional Context 232
10.3.3 Two Experimental Conditions 234
10.4 Discussion of Findings with Coding Scheme Only 235
10.4.1 Conceptual Quality 236
10.4.2 Grounds Quality and Frequency of Rebuttals 236
10.4.3 Discourse Moves 236
10.4.4 Level of Opposition 237
10.5 Discussion of Findings Using Sequential Analysis in Tandem with the Core Coding Scheme 238
10.5.1 Statistical Analysis 238
10.5.2 Differences in Transitional Probabilities 240
10.5.3 Differences in the Mean Number of Responses Elicited Per Message 242
10.5.3.1 Oppositional Exchanges 242
10.5.3.2 Supportive Exchanges 242
10.6 Affordances of Using Both Methods in Tandem 244
10.7 Directions for Future Research 245
References 248
Chapter 11: Is the Whole Greater than the Sum of Its Parts? Explaining the Role of Individual Learning and Group Processes in 250
11.1 The Power of Multilevel Analysis (MLA) in CSCL Research: N Is the Answer 250
11.2 The Sum Is Better than Its Parts 252
11.3 Mirrors for Metacognition: Visual Aids to See Oneself and Others in the Context of Problem Solving 253
11.4 From Temporal Data Mining Techniques to Sequential Pattern Recognition 255
11.5 Where Do We Go from Here? 256
References 257
Part III: Frameworks for Analyzing Interaction in CSCL 259
Chapter 12: Quantifying Qualities in Collaborative Knowledge Construction: The Analysis of Online Discussions 260
12.1 Multidimensional Approach for the Qualitative Coding of Online Discussions (MAQCOD) 261
12.1.1 The Initial Research Question 262
12.1.1.1 Illustrative Example 262
12.1.2 Definition of Data Structure and Rules for Coding 265
12.1.2.1 Categories and Scaling 266
12.1.2.2 Additional Dimensions Addressing Methodological Questions 268
12.1.3 Segmentation 269
12.1.3.1 Skipping Segmentation 272
12.1.4 Training of Coders 272
12.1.4.1 Training for High Objectivity (Without a Loss of Validity) 273
12.1.4.2 Handling Biased Data 274
12.1.5 Aggregation and High-Level Analyses 275
12.1.5.1 Tracing Knowledge 276
12.1.5.2 Quantifying Mutual Influence 276
12.1.5.3 Visualising Mutual Inf.luence 277
12.2 Automated Analyses of Online Discussions Using MAQCOD with Natural Language Processing 277
12.3 Summary and Conclusion 278
References 279
Chapter 13: An Interaction-Aware Design Process for the Integration of Interaction Analysis into Mainstream CSCL Practices 282
13.1 Introduction 283
13.2 Overview and Model of the Interaction Analysis Process 284
13.3 Main Problems for the Integration of Interaction Analysis into Mainstream CSCL Practices 286
13.3.1 An Illustrating Example: The Mosaic Experience 286
13.3.2 A Data-Driven Analysis of Problems Regarding Integration of IA Tools and CSCL Environments 288
13.4 Towards an Integrated Perspective on Learning and Analysis Activities in CSCL 293
13.4.1 Design-Driven Solutions 293
13.4.2 Technology-Driven Solutions 297
13.5 Conclusions 300
References 302
Chapter 14: A Framework for Assessment of Student Project Groups On-Line and Off-Line 305
14.1 Introduction 306
14.2 Theoretical Framework on Group Processesin Project Teams 309
14.3 Study 1: Interviews with Instructors 310
14.3.1 Method: Data Collection and Analysis 311
14.3.2 Results 313
14.3.2.1 Personal Goal Setting and Group Goal Setting 315
14.3.2.2 Personal Progress and Group Progress 316
14.3.2.3 Knowledge Contribution and Group Knowledge Building 317
14.3.2.4 Participation and Division of Labor 318
14.3.2.5 Team Player and Interpersonal Dynamics 318
14.4 Study 2: Project Group Observations 319
14.4.1 Method 319
14.4.2 Results 320
14.5 Study 3: Developing Technology for Automatic Monitoring of Group Processes 321
14.5.1 Automatic Assessment from Message Board Data 322
14.5.2 Automatic Assessment from Speech 323
14.6 Discussion 325
14.7 Conclusions and Future Work 327
References 328
Chapter 15: Analyzing Productive Interactions in CSCL: Collaborations, Computers and Contradictions 330
15.1 Introduction 331
15.2 General Trends in Methods 331
15.3 Case Studies 333
15.4 Methods of Data Collection 334
15.5 Analysis 335
15.5.1 Case 1: Running in the Rain 336
15.5.2 Context 337
15.5.3 Case 2: Gameshow 338
15.5.4 Discussion: Dealing with Complex Learning Settings 340
15.5.5 Case 3: Mobile Collaborative Learning in Formal and Informal Learning Settings/Personal Inquiry 342
15.6 Conclusions 346
References 347
Chapter 16: Tracing Interaction in Distributed Collaborative Learning 351
16.1 Introduction 351
16.2 Theoretical Assumptions 353
16.3 Data Requirements 354
16.4 Analytic Challenges 355
16.4.1 The Distributed Nature of the Data 356
16.4.2 The Contingent Nature of Human Behavior 356
16.4.3 The Meaning of Nonverbal Behavior 357
16.4.4 Selective Attention to Large Data Sets 357
16.4.5 Multi-scale Phenomena 357
16.5 Analyzing Distributed Interaction with Contingency Graphs 358
16.5.1 Contingency Graphs 358
16.5.1.1 Vertices: Events 358
16.5.1.2 Arcs: Contingencies 358
16.5.1.3 Addressing the Challenges 359
16.5.2 Example Analysis 360
16.5.2.1 Source of Data 361
16.5.2.2 Log File Description 362
16.5.2.3 Contingency Graph Construction 362
16.5.2.4 Contingency Graph Analysis 363
Selective Attention: Identifying Convergent Conclusions 363
Tracing: Subgraph Building Reveals Non Verbal Interaction Pattern 363
Micro Analysis: Indexing Video to Correlate Individual and Social Phenomena 365
16.5.2.5 Analytic Rationale for Using the Contingency Graph 370
16.5.2.6 Scaling up 371
16.6 Remaining Challenges 371
16.6.1 The Distributed Nature of the Data 371
16.6.2 The Contingent Nature of Human Behavior 371
16.6.3 The Meaning of Nonverbal Behavior 372
16.6.4 Selective Attention to Large Data Sets 372
16.6.5 Multi-scale Phenomena 373
16.7 Broader Implications 373
References 374
Chapter 17: Analyzing Collaborative Interactions Across Domains and Settings: An Adaptable Rating Scheme.* 377
17.1 The Original Rating Scheme1 379
17.1.1 Aspects of Collaboration Quality 380
17.1.1.1 Communication 380
17.1.1.2 Joint Information Processing 381
17.1.1.3 Coordination 382
17.1.1.4 Interpersonal Relationship 382
17.1.1.5 Motivation 383
17.1.2 Applying the Original Rating Scheme 383
17.1.3 Empirical Evaluation of the Rating Scheme 384
17.2 Adaptation of the Rating Scheme 385
17.2.1 Adaptation Process 386
17.2.1.1 Redefining the Rating Dimensions 386
17.2.1.2 Adapting the Rating Handbook 388
17.2.2 Empirical Evaluation of the Adapted Rating Scheme 388
17.2.3 Giving Adaptive Feedback Based on Collaboration Quality Assessment 390
17.3 Implementing the Rating Scheme in ActivityLens 391
17.3.1 Adaptation of ActivityLens 391
17.3.2 Using ActivityLens to Analyze Process Datafrom Algebra Study 392
17.4 Implications for Practitioners 394
17.4.1 Adapting the Dimensions to Further Settings 395
17.4.2 Concluding Thoughts on When to Apply a Rating Scheme in the Assessment of Collaboration Quality 396
References 397
Chapter 18: Analytical Frameworks for Group Interactions in CSCL Systems 401
18.1 Introduction 401
18.2 Objectives of Analytical Frameworks for Group Interactions 401
18.2.1 Linking Process Quality and Knowledge Construction 401
18.2.2 Mediating and Transforming Learning and Teaching with Technology 402
18.2.3 Supporting Instructors 402
18.2.4 Measuring the Quality of Collaboration 402
18.2.5 Defining the Process of Interaction Analysis 403
18.2.6 Making Interaction Apparent 403
18.2.7 Summary of Objectives for Analytical Frameworks 403
18.3 A Selection of Theoretical Assumptions Made by Authors of Analytical Frameworks for Group Interactions 404
18.3.1 Positive Relations between Process Features and Knowledge Construction 404
18.3.2 Positive and Negative Relations between Process Features and Context 405
18.3.3 Predictive Relations between the Flow of Language Communication and Group Difficulties 405
18.3.4 Definitive Relations between Nine Different Dimensions of the Collaborative Process and the Quality of Collaboration 406
18.3.5 Interdependent Relations between the CSCL Environment, Its Context of Use and the Interaction Analysis Tools 407
18.3.6 Contingency Relations between the Trajectories of Intra- and Inter-subjective Meaning Making and the Media Participa 408
18.3.7 Summary of Selected Theoretical Assumptions 409
18.3.8 Unit of Analysis: Individual, Group or Both? 410
18.4 A Selection of Methodological Assumptions Made by Authors of Analytical Frameworks for Group Interactions 411
18.4.1 Origins of Research 412
18.4.2 Tensions between Methodological Constraints and Views on Interaction 413
18.4.3 Is Technology a Variable? 413
18.4.4 Top-down vs. Bottom Up? 414
18.4.5 Summary of Selected Methodological Assumptions 415
18.5 Remaining Challenges 416
18.5.1 Automating Assessing and/or Intervening 416
18.5.2 Assessing Quality of Processes 416
18.5.3 Research Questions, Data, Methods: Interdependencies 417
18.5.4 Interdependencies of CSCL Systems and Interaction Analysis Software 418
18.6 Conclusions 418
References 419
Index 422