Traceable Human Experiment Design Research

Nadine Mandran, LIG/CNRS, France.

Preface ix

Introduction xi

Chapter 1 Human-Centered Computer Science Research (HCCSR) 1

1.1 Concepts and features of HCCSR 1

1.2 HCCSR: science of the artificial 6

1.3 Difficulties in building and evaluating HCCSR instruments 8

1.4 Conclusion 10

Chapter 2 THEDRE Principles 13

2.1 Introduction 13

2.2 Epistemological paradigms 13

2.2.1 Four paradigms 14

2.2.2 What are the reasoning methods? 18

2.2.3 How is the value or validity of knowledge appreciated? 19

2.2.4 Conclusion 22

2.3 Data production and analysis methods 24

2.3.1 Objective of qualitative and quantitative methods 24

2.3.2 Means to produce and analyze data 26

2.3.3 Combining methods for HCCSR 29

2.4 User-centered design 29

2.5 Traceability, continuous improvement approach and indicators 31

2.5.1 Traceability 31

2.5.2 Continuous improvement process 33

2.5.3 Quality indicators 34

Chapter 3 Four Research Methods 39

3.1 Research methods 39

3.2 Design-based research (DBR) 43

3.2.1 Foundations 43

3.2.2 Characteristics of DBR 44

3.2.3 DBR and activatable tools (TEL) 47

3.2.4 Summary of the DBR method 49

3.3 Design science research for information science 51

3.3.1 Foundations 51

3.3.2 Initial model [HEV 04] 52

3.3.3 Improvements to the initial model 56

3.3.4 Summary of the design science method and its extensions 62

3.4 Action research and action design research for research in SE and IS 64

3.4.1 Foundations of the action research 64

3.4.2 Action design research [SEI 11] 66

3.4.3 Summary of the design action research method for SE 69

3.5 Dialogical model for research in management sciences 71

3.5.1 Paradigm and knowledge validation 72

3.5.2 Characteristics of the dialogical model 73

3.5.3 Summary of the dialogical model 76

3.6 Summary of the methods studied 79

Chapter 4 THEDRE: Theoretical Model 83

4.1 A traceable method in HCCSR: paradigm and formalism 83

4.2 Pragmatic constructivism for the THEDRE method 85

4.3 Concepts and comprehensive view of the THEDRE method 88

4.3.1 Terminology for the THEDRE method 88

4.3.2 Actors and roles in the THEDRE method 89

4.3.3 Comprehensive view of the THEDRE method 90

4.4 Structure and traceability in the THEDRE method 93

4.4.1 Process structure in the THEDRE method 93

4.4.2 PDCA cycle for structuring the THEDRE method 93

4.4.3 Tasks and blocks for dividing processes into sub-processes 94

4.4.4 Traceability in the THEDRE method 95

4.4.5 Traceability indicators in the THEDRE method 96

4.4.6 Formalism of indicators in the THEDRE model 98

4.5 THEDRE: an adaptable language 100

4.5.1 Construction language for research processes 100

4.5.2 Concepts and relationships for using the THEDRE formalism 100

4.5.3 Graphical notations of the THEDRE method 102

4.6 Conclusion on the THEDRE process language 107

Chapter 5 THEDRE Implementation 109

5.1 Putting THEDRE into practice 109

5.1.1 “Research planning” sub-process 111

5.1.2 “Experiment” sub-process 123

5.1.3 “Control” sub-process 138

5.1.4 The “building and decision-making” sub-process 140

5.1.5 Conclusion 144

5.2 Description of guides provided by THEDRE 145

5.2.1 “Brainstorming” guide 146

5.2.2 Guide for writing the “problem” 148

5.2.3 Guide for describing activity, production, objective or data quality indicators 149

5.2.4 “Activatable tool breakdown” guide 150

5.2.5 Guide for “orchestrating experiments” 150

5.2.6 Guide for writing the “experiment protocol” 151

5.2.7 Guide for writing the “facilitation guide for conducting experiments” 153

5.2.8 Guide for “writing an interview guide or questionnaire” 155

5.2.9 “Specifications” guide for activatable tool and components 160

5.2.10 Guide for “effective use cases” 161

5.2.11 Guide for “experiment synthesis” 162

5.2.12 Guide to “capitalizing data and analysis scenarios” 163

5.2.13 Logic diagram for choosing data production methods 164

5.2.14 Deliverables: facilitators for multidisciplinarity 168

5.3 Conclusion 169

Chapter 6 Construction and Evaluation 171

6.1 The working context and THEDRE construction method 171

6.2 Continuous construction of the THEDRE method 172

6.2.1 IT and IDM systems 173

6.2.2 Human–computer interface (HCI) 180

6.2.3 User tests for innovative interfaces: 2008–2014 186

6.2.4 Technologies for enhancing learning (TEL) 187

6.2.5 Engineering multi-agent systems (MAS) 192

6.2.6 Conclusion on building THEDRE 195

6.3 Evaluation of the THEDRE method 197

6.3.1 Evaluation of language and the experimental process 197

6.3.2 Conclusions 202

6.3.3 Evaluation of the usability of the guides suggested in the THEDRE method 204

Conclusions and Perspectives 209

Appendices 215

Appendix 1 Evaluation of THEDRE Modeling Language 217

Appendix 2 Evaluation of the Usability of THEDRE Guides 223

Glossary 231

Bibliography 235

Index 247