Elicitation (eBook)

Luis C. Dias obtained a degree in Informatics Engineering from the School of Science and Technology at the University of Coimbra in 1992, a Ph.D. in Management by the University of Coimbra in 2001, and Habilitation in Decision Aiding Science in 2013 in the same university. He is currently Associate Professor and Vice-Dean for Research the Faculty of Economics, University of Coimbra (FEUC), where he has been teaching courses on decision analysis, operations research, informatics, and related areas. He held temporary invited positions at the University Paris-Dauphine and the University of Vienna. Luis is also a researcher at the CeBER and INESC Coimbra R&D centers, a member of the coordination board of U.Coimbra’s Energy for Sustainability Initiative, and currently a Vice-President of APDIO, the Portuguese Operational Research Society. He is on the Editorial Board of the EURO Journal on Decision Processes and Omega. His research interests include multicriteria decision analysis, performance assessment, group decision and negotiation support, decision support systems, and applications in the areas of energy and environment.John Quigley has a Bachelor of Mathematics in Actuarial Science from the University of Waterloo, Canada and a PhD in Management Science from the University of Strathclyde, where he is currently Professor.  He is an Industrial Statistician with extensive experience in elicitation of expert judgment to support model development and quantification through subjective probability distributions, having worked closely over the past 25 years with various engineering organizations on problems concerned with risk and reliability.  John has been involved in consultancy and applied research projects with, for example, Aero-Engine Controls, Rolls Royce, Airborne Systems, BAE SYSTEMS and the Ministry of Defense (MOD). His collaborative work on Bayesian model development as part of the Reliability Enhancement Methods and Models (REMM) project is included in the industry standard for reliability growth analysis methods.  John is a tutor for the European Food Safety Agency (EFSA) on Expert Knowledge Elicitation (EKE) as well as being an Associate of the Society of Actuaries, a Chartered Statistician, and a member of the Safety and Reliability Society. Alec Morton has degrees from the University of Manchester and the University of Strathclyde. He has worked for Singapore Airlines, the National University of Singapore, and the London School of Economics, has held visiting positions at Carnegie Mellon University in Pittsburgh, Aalto University in Helsinki, and the University of Science and Technology of China (USTC) in Hefei, and has been on secondment at the National Audit Office. His main interests are in decision analysis and health economics.  Alec has been active in the INFORMS Decision Analysis Society and the OR Society. He is on the Editorial Board of Decision Analysis and is an Associate Editor for the EURO Journal on Decision Processes, the Transactions of the Institute of Industrial Engineers, and OR Spectrum. His research has won awards from the International Society for Pharmacoeconomics and Outcomes Research and the Society for Risk Analysis and from the the INFORMS Decision Analysis Society publication award.

Contents 5
About the Editors 7
1 Elicitation: State of the Art and Science 9
1.1 Conceptual Background 10
1.2 The Need for and Barriers to Elicitation 12
1.2.1 The Need for Elicitation of Judgement 12
1.2.1.1 Case 1. Swine Flu 13
1.2.1.2 Case 2. Airport Location 13
1.2.1.3 Case 3. Assessment of the Risk of Earthquake 14
1.2.1.4 Case 4. Radioactive Waste Management 15
1.2.2 Why do People Resist Expressing Their Uncertainty and Values Quantitatively? 15
1.2.2.1 Deep Uncertainty Case 1: Deepwater Horizon 16
1.2.2.2 Deep Uncertainty Case 2: The Fukushima Disaster 16
1.2.2.3 Sacred Values Case 1: the Approval of New Drugs 16
1.2.2.4 Sacred Values Case 2: The Concept of “Capability” in Military Planning 17
1.3 Overview of the Book 18
1.4 Conclusions and Future Directions 20
References 21
2 Elicitation in the Classical Model 23
2.1 Introduction 23
2.2 Classical Model Basics 24
2.2.1 Elicitation Questions 25
2.2.2 Calibration Score 26
2.2.3 Distribution and Discrimination of Calibration Score 30
2.2.4 Information Score 31
2.2.5 Weights 34
2.2.5.1 Global Weights 34
2.2.5.2 Itemized Weights 35
2.2.5.3 Optimized Weights 35
2.2.6 Summary 36
2.3 Finding Seed Variables 36
2.4 Elicitation Styles 40
2.5 Discussion 42
References 43
3 Validation in the Classical Model 45
3.1 Introduction: Why Validate? 45
3.2 Mathematical Pooling: Harmonic, Geometric and Arithmetic Means 49
3.2.1 Analysis 49
3.2.2 Performance on Real Expert Data 50
3.3 Review of Expert Judgment Cross Validation Research 54
3.3.1 ROAT Bias 54
3.4 Post 2006 Data Sets and Applications Documentation 61
3.5 Conclusion 61
References 65
4 SHELF: The Sheffield Elicitation Framework 68
4.1 Introduction 68
4.2 The Elicitation Framework 70
4.2.1 Exercise Specification 71
4.2.2 Expert Selection 73
4.2.3 Training in EKE Process 75
4.2.4 Information Sharing 76
4.2.5 Individual Judgements 78
4.2.6 Distribution Fitting 81
4.2.7 Aggregation of Distributions 82
4.2.8 Feedback on Distributions 84
4.2.9 Completing the Exercise 85
4.3 Notable Applications of the Framework 86
4.3.1 Healthcare and Medicine 86
4.3.2 Environmental Sciences 87
4.3.3 Other Applications 88
4.4 Extensions of the Framework 89
4.4.1 Elicitation for Multivariate Quantities 90
4.4.2 Distributed Experts 91
4.5 Discussion 96
References 98
5 IDEA for Uncertainty Quantification 101
5.1 Introduction 101
5.2 The IDEA Protocol 103
5.2.1 Eliciting Probabilities 104
5.2.2 Eliciting Quantiles of Probability Distributions 105
5.3 Data Analysis 106
5.3.1 Measures of Performance 108
5.3.1.1 Accuracy 108
5.3.1.2 Calibration 109
5.3.1.3 Informativeness 111
5.3.1.4 Correlated Expert Judgements 112
5.3.2 The Merits of Discussion 113
5.3.3 Prior Performance as a Guide to Future Performance 114
5.4 A Guide to Facilitating the IDEA Elicitation Protocol 115
5.4.1 Preparing for an Elicitation 116
5.4.1.1 Key Documents 116
5.4.1.2 The Questions 117
5.4.1.3 The Experts 117
5.4.1.4 The Facilitator 118
5.4.2 Implementing the IDEA Protocol 118
5.4.2.1 The Initial Meeting 118
5.4.2.2 The Elicitation 119
5.5 Discussion 120
References 121
6 Elicitation and Calibration: A Bayesian Perspective 124
6.1 Introduction 124
6.2 Context 125
6.3 The Bayesian Approach to Structured Expert Judgement 128
6.4 Survey of Bayesian Models for Structured Expert Judgement 132
6.5 Practical Procedures 137
6.6 Conclusions 142
References 143
7 A Methodology for Constructing Subjective Probability Distributions with Data 146
7.1 Introduction 146
7.2 On the Nature of the Problem 147
7.2.1 Motivating Industry Challenges 147
7.2.2 Generalisation of the Problem 148
7.2.3 Implications of Inference Principles for Elicitation 149
7.2.4 Principles of Empirical Bayes Inference 151
7.3 General Methodological Steps 152
7.3.1 Characterise the Population DGP 152
7.3.2 Identify Candidate Sample DGPs Matching Population 153
7.3.3 Sentence Empirical Data to Construct Sample DGPs 154
7.3.4 Select Probability Model for Population DGP 155
7.3.5 Estimate Model Parameters to Obtain Prior Distribution 155
7.4 Example Applications of the Elicitation Process 155
7.4.1 Assessing Uncertainty in Supplier Quality 156
7.4.1.1 Characterise the Population DGP 156
7.4.1.2 Identify Candidate Sample DGPs Matching Population 157
7.4.1.3 Sentence Empirical Data to Construct Sample DGPs 158
7.4.1.4 Select Probability Model for Population DGP 159
7.4.1.5 Estimate Model Parameters to Obtain Prior Distribution 160
7.4.2 Assessing Uncertainty About Reliability of an Engineering Design 163
7.4.2.1 Characterise the Population DGP 164
7.4.2.2 Identify Candidate Sample DGP Matching Population 165
7.4.2.3 Sentence Empirical Data to Construct Sample DGPs 166
7.4.2.4 Select Probability Model for Population DGP 168
7.4.2.5 Estimate Model Parameters to Obtain Prior Distribution 169
7.5 Summary and Conclusions 170
7.5.1 Methodological Steps 170
7.5.2 Effect of Sample Size on Prior Distribution 172
7.5.3 Caveats and Challenges 172
Appendix 172
References 174
8 Eliciting Multivariate Uncertainty from Experts: Considerations and Approaches Along the Expert Judgement Process 176
8.1 Introduction 177
8.1.1 Objective and Structure of the Chapter 177
8.1.2 Dependence in the Subjective Probability Context 178
8.2 Structured Expert Judgement Processes: An Overview 178
8.3 Biases and Heuristics for Dependence Elicitation 181
8.3.1 Causal Reasoning and Inference 183
8.3.2 Biased Dependence Elicitation: An Overview 184
8.3.3 Implications of Biases for the Elicitation Process 191
8.4 Elicitation Process: Preparation/Pre-elicitation 192
8.4.1 Problem Identification and Modelling Context 192
8.4.2 Choice of Elicited Parameters 195
8.4.3 Specification of Marginal Distributions 199
8.4.4 Training and Motivation 199
8.5 Elicitation Process: Elicitation 201
8.5.1 Knowledge and Belief Structuring 201
8.5.2 Quantitative Elicitation 203
8.6 Elicitation Process: Post-elicitation 204
8.6.1 Aggregation of Expert Judgements 205
8.6.2 Feedback and Robustness Analysis 208
8.7 Conclusions 209
References 210
9 Combining Judgements from Correlated Experts 216
9.1 Introduction 216
9.2 Mathematical and Behavioural Aggregation 218
9.3 Sources of Correlation 220
9.4 Mathematical Aggregation Methods 222
9.4.1 Bayesian Methods 222
Multivariate Normal model 222
Copula Model 223
Empirical Bayes Model 223
9.4.2 Opinion Pooling Methods 224
Cooke's Classical Method 224
The Moment Method 225
Babuscia and Cheung Approach 225
Non-parametric Approach 226
9.4.3 Correlations in Mathematical Approaches 226
9.5 Correlations in Behavioural Approaches 226
9.6 Evaluation of Mathematical Approaches 231
9.6.1 Prediction 232
9.6.2 Uncertainty 234
9.7 Summary and Future Directions 239
Appendix 1 241
Appendix 2 242
Appendix 3 243
References 244
10 Utility Elicitation 246
10.1 Introduction 246
10.2 (Single Attribute) Utility Elicitation 248
10.2.1 Basic Utility Concepts 248
10.2.2 An Elicitation Protocol 249
10.2.3 Risk Attitudes and Utility Functional Forms 251
10.2.4 Behavioural Issues 255
10.3 (Multi-Attribute) Utility Elicitation 255
10.3.1 Multi-Attribute Hierarchies 255
10.3.2 Multi-Attribute Utilities 258
10.3.3 Time Dependent Utilities 260
10.3.4 An Elicitation Protocol 262
10.4 Eliciting Adversarial Preferences 264
10.5 Discussion 266
References 267
11 Elicitation in Target-Oriented Utility 270
11.1 Introduction 270
11.2 Default Decisions 273
11.3 Goals 277
11.4 Screening 281
11.5 Expectations 286
11.6 Conclusions 288
References 290
12 Multiattribute Value Elicitation 292
12.1 Background 292
12.2 Preferential Independence: A Foundational Concept of Multiattribute Value Theory 294
12.2.1 Generic Representation Theorem 294
12.2.2 Representation Theorem for the Existence of a Representing Function 294
12.2.3 Representation Theorem for the Existence of an Additive Representing Function 295
12.3 The Decision Analysis Process 298
12.3.1 Design and Planning 299
12.3.1.1 Step 1. Establish the Aims of the Analysis 299
12.3.1.2 Step 2. Identify Decision Makers, Stakeholders, and Persons with Relevant Expertise 299
12.3.1.3 Step 3. Design the Intervention 300
12.3.2 Structuring the Model 300
12.3.2.1 Step 4. Identify the Options 300
12.3.2.2 Step 5. Identify the Criteria 301
12.3.2.3 Step 6. Score the Options on the Criteria 303
12.3.2.4 Step 7. Weight the Criteria 307
12.3.3 Analysing the Model 309
12.3.3.1 Step 8. Compute Overall Rankings 309
12.3.3.2 Step 9. Conduct Sensitivity Analysis 310
12.3.4 Troubleshooting 312
12.4 Concluding Remarks 314
References 315
13 Disaggregation Approach to Value Elicitation 317
13.1 Introduction 317
13.2 A New Look on the UTA Method 320
13.2.1 Problem Statement and Notation 320
13.2.2 The UTASTAR Algorithm 321
13.2.2.1 Step 1 321
13.2.2.2 Step 2 322
13.2.2.3 Step 3 322
13.2.2.4 Step 4 322
13.3 Interactive Disaggregation and Robustness Control 323
13.3.1 Bipolar Robustness Control 323
13.3.2 Robustness Indices 324
13.3.2.1 Robustness Indices on the Disaggregation Pole 326
13.3.2.2 Robustness Indices on the Aggregation Pole 327
13.4 An Application Example 329
13.4.1 Problem Presentation 329
13.4.2 Reference Set and Preference Elicitation 330
13.4.3 Preference Disaggregation Using UTASTAR Method 331
13.4.3.1 Step 1 331
13.4.3.2 Step 2 331
13.4.3.3 Step 3 332
13.4.3.4 Step 4 332
13.4.4 Bipolar Robustness Control 333
13.4.4.1 UTASTAR Re-Activation (2nd Iteration) 334
13.4.4.2 Step 1 335
13.4.4.3 Step 2 335
13.4.4.4 Step 3 336
13.4.4.5 2nd Request for Feedback (3rd Iteration) 336
13.4.4.6 3rd Request for Feedback (4th Iteration) 339
13.4.4.7 4th Request for Feedback (5th Iteration) 341
13.4.4.8 5th Request for Feedback (6th Iteration) 342
13.5 Brief Overview of Existing Applications 344
13.6 Conclusions 347
References 348
14 Eliciting Multi-Criteria Preferences: ELECTRE Models 353
14.1 Introduction 353
14.2 Preference Models with ELECTRE 355
14.2.1 Outranking Relations for a Single Criterion 355
14.2.2 Concordance Relation 356
14.2.3 Discordance Relations 356
14.2.4 Valued Outranking Relations 358
14.2.5 Exploitation of the Outranking Relation 358
14.3 Direct Elicitation 360
14.3.1 Single-Criterion Concordance Parameters 360
14.3.2 Weights and Cutting Level 362
14.3.3 Discordance Parameters 364
14.3.3.1 Parameters Defining dj(a,b) 364
14.3.3.2 Parameters Defining dj(a,b) for Relation S'(a,b) or S''(a,b) 365
14.3.4 Profiles in Sorting Problems 366
14.4 Indirect Elicitation (Regression) 368
14.4.1 Inferring Weights and Cutting Level from S' Outranking Statements 369
14.4.2 Inferring Different Parameters for Sorting Problems 371
14.5 Elicitation Process 373
14.5.1 Elicitation Sequence 373
14.5.2 Numerical Precision 374
14.6 Concluding Remarks 376
References 377
15 Individual and Group Biases in Value and Uncertainty Judgments 380
15.1 Introduction 380
15.2 Relevant Individual Biases 381
15.2.1 Relevant Individual Cognitive Biases 381
15.2.2 Relevant Individual Motivational Biases 384
15.3 Relevant Group Biases 386
15.4 Conclusions 390
References 391
16 The Selection of Experts for (Probabilistic) Expert Knowledge Elicitation 396
16.1 Introduction 396
16.2 Part I: Defining, Identifying and Measuring Expertise 397
16.2.1 Defining Expertise 397
16.2.1.1 Expertise as Superior Knowledge and/or Ability 397
16.2.1.2 Socially Defined Expertise 398
16.2.1.3 Properties of Experts 398
16.2.1.4 Expertise Continuum 399
16.2.1.5 Granularity and Scope of Expertise 400
16.2.1.6 Types of Expertise 400
16.2.2 Identifying Expertise 403
16.2.2.1 Substantive Expertise 404
16.2.2.2 Normative Expertise 404
16.2.2.3 Social Expertise 405
16.2.3 Measuring Expertise 406
16.2.3.1 Reliability and Validity of Measurement 406
16.2.3.2 Measuring Substantive Expertise 408
16.2.3.3 Measuring Normative Expertise 413
16.2.4 The Nature of Expertise in Judgement of Uncertain Quantities 417
16.2.4.1 Judging Quantities 417
16.2.4.2 Assessing Uncertainty 421
16.2.4.3 Limits of Expertise 423
16.2.4.4 Determinants of the Quality of (Probability) Judgements 426
16.2.5 How Many Experts and How Many Judgements from Each? 427
16.2.5.1 How Many Experts? 427
16.2.5.2 How Many Judgements? 430
16.3 Part II: A Structured Approach to the Selection of Experts 432
16.3.1 The EKE Process 432
16.3.2 From Problem Identification to Long-Listing (Stage 1) 434
16.3.3 From Short-Listing to Wrap-Up (Stage 2) 438
16.3.3.1 Screening, Short-Listing, and Weighting 438
16.3.3.2 Training, Retention and Documentation 439
16.4 Conclusions 441
References 442
17 Eliciting Probabilistic Judgements for Integrating Decision Support Systems 447
17.1 Introduction 447
17.1.1 A Probabilistic IDSS: Its Genesis and Functionality 448
17.1.2 The Running Example of Food Security 450
17.2 Framing a Complex Dynamic System 453
17.3 An Agreed Picture of the Whole Probability Process 457
17.3.1 An Overarching Structure and Common Language 457
17.3.2 Defining the Features and Variables in a Problem 459
17.3.2.1 What Are the Centre's Attributes and Time Frames? 459
17.3.2.2 Who Can Inform These Attributes and How? 460
17.3.2.3 Firming Up Meaningful Inputs and Outputs 460
17.3.2.4 Iterations to Provide Causal Chains 461
17.3.2.5 Example 461
17.3.3 Listing Measurements in a Causal Order 463
17.3.4 Bayesian Networks and Dynamic Bayesian Networks 465
17.3.4.1 Defining a Graph 465
17.3.4.2 Feasible Graphical Models and Simplifying Structures 466
17.4 Bayesian Networks for a Component Model: A Case Study 470
17.4.1 Development of the Bayesian Network Structure 470
17.4.2 Eliciting Conditional Probability Tables 473
17.5 Communicating the Results 475
17.6 Quality Control of Integrating systems, Diagnostics and Robustness 477
17.7 Conclusions 478
References 479
18 Expert Elicitation to Inform Health Technology Assessment 481
18.1 Introduction 481
18.2 Representing Uncertainty in Adoption Decisions 482
18.3 Distinguishing Features of Health Care Decision Making and Requirements for Expert Elicitation 484
18.4 Methods for Expert Elicitation in Healthcare Decision Making 485
18.5 Examples of Applications in Health Care Decision Making 487
18.5.1 Negative Pressure Wound Therapy (Soares et al. 2011) 487
18.5.2 Photo Acoustic Mammography (PAM) (Haakma et al. 2014) 492
18.6 Conclusions and Requirements for Further Research 494
References 495
19 Expert Judgment Based Nuclear Threat Assessment for Vessels Arriving in the US 497
19.1 Introduction 497
19.2 Questionnaires 499
19.3 Analysis 502
19.4 Results 503
19.5 Representative Threat Predictions 508
19.6 Conclusions 509
References 510
20 Risk Assessment Using Group Elicitation: Case Study on Start-up of a New Logistics System 512
20.1 Introduction 512
20.2 North European Transport Supply System 514
20.3 Pre Workshop Preparation 515
20.3.1 Planning of Workshop Process 515
20.3.1.1 Definition of the Likelihood Scale 516
20.3.1.2 Definition of Consequence Types and Scales 516
20.3.2 Selection of Experts 518
20.4 Computer Assisted Expert Workshop 518
20.4.1 Introduction of the Workshop 519
20.4.2 Hazard Identification 520
20.4.3 Risk Estimation 521
20.4.3.1 Risk Index 522
20.4.3.2 Identification of Top Priority Risks 525
20.4.3.3 Top Priority Risks 525
20.4.4 Risk Control Ideas 526
20.4.5 Conclusion of the Workshop 526
20.5 Post Workshop Actions 526
20.6 Lessons Learned 527
References 528
21 Group Decision Support for Crop Planning: A Case Study to Guide the Process of Preferences Elicitation 529
21.1 Introduction 530
21.2 Problem Structuring 530
21.2.1 Problem Types 531
21.2.1.1 Crop Choice (Farm Level) 531
21.2.1.2 Crop Acreage 532
21.3 Case Study 533
21.3.1 Background 533
21.3.2 Process Overview 533
21.4 The Process Instantiation 535
21.5 Discussion 539
References 540