Design, Evaluation, and Analysis of Questionnaires for Survey Research

WILLEM E. SARIS, PHD, is Emeritus Professor in Methodology of the University of Amsterdam and the Universitat Pompeu Fabra, Barcelona. He is Laureate of the 2005 Descartes Prize for “Best Collaborative Research” as member of the Central Coordinating Team of the European Social Survey (ESS) and Recipient of the World Association of Public Opinion Research’s “Helen Dinerman Award” in 2009 for his lifelong contribution to the methodology of Opinion Research. Dr. Saris also received the “2013 Outstanding Service Prize” of the European Survey Research Association. IRMTRAUD N. GALLHOFER, PHD, is a linguist and was senior researcher on projects of the ESS, Research and Expertise Centre for Survey Methodology at the Universitat Pompeu Fabra, Barcelona. She is Laureate of the 2005 Descartes Prize for “Best Collaborative Research” as a member of the Central Coordinating Team of the ESS.

Preface to the Second Edition xiii

Preface xv

Acknowledgments xvii

Introduction 1

I.1 Designing a Survey 4

I.1.1 Choice of a Topic 4

I.1.2 Choice of the Most Important Variables 4

I.1.3 Choice of a Data Collection Method 5

I.1.4 Choice of Operationalization 6

I.1.5 Test of the Quality of the Questionnaire 8

I.1.6 Formulation of the Final Questionnaire 9

I.1.7 Choice of Population and Sample Design 9

I.1.8 Decide about the Fieldwork 10

I.1.9 What We Know about These Decisions 10

I.1.10 Summary 11

Exercises 12

Part I The Three-Step Procedure to Design Requests for Answers 13

1 Concepts-by-Postulation and Concepts-by-Intuition 15

1.1 Concepts-by-Intuition and Concepts-by-Postulation 15

1.2 Different Ways of Defining Concepts-by-Postulation through Concepts-by-Intuition 19

1.2.1 Job Satisfaction as a Concept-by-Intuition 19

1.2.2 Job Satisfaction as a Concept-by-Postulation 20

1.3 Summary 27

Exercises 28

2 From Social Science Concepts-by-Intuition to Assertions 30

2.1 Basic Concepts and Concepts-by-Intuition 31

2.2 Assertions and Requests for an Answer 32

2.3 The Basic Elements of Assertions 33

2.3.1 Indirect Objects as Extensions of Simple Assertions 36

2.3.2 Adverbials as Extensions of Simple Assertions 37

2.3.3 Modifiers as Extensions of Simple Assertions 37

2.3.4 Object Complements as Extensions of Simple Assertions 38

2.3.5 Some Notation Rules 38

2.4 Basic Concepts-by-Intuition 39

2.4.1 Subjective Variables 40

2.4.2 Objective Variables 47

2.4.3 In Summary 49

2.5 Alternative Formulations for the Same Concept 49

2.6 Extensions of Simple Sentences 51

2.6.1 Adding Indirect Objects 51

2.6.2 Adding Modifiers 52

2.6.3 Adding Adverbials 52

2.7 Use of Complex Sentences 53

2.7.1 Complex Sentences with No Shift in Concept 54

2.7.2 Complex Sentences with a Shift in Concept 54

2.7.3 Adding Conditions to Complex Sentences 56

2.8 Summary 56

Exercises 57

3 The Formulation of Requests for an Answer 60

3.1 From Concepts to Requests for an Answer 61

3.2 Different Types of Requests for an Answer 63

3.2.1 Direct Request 63

3.2.2 Indirect Request 66

3.3 The Meaning of Requests for an Answer with WH Request Words 69

3.3.1 “When,” “Where,” and “Why” Requests 70

3.3.2 “Who” Requests 70

3.3.3 “Which” Requests 70

3.3.4 “What” Requests 71

3.3.5 “How” Requests 72

3.4 Summary 74

Exercises 75

Part II Choices Involved in Questionnaire Design 77

4 Specific Survey Research Features of Requests for an Answer 79

4.1 Select Requests from Databases 79

4.2 Other Features Connected with the Research Goal 81

4.3 Some Problematic Requests 83

4.3.1 Double-Barreled Requests 83

4.3.2 Requests with Implicit Assumptions 84

4.4 Some Prerequests Change the Concept-by-Intuition 85

4.5 Batteries of Requests for Answers 86

4.5.1 The Use of Batteries of Stimuli 87

4.5.2 The Use of Batteries of Statements 88

4.6 Other Features of Survey Requests 92

4.6.1 The Formulation of Comparative or Absolute Requests for Answers 92

4.6.2 Conditional Clauses Specified in Requests for Answers 93

4.6.3 Balanced or Unbalanced Requests for Answers 93

4.7 Special Components within the Request 95

4.7.1 Requests for Answers with Stimulation for an Answer 95

4.7.2 Emphasizing the Subjective Opinion of the Respondent 95

4.8 Summary 96

Exercises 96

5 Response Alternatives 98

5.1 Open Requests for an Answer 99

5.2 Closed Categorical Requests 101

5.2.1 Nominal Categories 103

5.2.2 Ordinal Scales 104

5.2.3 Continuous Scales 108

5.3 How Many Categories are Optimal? 111

5.4 Summary 112

Exercises 114

6 The Structure of Open-Ended and Closed Survey Items 115

6.1 Description of the Components of Survey Items 115

6.2 Different Structures of Survey Items 118

6.2.1 Open-Ended Requests for an Answer 119

6.2.2 Closed Survey Items 120

6.2.3 The Frequency of Occurrence 124

6.2.4 The Complexity of Survey Items 125

6.3 What Form of Survey Items Should Be Recommended? 126

6.4 Summary 127

Exercises 128

7 Survey Items in Batteries 130

7.1 Batteries in Oral Interviews 131

7.2 Batteries in Mail Surveys 134

7.3 Batteries in CASI 138

7.4 Summary and Discussion 142

Exercises 144

8 Mode of Data Collection and Other Choices 146

8.1 The Choice of the Mode of Data Collection 147

8.1.1 Relevant Characteristics of the Different Modes 148

8.1.2 The Presence of the Interviewer 149

8.1.3 The Mode of Presentation 151

8.1.4 The Role of the Computer 152

8.1.5 Procedures without Asking Questions 155

8.1.6 Mixed-Mode Data Collection 155

8.2 The Position in the Questionnaire 156

8.3 The Layout of the Questionnaire 158

8.4 Differences due to Use of Different Languages 158

8.5 Summary and Discussion 159

Exercises 160

Part III Estimation and Prediction of the Quality of Questions 163

9 Criteria for the Quality of Survey Measures 165

9.1 Different Methods, Different Results 166

9.2 How These Differences Can Be Explained 173

9.2.1 Specifications of Relationships between Variables in General 173

9.2.2 Specification of Measurement Models 175

9.3 Quality Criteria for Survey Measures and Their Consequences 178

9.4 Alternative Criteria for Data Quality 181

9.4.1 Test–Retest Reliability 181

9.4.2 The Quasi-simplex Approach 182

9.4.3 Correlations with Other Variables 183

9.5 Summary and Discussion 184

Exercises 185

Appendix 9.1 The Specification of Structural Equation Models 187

10 Estimation of Reliability, Validity, and Method Effects 190

10.1 Identification of the Parameters of a Measurement Model 191

10.2 Estimation of Parameters of Models with Unmeasured Variables 195

10.3 Estimating Reliability, Validity, and Method Effects 197

10.4 Summary and Discussion 201

Exercises 202

Appendix 10.1 Input of Lisrel for Data Analysis of a Classic MTMM Study 205

Appendix 10.2 Relationship between the TS and the Classic MTMM Model 205

11 Split-Ballot Multitrait–Multimethod Designs 208

11.1 The Split-Ballot MTMM Design 209

11.1.1 The Two-Group Design 209

11.1.2 The Three-Group Design 210

11.1.3 Other SB-MTMM Designs 211

11.2 Estimating and Testing Models for Split-Ballot MTMM Experiments 212

11.3 Empirical Examples 213

11.3.1 Results for the Three-Group Design 213

11.3.2 Two-Group SB-MTMM Design 215

11.4 The Empirical Identifiability and Efficiency of the Different SB-MTMM Designs 218

11.4.1 The Empirical Identifiability of the SB-MTMM Model 218

11.4.2 The Efficiency of the Different Designs 221

11.5 Summary and Discussion 221

Exercises 222

Appendix 11.1 The Lisrel Input for the Three-Group SB-MTMM Example 222

12 MTMM Experiments and the Quality of Survey Questions 225

12.1 The Data from the MTMM Experiments 226

12.2 The Coding of the Characteristics of the MTMM Questions 229

12.3 The Database and Some Results 230

12.3.1 Differences in Quality across Countries 231

12.3.2 Differences in Quality for Domains and Concepts 234

12.3.3 Effect of the Question Formulation on the Quality 235

12.4 Prediction of the Quality of Questions Not Included in the MTMM Experiments 237

12.4.1 Suggestions for Improvement of Questions 239

12.4.2 Evaluation of the Quality of the Prediction Models 240

12.5 Summary 241

Exercises 242

Part IV Applications in Social Science Research 243

13 The SQP 2.0 Program for Prediction of Quality and Improvement of Measures 245

13.1 The Quality of Questions Involved in the MTMM Experiments 246

13.1.1 The Quality of Specific Questions 246

13.1.2 Looking for Optimal Measures for a Concept 250

13.2 The Quality of Non-MTMM Questions in the Database 252

13.3 Predicting the Quality of New Questions 256

13.4 Summary 261

Exercises 262

14 The Quality of Measures for Concepts-by-Postulation 263

14.1 The Structures of Concepts-by-Postulation 264

14.2 The Quality of Measures of Concepts-by-Postulation with Reflective Indicators 264

14.2.1 Testing the Models 265

14.2.2 Estimation of the Composite Scores 268

14.2.3 The Quality of Measures for Concepts-by-Postulation 270

14.2.4 Improvement of the Quality of the Measure 274

14.3 The Quality of Measures for Concepts-by-Postulation with Formative Indicators 276

14.3.1 Testing the Models 278

14.3.2 Estimation of the Composite Score 281

14.3.3 The Estimation of the Quality of the Composite Scores 282

14.4 Summary 283

Exercises 284

Appendix 14.1 Lisrel Input for Final Analysis of the Effect of “Social Contact” on “Happiness” 284

Appendix 14.2 Lisrel Input for Final Analysis of the Effect of “Interest in Political Issues in the Media” on “Political Interest in General” 285

15 Correction for Measurement Errors 287

15.1 Correction for Measurement Errors in Models with only Concepts-by-Intuition 287

15.2 Correction for Measurement Errors in Models with Concepts-by-Postulation 292

15.2.1 Operationalization of the Concepts 292

15.2.2 The Quality of the Measures 294

15.2.3 Correction for Measurement Errors in the Analysis 297

15.3 Summary 298

Exercises 299

Appendix 15.1 Lisrel Inputs to Estimate the Parameters of the Model in Figure 15.1 300

Appendix 15.2 Lisrel Input for Estimation of the Model with Correction for Measurement Errors using Variance Reduction by Quality for all Composite Scores 301

16 Coping with Measurement Errors in Cross-Cultural Research 302

16.1 Notations of Response Models for Cross-Cultural Comparisons 303

16.2 Testing for Equivalence or Invariance of Instruments 307

16.2.1 The Standard Approach to Test for Equivalence 307

16.3 Problems Related with the Procedure 309

16.3.1 Using Information about the Power of the Test 309

16.3.2 An Alternative Test for Equivalence 315

16.3.3 The Difference between Significance and Relevance 317

16.4 Comparison of Means and Relationships across Groups 318

16.4.1 Comparison of Means and Relationships between Single Requests for Answers 318

16.4.2 Comparison of Means and Relationships Based on Composite Scores 319

16.4.3 Comparison of Means and Relationships between Latent Variables 321

16.5 Summary 324

Exercises 325

Appendix 16.1 The Two Sets of Requests Concerning “Subjective Competence” 326

Appendix 16.2 ESS Requests Concerning “Political Trust” 327

Appendix 16.3 The Standard Test of Equivalence for “Subjective Competence” 328

Appendix 16.4 The Alternative Equivalence Test for “Subjective Competence” in Three Countries 329

Appendix 16.5 Lisrel Input to Estimate the Null Model for Estimation of the Relationship between “Subjective Competence” and “Political Trust” 331

Appendix 16.6 Derivation of the Covariance between the Composite Scores 333

References 336

Index 352