Survey Data Harmonization in the Social Sciences

Irina Tomescu-Dubrow is Professor of Sociology at the Institute of Philosophy and Sociology at the Polish Academy of Sciences (PAN), and director of the Graduate School for Social Research at PAN. Christof Wolf is President of GESIS — Leibniz Institute for the Social Sciences and Professor of Sociology at the University of Mannheim in Germany. Kazimierz M. Slomczynski is Professor of Sociology at the Institute of Philosophy and Sociology, the Polish Academy of Sciences (IFiS PAN) and Academy Professor of Sociology at the Ohio State University (OSU). He co-directs CONSIRT - the Cross-national Studies: Interdisciplinary Research and Training program at OSU and IFiS PAN. J. Craig Jenkins is Academy Professor of Sociology and Senior Research Scientist at the Mershon Center for International Security at the Ohio State University.

Preface and Acknowledgments xv

About the Editors xvii

About the Contributors xviii

1 Objectives and Challenges of Survey Data Harmonization 1
Kazimierz M. Slomczynski, Irina Tomescu-Dubrow, J. Craig Jenkins, and Christof Wolf

1.1 Introduction 1

1.2 What is the Harmonization of Survey Data? 2

1.2.1 Ex-ante, Input and Output, Survey Harmonization 3

1.3 Why Harmonize Social Survey Data? 5

1.3.1 Comparison and Equivalence 6

1.4 Harmonizing Survey Data Across and Within Countries 7

1.4.1 Harmonizing Across Countries 7

1.4.2 Harmonizing Within the Country 8

1.5 Sources of Knowledge for Survey Data Harmonization 8

1.6 Challenges to Survey Harmonization 9

1.6.1 Population Representation (Sampling Design) 10

1.6.2 Instruments and Their Adaptation (Including Translation) 10

1.6.3 Preparation for Interviewing (Including Pretesting) 11

1.6.4 Fieldwork (Including Modes of Interviewing) 11

1.6.5 Data Preparation (Including Building Data Files) 12

1.6.6 Data Processing, Quality Controls, and Adjustments 12

1.6.7 Data Dissemination 13

1.7 Survey Harmonization and Standardization Processes 13

1.8 Quality of the Input and the End-product of Survey Harmonization 14

1.9 Relevance of Harmonization Methodology to the FAIR Data Principles 15

1.10 Ethical and Legal Issues 15

1.11 How to Read this Volume? 16

References 17

2 The Effects of Data Harmonization on the Survey Research Process 21
Ranjit K. Singh, Arnim Bleier, and Peter Granda

2.1 Introduction 21

2.2 Part 1: Harmonization: Origins and Relation to Standardization 22

2.2.1 Early Conceptions of Standardization and Harmonization 22

2.2.2 Foundational Work of International Survey Programs 23

2.2.3 The Growing Impact of Data Harmonization 23

2.3 Part 2: Stakeholders and Division of Labor 25

2.3.1 Stakeholders 26

2.3.1.1 International Actors and Funding Agencies 26

2.3.1.2 Data Producers 26

2.3.1.3 Archives 27

2.3.1.4 Data Users 27

2.3.2 Toward an Integrative View on Harmonization 28

2.3.2.1 Harmonization Cost 29

2.3.2.2 Harmonization Quality 29

2.3.2.3 Harmonization Fit 30

2.3.2.4 Moving Forward 30

2.4 Part 3: New Data Types, New Challenges 31

2.4.1 Designed Data and Organic Data 31

2.4.2 Stakeholders in the Collection of Organic Data 32

2.4.2.1 Producers 32

2.4.2.2 Archives 32

2.4.2.3 Users 33

2.4.2.4 Harmonization of Organic Data 33

2.5 Conclusion 33

References 35

Part I Ex-ante harmonization of survey instruments and non-survey data 39

3 Harmonization in the World Values Survey 41
Kseniya Kizilova, Jaime Diez-Medrano, Christian Welzel, and Christian Haerpfer

3.1 Introduction 41

3.2 Applied Harmonization Methods 42

3.3 Documentation and Quality Assurance 48

3.4 Challenges to Harmonization 49

3.5 Software Tools 51

3.6 Recommendations 52

References 54

4 Harmonization in the Afrobarometer 57
Carolyn Logan, Robert Mattes, and Francis Kibirige

4.1 Introduction 57

4.2 Core Principles 58

4.3 Applied Harmonization Methods 60

4.3.1 Sampling 60

4.3.2 Training 61

4.3.3 Fieldwork and Data Collection 62

4.3.4 Questionnaire 62

4.3.5 Translation 64

4.3.6 Data Management 65

4.3.7 Documentation 65

4.4 Harmonization and Country Selection 66

4.5 Software Tools and Harmonization 66

4.6 Challenges to Harmonization 67

4.6.1 Local Knowledge, Flexibility/Adaptability, and the “Dictatorship of Harmonization” 68

4.6.2 The Quality-Cost Trade-off and Implications for Harmonization 68

4.6.3 Final Challenge: “Events” 69

4.7 Recommendations 70

References 71

5 Harmonization in the National Longitudinal Surveys of Youth (NLSY) 73
Elizabeth Cooksey, Rosella Gardecki, Carole Lunney, and Amanda Roose

5.1 Introduction 73

5.2 Cross-Cohort Design 75

5.3 Applied Harmonization 76

5.4 Challenges to Harmonization 80

5.5 Documentation and Quality Assurance 82

5.6 Software Tools 84

5.7 Recommendations and Some Concluding Thoughts 86

References 87

6 Harmonization in the Comparative Study of Electoral Systems (CSES) Projects 89
Stephen Quinlan, Christian Schimpf, Katharina Blinzler, and Slaven Zivkovic

6.1 Introducing the CSES 89

6.2 Harmonization Principles and Technical Infrastructure 91

6.3 Ex-ante Input Harmonization 91

6.3.1 Module Questionnaire 92

6.3.2 Macro Data 94

6.4 Ex-ante Output Harmonization 97

6.4.1 Demographic Variables in CSES Modules 97

6.4.2 Harmonizing Party Data in Modules 98

6.4.3 Derivative Variables 99

6.5 Exploring Interplay Between Ex-ante and Ex-post Harmonization 101

6.5.1 Demographic Variables in CSES IMD 101

6.5.2 Harmonizing Party Data in CSES IMD 102

6.6 Taking Stock and New Frontiers in Harmonization 104

References 105

7 Harmonization in the East Asian Social Survey 107
Noriko Iwai, Tetsuo Mo, Jibum Kim, Chyi-In Wu, and Weidong Wang

7.1 Introduction 107

7.2 Characteristics of the EASS and its Harmonization Process 108

7.2.1 Outline of the East Asian Social Survey 108

7.2.2 Harmonization Process of the EASS 111

7.2.2.1 Establishing the Module Theme 111

7.2.2.2 Selecting Subtopics and Questions 112

7.2.2.3 Harmonization of Standard Background Variables 113

7.2.2.4 Harmonization of Answer Choices and Scales 114

7.2.2.5 Translation of Questions and Answer Choices 115

7.3 Documentation and Quality Assurance 115

7.3.1 Five Steps to Harmonize the EASS Integrated Data 115

7.3.2 Documentation of the EASS Integrated Data 117

7.4 Challenges to Harmonization 118

7.4.1 How to Translate “Fair” and Restriction by Copyright 118

7.4.2 Difficulty in Synchronizing the Data Collection Phase 121

7.5 Software Tools 122

7.6 Recommendations 122

Acknowledgment 123

References 123

8 Ex-ante Harmonization of Official Statistics in Africa (SHaSA) 125
Dossina Yeo

Abbreviations 125

8.1 Introduction 127

8.2 Applied Harmonization Methods 128

8.2.1 Examples of Ex-ante Harmonization Methods: The Cases of GPS Data and CRVS 131

8.2.1.1 Governance, Peace and Security (GPS) Statistics Initiative 131

8.2.1.2 Development of Civil Registration and Vital Statistics (CRVS) 132

8.2.2 Examples of Ex-post Harmonization: The Cases of Labor Statistics, ATSY, ASY and KeyStats, and ICP-Africa Program 132

8.3 Quality Assurance Framework 134

8.4 Challenges to Statistical Harmonization in Africa 136

8.4.1 Challenges to the Implementation of NSDS 137

8.4.2 Challenges with Ex-ante Harmonization: Examples of GPS and ICP Initiatives 138

8.4.3 Challenges with Ex-post Harmonization: Examples of KeyStats and ATSY 139

8.5 Common Software Tools Used 139

8.6 Conclusion and Recommendations 140

References 142

Part II Ex-post harmonization of national social surveys 145

9 Harmonization for Cross-National Secondary Analysis: Survey Data Recycling 147
Irina Tomescu-Dubrow, Kazimierz M. Slomczynski, Ilona Wysmulek, Przemek Powałko, Olga Li, Yamei Tu, Marcin Slarzynski, Marcin W. Zielinski, and Denys Lavryk

9.1 Introduction 147

9.2 Harmonization Methods in the SDR Project 149

9.2.1 Building the Harmonized SDR2 Database 150

9.3 Documentation and Quality Assurance 155

9.4 Challenges to Harmonization 156

9.5 Software Tools of the SDR Project 161

9.5.1 The SDR Portal 161

9.5.2 The SDR2 COTTON FILE 162

9.6 Recommendations 162

9.6.1 Recommendations for Researchers Interested in Harmonizing Survey Data Ex-Post 162

9.6.2 Recommendations for SDR2 Users 163

Acknowledgments 164

References 164

9.A Data Quality Indicators in SDR2 166

10 Harmonization of Panel Surveys: The Cross-National Equivalent File 169
Dean R. Lillard

10.1 Introduction 169

10.2 Applied Harmonization Methods 170

10.2.1 CNEF Country Data Sources, Current and Planned 176

10.3 Current CNEF Partners 176

10.3.1 The HILDA Survey 176

10.3.2 The SLID 176

10.3.3 The CFPS 177

10.3.4 The SOEP 177

10.3.4.1 The BHPS 177

10.3.4.2 Understanding Society, UKHLS 178

10.3.5 The ITA.LI 178

10.3.6 The JHPS 178

10.3.7 The RLMS-HSE 178

10.3.8 The KLIPS 179

10.3.9 The Swedish Pseudo-Panel 179

10.3.10 The SHP 179

10.3.11 The PSID 179

10.4 Planned CNEF Partners 180

10.4.1 The ASEP 180

10.4.2 LISA 180

10.4.3 The ILS 180

10.4.4 The MxFLS 180

10.4.5 The NIDS 181

10.4.6 The PSFD 181

10.5 Documentation and Quality Assurance 181

10.6 Challenges to Harmonization 183

10.7 Recommendations for Researchers Interested in Harmonizing Panel Survey Data 185

10.8 Conclusion 186

References 187

11 Harmonization of Survey Data from UK Longitudinal Studies: CLOSER 189
Dara O’Neill and Rebecca Hardy

11.1 Introduction 189

11.2 Applied Harmonization Methods 191

11.2.1 Occupational Social Class 191

11.2.2 Body Size/Anthropometric Data 193

11.2.3 Mental Health 194

11.2.4 Harmonization Methods: Divergence and Convergence 195

11.3 Documentation and Quality Assurance 196

11.4 Challenges to Harmonization 198

11.5 Software Tools 199

11.6 Recommendations 200

Acknowledgments 202

References 202

12 Harmonization of Census Data: IPUMS – International 207
Steven Ruggles, Lara Cleveland, and Matthew Sobek

12.1 Introduction 207

12.2 Project History 208

12.2.1 Evolution of the Web Dissemination System 210

12.3 Applied Harmonization Methods 210

12.4 Documentation and Quality Assurance 215

12.5 Challenges to Harmonization 217

12.6 Software Tools 221

12.6.1 Metadata Tools 221

12.6.2 Data Reformatting 221

12.6.3 Data Harmonization 221

12.6.4 Dissemination System 222

12.7 Team Organization and Project Management 222

12.8 Lessons and Recommendations 223

References 225

Part III Domain-driven ex-post harmonization 227

13 Maelstrom Research Approaches to Retrospective Harmonization of Cohort Data for Epidemiological Research 229
Tina W. Wey and Isabel Fortier

13.1 Introduction 229

13.2 Applied Harmonization Methods 230

13.2.1 Implementing the Project 233

13.2.1.1 Initiating Activities and Organizing the Operational Framework 233

13.2.1.2 Assembling Study Information and Selecting Final Participating Studies (Guidelines Step 1) 234

13.2.1.3 Defining Target Variables to be Harmonized (the DataSchema) and Evaluating Harmonization Potential across Studies (Guidelines Step 2) 235

13.2.2 Producing the Harmonized Datasets 236

13.2.2.1 Processing Data (Guidelines Step 3a) 236

13.2.2.2 Processing Study-Specific Data to Generate Harmonized Datasets (Guidelines Step 3b) 237

13.3 Documentation and Quality Assurance 238

13.4 Challenges to Harmonization 240

13.5 Software Tools 241

13.6 Recommendations 243

Acknowledgments 244

References 245

14 Harmonizing and Synthesizing Partnership Histories from Different German Survey Infrastructures 249
Bernd Weiß, Sonja Schulz, Lisa Schmid, Sebastian Sterl, and Anna-Carolina Haensch

14.1 Introduction 249

14.2 Applied Harmonization Methods 250

14.2.1 Data Search Strategy and Data Access 250

14.2.2 Processing and Harmonizing Data 253

14.2.2.1 Harmonizing Partnership Biography Data 253

14.2.2.2 Harmonizing Additional Variables on Respondents’ or Couples’ Characteristics 254

14.3 Documentation and Quality Assurance 255

14.3.1 Documentation 255

14.3.2 Quality Assurance 256

14.3.2.1 Process-Related Quality Assurance 256

14.3.2.2 Benchmarking the Harmonized HaSpaD Data Set with Official Statistics 256

14.4 Challenges to Harmonization 258

14.4.1 Analyzing Harmonized Complex Survey Data 258

14.4.2 Sporadically and Systematically Missing Data 259

14.5 Software Tools 260

14.6 Recommendations 262

14.6.1 Harmonizing Biographical Data 262

14.6.1.1 Methodological Recommendations 262

14.6.1.2 Procedural Recommendations 263

14.6.1.3 Technical Recommendations 263

14.6.2 Getting Started with the Cumulative HaSpaD Data Set 263

Acknowledgments 264

References 264

15 Harmonization and Quality Assurance of Income and Wealth Data: The Case of LIS 269
Jörg Neugschwender, Teresa Munzi, and Piotr R. Paradowski

15.1 Introduction 269

15.2 Applied Harmonization Methods 271

15.3 Documentation and Quality Assurance 275

15.3.1 Quality Assurance 275

Selection of Source Datasets 276

Harmonization 276

Validation – “Green Light” Check 276

15.3.2 Documentation 278

15.4 Challenges to Harmonization 278

15.5 Software Tools 281

15.6 Conclusion 282

References 283
16 Ex-Post Harmonization of Time Use Data: Current Practices and Challenges in the Field 285
Ewa Jarosz, Sarah Flood, and Margarita Vega-Rapun

16.1 Introduction 285

16.2 Applied Harmonization Methods 289

16.2.1 Harmonizing the Matrix of the Diary 289

16.2.2 Variable Harmonization 291

16.2.3 Other Variables 293

16.2.4 Other Types of Time Use Data 294

16.3 Documentation and Quality Assurance 294

16.3.1 Documentation 294

16.3.2 Quality Checks 296

16.4 Challenges to Harmonization 297

16.5 Software Tools 300

16.6 Recommendations 301

References 302

Part IV Further Issues: Dealing with Methodological Issues in Harmonized Survey Data 305

17 Assessing and Improving the Comparability of Latent Construct Measurements in Ex-Post Harmonization 307
Ranjit K. Singh and Markus Quandt

17.1 Introduction 307

17.2 Measurement and Reality 307

17.3 Construct Match 308

17.3.1 Consequences of a Mismatch 309

17.3.2 Assessment 309

17.3.2.1 Qualitative Research Methods 309

17.3.2.2 Construct and Criterion Validity 309

17.3.2.3 Techniques for Multi-Item Instruments 310

17.3.2.4 Improving Construct Comparability 311

17.4 Reliability Differences 311

17.4.1 Consequences of Reliability Differences 311

17.4.2 Assessment 312

17.4.3 Improving Reliability Comparability 312

17.5 Units of Measurement 312

17.5.1 Consequences of Unit Differences 313

17.5.2 Improving Unit Comparability 313

17.5.3 Controlling for Instrument Characteristics 314

17.5.4 Harmonizing Units Based on Repeated Measurements 315

17.5.5 Harmonizing Units Based on Measurements Obtained from the Same Population 315

17.6 Cross-Cultural Comparability 316

17.6.1 Construct Match 316

17.6.1.1 Translation and Cognitive Probing 317

17.6.2 Reliability 317

17.6.3 Units of Measurement 318

17.6.3.1 Harmonizing Units of Localized Versions of the Same Instrument 318

17.6.3.2 Harmonizing Units Across Cultures and Instruments 318

17.6.4 Cross-Cultural Comparability of Multi-Item Instruments 318

17.7 Discussion and Outlook 319

References 320

18 Comparability and Measurement Invariance 323
Artur Pokropek

18.1 Latent Variable Framework for Testing and Accounting for Measurement Non-Invariance 324

18.2 Approaches to Empirical Assessment of Measurement Equivalence 325

18.2.1 Classical Invariance Analysis (MG-CFA) 326

18.2.2 Partial Invariance (MG-CFA) 327

18.2.3 Approximate Invariance 327

18.2.4 Approximate Partial Invariance (Alignment, BSEM Alignment, Partial BSEM) 328

18.3 Beyond Multiple Indicators 329

18.4 Conclusions 329

References 330

19 On the Creation, Documentation, and Sensible Use of Weights in the Context of Comparative Surveys 333
Dominique Joye, Marlène Sapin, and Christof Wolf

19.1 Introduction 333

19.2 Design Weights 335

19.2.1 What to do? 336

19.3 Post-stratification Weights 337

19.3.1 What Should be Done? 340

19.4 Population Weights 341

19.4.1 What Should be Done? 342

19.5 Conclusion 342

References 344

20 On Using Harmonized Data in Statistical Analysis: Notes of Caution 347
Claire Durand

20.1 Introduction 347

20.2 Challenges in the Combination of Data Sets 347

20.2.1 A First Principle: A No Censorship Inclusive Approach 348

20.2.2 A Second Principle: Using Multilevel Analysis and Introducing a Measurement Level 349

20.2.3 A Third Principle: Assessing the Equivalence of Survey Projects 351

20.3 Challenges in the Analysis of Combined Data Sets 353

20.3.1 Dealing with Time 354

20.3.2 Dealing with Missing Values 358

20.3.2.1 Missing Values at the Respondent and Measurement Level 358

20.3.2.2 Missing Values at the Survey Level 359

20.3.3 Dealing with Weights 361

20.4 Recommendations 362

References 363

21 On the Future of Survey Data Harmonization 367
Kazimierz M. Slomczynski, Christof Wolf, Irina Tomescu-Dubrow, and J. Craig Jenkins

21.1 What We Have Learned from Contributions on Survey Data Harmonization in this Volume 368

21.2 New Opportunities and Challenges 370

21.2.1 Reorientation of Survey Research in the Era of New Technology 370

21.2.2 Advances in Technical Aspects of Data Management 370

21.2.3 Harmonizing Survey Data with Other Types of Data 371

21.3 Developing a New Methodology of Harmonizing Non-Survey Data 372

21.3.1 Emerging Legal and Ethical Issues 372

21.4 Globalization of Science and Harmonizing Scientific Practice 373

References 373

Index 377