Register-based Statistics

Anders Wallgren and Britt Wallgren worked for many years in teaching and research as senior lecturers in statistics at Stockholm, Linköping and Örebro universities. They have also worked as consultants for several Swedish manufacturing industries. In connection with these consulting assignments, they began working with corporate administrative registers and developed statistical methods for business intelligence. After their academic career, they came to Statistics Sweden and worked for about 20 years at the Department of Research and Development. They worked mainly with survey methodology for the statistical registers which form the basis for the production of official statistics in Sweden. Since 2011 they have provided courses and consulting services on behalf of the Inter-American Development Bank in several Latin American countries and in the Caribbean.

Index to Charts xi

Preface xix

Chapter 1 Censuses, Sample Surveys and Register Surveys 1

1.1 The national statistical system 2

1.2 The traditional census-based system 3

1.3 New sources: Administrative registers and Big data 5

1.4 Basic concepts and terms 7

1.4.1 What is a register? 7

1.4.2 Databases, records and observations 8

1.4.3 What is a register survey? 10

1.4.4 A register survey: The Income and Taxation Register 12

1.5 New demands and opportunities require new methods 14

1.5.1 A new paradigm is necessary 14

1.5.2 New statistical methods 15

1.5.3 The basic principles of register-based statistics 18

1.6 Preconditions for register-based statistics 19

1.6.1 Reliable administrative systems 20

1.6.2 Legal base and public approval 21

1.6.3 Political support to strengthen the statistical system 23

Chapter 2 The Transition to a Register-based Production System 25

2.1 First obstacle: How to gain access to microdata? 26

2.2 Protection of privacy and confidentiality 26

2.3 Second obstacle: How to take care of dirty data? 29

2.4 The new production process 30

2.4.1 Contacts with administrative authorities 31

2.4.2 Metadata have a new role 31

2.4.3 Anonymisation of identity numbers 32

2.4.4 Editing of a single administrative register 33

2.4.5 Organising the work with administrative registers 36

2.5 Third obstacle: The national registration system 37

2.5.1 Legislation governs access to data 38

2.5.2 Too many registers, but no good registers – what to do? 38

2.5.3 Legislation rules obligations to report and what to report 40

2.6 Why has the census been so important? 41

2.7 Creating the register system 42

2.7.1 Where do you live? 43

2.7.2 Where do you work? 45

2.7.3 With whom do you live? 46

2.7.4 A centralised or decentralised national system? 47

2.8 Register surveys and estimation methods 48

2.9 A traditional census or a register-based census? 49

Chapter 3 The Nature of Administrative Data 51

3.1 Comparing questionnaire and register data 51

3.1.1 A questionnaire to persons compared with register data 51

3.1.2 An enterprise questionnaire compared with register data 54

3.2 Enterprise registers for combined use 56

3.2.1 Corrections in accounting data 57

3.2.2 Missing values in accounting data 58

3.2.3 Administrative and statistical information systems 59

3.3 Measurement errors in questionnaire and register data 60

3.3.1 Measurement errors 61

3.3.2 Taxation errors 62

Chapter 4 Building the System – Record Linkage 65

4.1 Record linkage 65

4.2 Record linkage in the Nordic countries 66

4.3 Deterministic record linkage 68

4.4 Creating variables by adjoining and aggregation 70

4.5 Probabilistic record linkage 73

4.6 Four causes of matching errors 79

4.7 The statistical system and record linkage 82

Chapter 5 Building the System – Quality Assessment 85

5.1 Four quality concepts 85

5.2 Making an inventory of potential sources 87

5.3 How can a source be used? 87

5.4 Quality assessment in a register-based production system 90

5.4.1 Analysing metadata 91

5.4.2 Analysis and data editing of the source 92

5.4.3 Comparing a source with the base register 92

5.4.4 Comparing a source with surveys with similar variables 93

5.5 Output data quality and quality of estimates 94

5.5.1 Analysing quality with a test census 94

5.5.2 Analysing quality with samples from the new register 95

5.5.3 Analysing quality with area samples 96

5.5.4 Measuring quality of basic register variables with the LFS 98

5.6 A coordinated system of registers 98

5.6.1 Are the base registers a coordinated system? 98

5.6.2 Quality indicators at the system level 99

5.7 Using the quality indicators 101

Chapter 6 Building the System – Editing Register Data 107

6.1 Editing in register surveys 108

6.2 Editing of a single administrative register 109

6.3 Consistency editing 110

6.3.1 Consistency editing – is the population correct? 111

6.3.2 Consistency editing – are the units correct? 118

6.3.3 Consistency editing – are the variables correct? 120

6.4 Case studies – editing register data 121

6.4.1 Editing work within the Income and Taxation Register 121

6.4.2 Editing work within the Income Statement Register 123

6.4.3 What more can be learned from these examples? 124

6.5 Editing, quality assessment and survey design 125

6.5.1 Survey design in a register-based production system 125

6.5.2 Survey design – management problems 127

6.5.3 Total survey error in a register-based system 128

Chapter 7 Building the System – The Population Register 129

7.1 Inventory of sources 131

7.1.1 Time references 131

7.1.2 Activities or ‘signs of life’ 131

7.2 The Population Register based on full information 133

7.2.1 Object types – Changing and unchanging registers 133

7.2.2 Variables with different functions in the system 134

7.2.3 Updating the Population Register 136

7.2.4 Registers and time 137

7.2.5 Variables and time 140

7.3 The Population Register in new register countries 140

7.3.1 Different systems of identity numbers 141

7.3.2 Problems in countries without a central Population Register 142

7.3.3 How to improve coverage of the Population Register 143

7.3.4 Inventory of sources – addresses and time references 146

7.4 Methods to measure and improve quality 148

7.4.1 Three kinds of surveys should be combined 148

7.4.2 A new register-based system for statistics on persons 150

7.5 Conclusions 151

7.6 Challenges in old register countries 152

Chapter 8 The Population Register – Estimation Methods 155

8.1 Estimation in sample surveys and register surveys 156

8.1.1 Estimation methods for register surveys that use weights 157

8.1.2 Calibration of weights in register surveys 157

8.2 Calibration of weights – the Swedish LFS 161

8.2.1 Use of auxiliary information in the LFS 161

8.2.2 Nonresponse bias in the LFS 162

8.3 Calibration – where do people live? 163

8.4 Methods to handle overcoverage 167

Chapter 9 Defining Register Populations – Coverage Errors 171

9.1 Defining a register’s object set 172

9.1.1 Defining a population 172

9.1.2 Can you alter data from the National Tax Agency? 176

9.1.3 Defining a population – the Farm Register 176

9.1.4 Defining a population – integrated registers 178

9.2 Defining a calendar year population 179

9.2.1 Defining a population – frame or register population? 180

9.2.2 Sampling paradigm versus register paradigm 184

Chapter 10 Building the System – The Business Register 185

10.1 The Business Register and the National Accounts 185

10.2 The base register for economic statistics 187

10.3 The scope of the register and choice of object types 188

10.3.1 The register with legal units and local units 189

10.3.2 The register with enterprise units and kind of activity units 191

10.4 Inventory of sources 195

10.5 Creating and maintaining the Business Register 198

Chapter 11 The Business Register – Estimation Methods 201

11.1 Multi-valued variables 202

11.2 Estimation methods 205

11.2.1 Occupation in the Activity and Occupation Registers 206

11.2.2 Industrial classification in the Business Register 210

11.2.3 Estimates from different register versions 213

11.3 Application of the method 214

11.3.1 Change of industry and time series quality 215

11.3.2 Transformation of weights 217

11.4 A decentralised or centralised statistical system? 218

11.4.1 The Calendar Year Register and the National Accounts 219

11.4.2 Choosing the best source for the National Accounts 220

11.5 Conclusions 224

Chapter 12 Censuses, Sample Surveys and Register Surveys –Conclusions 227

12.1 Attitudes towards the register-based census 227

12.2 The new national statistical system 231

12.2.1 The system of base registers 232

12.2.2 Activity registers and longitudinal registers 234

12.3 Survey design 237

12.3.1 Sample survey design 237

12.3.2 Register survey design 238

12.3.3 Creating register variables 241

12.4 Survey quality 245

12.4.1 Quality of registers and register surveys 246

12.4.2 The integration process – integration errors 247

12.4.3 Frame errors 247

12.5 Organising the new production system 248

12.5.1 Enterprise architecture and the register system 248

12.5.2 The register system and data warehousing 249

12.5.3 Missing values – a system-based approach 252

12.6 Final remarks 254

12.6.1 The Statistical Population Register 254

12.6.2 The system of base registers 255

References 257

Index 261