Total Survey Error in Practice
Description
More details
Other editions
Additional editions
Persons
Content
Notes on Contributors xix
Preface xxv
Section 1 The Concept of TSE and the TSE Paradigm 1
1 The Roots and Evolution of the Total Survey Error Concept 3
Lars E. Lyberg and Diana Maria Stukel
1.1 Introduction and Historical Backdrop 3
1.2 Specific Error Sources and Their Control or Evaluation 5
1.3 Survey Models and Total Survey Design 10
1.4 The Advent of More Systematic Approaches Toward Survey Quality 12
1.5 What the Future Will Bring 16
References 18
2 Total Twitter Error: Decomposing Public Opinion Measurement on Twitter from a Total Survey Error Perspective 23
Yuli Patrick Hsieh and Joe Murphy
2.1 Introduction 23
2.2 Social Media: An Evolving Online Public Sphere 25
2.3 Components of Twitter Error 27
2.4 Studying Public Opinion on the Twittersphere and the Potential Error Sources of Twitter Data: Two Case Studies 31
2.5 Discussion 40
2.6 Conclusion 42
References 43
3 Big Data: A Survey Research Perspective 47
Reg Baker
3.1 Introduction 47
3.2 Definitions 48
3.3 The Analytic Challenge: From Database Marketing to Big Data and Data Science 56
3.4 Assessing Data Quality 58
3.5 Applications in Market, Opinion, and Social Research 59
3.6 The Ethics of Research Using Big Data 62
3.7 The Future of Surveys in a Data-Rich Environment 62
References 65
4 The Role of Statistical Disclosure Limitation in Total Survey Error 71
Alan F. Karr
4.1 Introduction 71
4.2 Primer on SDL 72
4.3 TSE-Aware SDL 75
4.4 Edit-Respecting SDL 79
4.5 SDL-Aware TSE 83
4.6 Full Unification of Edit, Imputation, and SDL 84
4.7 "Big Data" Issues 87
4.8 Conclusion 89
Acknowledgments 91
References 92
Section 2 Implications for Survey Design 95
5 The Undercoverage-Nonresponse Tradeoff 97
Stephanie Eckman and Frauke Kreuter
5.1 Introduction 97
5.2 Examples of the Tradeoff 98
5.3 Simple Demonstration of the Tradeoff 99
5.4 Coverage and Response Propensities and Bias 100
5.5 Simulation Study of Rates and Bias 102
5.6 Costs 110
5.7 Lessons for Survey Practice 111
References 112
6 Mixing Modes: Tradeoffs Among Coverage, Nonresponse, and Measurement Error 115
Roger Tourangeau
6.1 Introduction 115
6.2 The Effect of Offering a Choice of Modes 118
6.3 Getting People to Respond Online 119
6.4 Sequencing Different Modes of Data Collection 120
6.5 Separating the Effects of Mode on Selection and Reporting 122
6.6 Maximizing Comparability Versus Minimizing Error 127
6.7 Conclusions 129
References 130
7 Mobile Web Surveys: A Total Survey Error Perspective 133
Mick P. Couper, Christopher Antoun, and Aigul Mavletova
7.1 Introduction 133
7.2 Coverage 135
7.3 Nonresponse 137
7.4 Measurement Error 142
7.5 Links Between Different Error Sources 148
7.6 The Future of Mobile Web Surveys 149
References 150
8 The Effects of a Mid-Data Collection Change in Financial Incentives on Total Survey Error in the National Survey of Family Growth: Results from a Randomized Experiment 155
James Wagner, Brady T. West, Heidi Guyer, Paul Burton, Jennifer Kelley, Mick P. Couper, and William D. Mosher
8.1 Introduction 155
8.2 Literature Review: Incentives in Face-to-Face Surveys 156
8.3 Data and Methods 159
8.4 Results 163
8.5 Conclusion 173
References 175
9 A Total Survey Error Perspective on Surveys in Multinational, Multiregional, and Multicultural Contexts 179
Beth-Ellen Pennell, Kristen Cibelli Hibben, Lars E. Lyberg, Peter Ph. Mohler, and Gelaye Worku
9.1 Introduction 179
9.2 TSE in Multinational, Multiregional, and Multicultural Surveys 180
9.3 Challenges Related to Representation and Measurement Error Components in Comparative Surveys 184
9.4 QA and QC in 3MC Surveys 192
References 196
10 Smartphone Participation in Web Surveys: Choosing Between the Potential for Coverage, Nonresponse, and Measurement Error 203
Gregg Peterson, Jamie Griffin, John LaFrance, and JiaoJiao Li
10.1 Introduction 203
10.2 Prevalence of Smartphone Participation in Web Surveys 206
10.3 Smartphone Participation Choices 209
10.4 Instrument Design Choices 212
10.5 Device and Design Treatment Choices 216
10.6 Conclusion 218
10.7 Future Challenges and Research Needs 219
Appendix 10.A: Data Sources 220
Appendix 10.B: Smartphone Prevalence in Web Surveys 221
Appendix 10.C: Screen Captures from Peterson et al. (2013) Experiment 225
Appendix 10.D: Survey Questions Used in the Analysis of the Peterson et al. (2013) Experiment 229
References 231
11 Survey Research and the Quality of Survey Data Among Ethnic Minorities 235
Joost Kappelhof
11.1 Introduction 235
11.2 On the Use of the Terms Ethnicity and Ethnic Minorities 236
11.3 On the Representation of Ethnic Minorities in Surveys 237 Ethnic Minorities 241
11.4 Measurement Issues 242
11.5 Comparability, Timeliness, and Cost Concerns 244
11.6 Conclusion 247
References 248
Section 3 Data Collection and Data Processing Applications 253
12 Measurement Error in Survey Operations Management: Detection, Quantification, Visualization, and Reduction 255
Brad Edwards, Aaron Maitland, and Sue Connor
12.1 TSE Background on Survey Operations 256
12.2 Better and Better: Using Behavior Coding (CARIcode) and Paradata to Evaluate and Improve Question (Specification) Error and Interviewer Error 257
12.3 Field-Centered Design: Mobile App for Rapid Reporting and Management 261
12.4 Faster and Cheaper: Detecting Falsification With GIS Tools 265
12.5 Putting It All Together: Field Supervisor Dashboards 268
12.6 Discussion 273
References 275
13 Total Survey Error for Longitudinal Surveys 279
Peter Lynn and Peter J. Lugtig
13.1 Introduction 279
13.2 Distinctive Aspects of Longitudinal Surveys 280
13.3 TSE Components in Longitudinal Surveys 281
13.4 Design of Longitudinal Surveys from a TSE Perspective 285
13.5 Examples of Tradeoffs in Three Longitudinal Surveys 290
13.6 Discussion 294
References 295
14 Text Interviews on Mobile Devices 299
Frederick G. Conrad, Michael F. Schober, Christopher Antoun, Andrew L. Hupp, and H. Yanna Yan
14.1 Texting as a Way of Interacting 300
14.2 Contacting and Inviting Potential Respondents through Text 303
14.3 Texting as an Interview Mode 303
14.4 Costs and Efficiency of Text Interviewing 312
14.5 Discussion 314
References 315
15 Quantifying Measurement Errors in Partially Edited Business Survey Data 319
Thomas Laitila, Karin Lindgren, Anders Norberg, and Can Tongur
15.1 Introduction 319
15.2 Selective Editing 320
15.3 Effects of Errors Remaining After SE 325
15.4 Case Study: Foreign Trade in Goods Within the European Union 328
15.5 Editing Big Data 334
15.6 Conclusions 335
References 335
Section 4 Evaluation and Improvement 339
16 Estimating Error Rates in an Administrative Register and Survey Questions Using a Latent Class Model 341
Daniel L. Oberski
16.1 Introduction 341
16.2 Administrative and Survey Measures of Neighborhood 342
16.3 A Latent Class Model for Neighborhood of Residence 345
16.4 Results 348
Appendix 16.A: Program Input and Data 355
Acknowledgments 357
References 357
17 ASPIRE: An Approach for Evaluating and Reducing the Total Error in Statistical Products with Application to Registers and the National Accounts 359
Paul P. Biemer, Dennis Trewin, Heather Bergdahl, and Yingfu Xie
17.1 Introduction and Background 359
17.2 Overview of ASPIRE 360
17.3 The ASPIRE Model 362
17.4 Evaluation of Registers 367
17.5 National Accounts 371
17.6 A Sensitivity Analysis of GDP Error Sources 376
17.7 Concluding Remarks 379
Appendix 17.A: Accuracy Dimension Checklist 381
References 384
18 Classification Error in Crime Victimization Surveys: A Markov Latent Class Analysis 387
Marcus E. Berzofsky and Paul P. Biemer
18.1 Introduction 387
18.2 Background 389
18.3 Analytic Approach 392
18.4 Model Selection 396
18.5 Results 399
18.6 Discussion and Summary of Findings 404
18.7 Conclusions 407
Appendix 18.A: Derivation of the Composite False-Negative Rate 407
Appendix 18.B: Derivation of the Lower Bound for False-Negative Rates from a Composite Measure 408
Appendix 18.C: Examples of Latent GOLD Syntax 408
References 410
19 Using Doorstep Concerns Data to Evaluate and Correct for Nonresponse Error in a Longitudinal Survey 413
Ting Yan
19.1 Introduction 413
19.2 Data and Methods 416
19.3 Results 418
19.4 Discussion 428
Acknowledgment 430
References 430
20 Total Survey Error Assessment for Sociodemographic Subgroups in the 2012 U.S. National Immunization Survey 433
Kirk M. Wolter, Vicki J. Pineau, Benjamin Skalland, Wei Zeng, James A. Singleton, Meena Khare, Zhen Zhao, David Yankey, and Philip J. Smith
20.1 Introduction 433
20.2 TSE Model Framework 434
20.3 Overview of the National Immunization Survey 437
20.4 National Immunization Survey: Inputs for TSE Model 440
20.5 National Immunization Survey TSE Analysis 445
20.6 Summary 452
References 453
21 Establishing Infrastructure for the Use of Big Data to Understand Total Survey Error: Examples from Four Survey Research Organizations Overview 457
Brady T. West
Part 1 Big Data Infrastructure at the Institute for Employment Research (IAB) 458
Antje Kirchner, Daniela Hochfellner, Stefan Bender
Acknowledgments 464
References 464
Part 2 Using Administrative Records Data at the U.S. Census Bureau: Lessons Learned from Two Research Projects Evaluating Survey Data 467
Elizabeth M. Nichols, Mary H. Mulry, and Jennifer Hunter Childs
Acknowledgments and Disclaimers 472
References 472
Part 3 Statistics New Zealand's Approach to Making Use of Alternative Data Sources in a New Era of Integrated Data 474
Anders Holmberg and Christine Bycroft
References 478
Part 4 Big Data Serving Survey Research: Experiences at the University of Michigan Survey Research Center 478
Grant Benson and Frost Hubbard
Acknowledgments and Disclaimers 484
References 484
Section 5 Estimation and Analysis 487
22 Analytic Error as an Important Component of Total Survey Error: Results from a Meta-Analysis 489
Brady T. West, Joseph W. Sakshaug, and Yumi Kim
22.1 Overview 489
22.2 Analytic Error as a Component of TSE 490
22.3 Appropriate Analytic Methods for Survey Data 492
22.4 Methods 495
22.5 Results 497
22.6 Discussion 505
Acknowledgments 508
References 508
23 Mixed-Mode Research: Issues in Design and Analysis 511
Joop Hox, Edith de Leeuw, and Thomas Klausch
23.1 Introduction 511
23.2 Designing Mixed-Mode Surveys 512
23.3 Literature Overview 514
23.4 Diagnosing Sources of Error in Mixed-Mode Surveys 516
23.5 Adjusting for Mode Measurement Effects 523
23.6 Conclusion 527
References 528
24 The Effect of Nonresponse and Measurement Error on Wage Regression across Survey Modes: A Validation Study 531
Antje Kirchner and Barbara Felderer
24.1 Introduction 531
24.2 Nonresponse and Response Bias in Survey Statistics 532
24.3 Data and Methods 534
24.4 Results 541
24.5 Summary and Conclusion 546
Acknowledgments 547
Appendix 24.A 548
Appendix 24.B 549
References 554
25 Errors in Linking Survey and Administrative Data 557
Joseph W. Sakshaug and Manfred Antoni
25.1 Introduction 557
25.2 Conceptual Framework of Linkage and Error Sources 559
25.3 Errors Due to Linkage Consent 561
25.4 Erroneous Linkage with Unique Identifiers 565
25.5 Erroneous Linkage with Nonunique Identifiers 567
25.6 Applications and Practical Guidance 568
25.7 Conclusions and Take-Home Points 571
References 571
Index 575
1
The Roots and Evolution of the Total Survey Error Concept
Lars E. Lyberg1 and Diana Maria Stukel2
1 Inizio, Stockholm, Sweden
2 FHI 360, Washington, DC, USA
1.1 Introduction and Historical Backdrop
Sir Ronald Fisher
Jerzy Neyman
In this chapter, we discuss the concept of total survey error (TSE), how it originated and developed both as a mindset for survey researchers and as a criterion for designing surveys. The interest in TSE has fluctuated over the years. When Jerzy Neyman published the basic sampling theory and some of its associated sampling schemes in 1934 onward, it constituted the first building block of a theory and methodology for surveys. However, the idea that a sample could be used to represent an entire population was not new. The oldest known reference to estimating a finite population total on the basis of a sample dates back to 1000 BC and is found in the Indian epic Mahabharata (Hacking, 1975; Rao, 2005). Crude attempts at measuring parts of a population rather than the whole had been used in England and some other European countries quite extensively between 1650 and 1800. The methods on which these attempts were based were referred to as political arithmetic (Fienberg and Tanur, 2001), and they resembled ratio estimation using information of birth rates, family size, and average number of persons living in selected buildings and other observations. In 1895, at an International Statistical Institute meeting, Kiaer argued for developing a representative or partial investigation method (Kiaer, 1897). The representative method aimed at creating a sample that would reflect the composition of the population of interest. This could be achieved by using balanced sampling through purposive selection or various forms of random sampling. During the period 1900-1920, the representative method was used extensively, at least in Russia and the U.S.A. In 1925, the International Statistical Institute released a report on various aspects of random sampling (Rao, 2005, 2013; Rao and Fuller, 2015). The main consideration regarding sampling was likely monetary, given that it was resource-intensive and time-consuming to collect data from an entire population. Statistical information compiled using a representative sample was an enormous breakthrough. But it would be almost 40 years after Kiaer's proposal before Neyman published his landmark paper from 1934 "On the Two Different Aspects of the Representative Method: The Method of Stratified Sampling and the Method of Purposive Selection." At this time, there existed some earlier work by the Russian statistician Tschuprow (1923a, b) on stratified sampling and optimal allocation. It is not clear whether Neyman was aware of this work when he started to develop the sampling theory in the 1920s (Fienberg and Tanur, 1996) since he did not mention Tschuprow's work when discussing optimal allocation. Neyman definitely had access to Ronald Fisher's (1925) ideas on randomization (as opposed to various kinds of purposive selection) and their importance for the design and analysis of experiments, and also to Bowley's (1926) work on stratified random sampling.
Prasanta Mahalanobis
Morris Hansen
Edwards Deming
The sampling methods proposed by Neyman were soon implemented in agencies such as the Indian Statistical Institute and the U.S. Bureau of the Census (currently named the U.S. Census Bureau). Prasanta Mahalanobis, the founder of the Indian Statistical Institute, and Morris Hansen and colleagues at the U.S. Census Bureau, became the main proponents of scientific sampling in a number of surveys in the 1940s. The development was spurred on by Literary Digest's disastrously inaccurate prediction in the 1936 U.S. presidential election poll that was based on a seriously deficient sampling frame. However, Neyman's sampling theory did not take into account nonsampling errors and relied on the assumption that sampling was the only major error source that affected estimates of population parameters and associated calculations of confidence intervals or margins of error. However, Neyman and his peers understood that this was indeed an unrealistic assumption that might lead to understated margins of error. The effect of nonsampling errors on censuses was acknowledged and discussed in a German textbook on census methodology relatively early on (Zizek, 1921). The author discussed what he called control of contents and coverage. In addition, Karl Pearson (1902) discussed observer errors much earlier than that. An early example of interviewer influence on survey response was the study on the consumption of hard liquor during the prohibition days in the U.S.A., where Rice (1929) showed that interviewers who were prohibitionists tended to obtain responses that mirrored their own views and that differed from those of respondents that were interviewed by other interviewers.
In 1944, Edwards Deming published the first typology of sources of error beyond sampling. He listed 13 factors that he believed might affect the utility of a survey. The main purpose of the typology was to demonstrate the need for directing efforts to all potential sources in the survey planning process while considering the resources available. This first typology included some error sources that are not frequently referenced today, such as bias of the auspices (i.e., the tendency to indicate a particular response because of the organization sponsoring the study). Others, to which more attention is currently given, such as coverage error, were not included, however. Even though Deming did not explicitly reference TSE, he emphasized the limitations of concentrating on a few error sources only and highlighted the need for theories of bias and variability based on accumulated experience.
Rapid development of the area followed shortly thereafter. Mahalanobis (1946) developed the method of interpenetration, which could be used to estimate the variability generated by interviewers and other data collectors. Another error source recognized early on was nonresponse. Hansen and Hurwitz (1946) published an article in the Journal of the American Statistical Association on follow-up sampling from the stratum of initial nonrespondents. While the basic assumption of 100% participation in a follow-up sample was understood not to be realistic, at the time, there were relatively small nonresponse rates, and it was possible to estimate, at least approximately, the characteristics of those in the nonresponse stratum.
Even though it is not explicitly stated, TSE has its roots in cautioning against sole attention focused on sampling error along with possibly one or two other error sources, rather than the entire scope of potential errors. In response, two lines of strategic development occurred. One strategy entailed the identification of specific error sources, coupled with an attempt to control them or at least minimize them. The other strategy entailed the development of the so-called survey error models, where the TSE was decomposed and the magnitude of different error components, and ultimately the combination of them (i.e., the TSE), could be estimated. The two strategies were intertwined in the sense that a survey model could be applied not only on the entire set of survey operations but also on a subset of specific survey operations.
1.2 Specific Error Sources and Their Control or Evaluation
Leslie Kish
Apart from that of Deming (1944), there are a number of typologies described in the survey literature. Examples include Kish (1965), Groves (1989), Biemer and Lyberg (2003), Groves et al. (2009), Smith (2011), and Pennell et al. (Chapter 9 in this volume). Some of them are explicitly labeled TSE, while others consist of listings of different types of errors; however, all are incomplete. In some cases, known error sources (as well as their interactions with other error sources) are simply omitted, and in other cases, all possible error sources are not known or the sources defy expression. For instance, new error structures have emerged when new data collection modes or new data sources, such as Big Data (see, e.g., Chapter 3 in this volume), have become popular-but the comprehension and articulation of the associated error structures have lagged in time.
Early on, the work toward the treatment of specific error sources followed two separate types of strategies: control and evaluation.
Related to the first strategy of control, one line of thinking was that statistical agencies were "data factories" that produced tables and limited analyses as their outputs. As such, they resembled an industrial assembly line. Therefore, the application of methods for industrial quality control (QC) was deemed suitable. Several statistical agencies adopted this approach for some of their operations, and the U.S. Census Bureau was clearly at the forefront. Most of these QCs were focused toward manual operations such as enumeration and interviewing, listing, coding, card punching, and editing, although it was also possible to use QC to check automatic operations such as scanning, which at the time was implemented through Film Optical Sensing Device for Input to Computers (FOSDIC). For the manual operations, the main control method was verification, where one operator's work was checked by another operator. A long list of census methodologists including Morris Hansen, Bill Hurwitz, Eli Marks, Edwards Deming, Ross Eckler, Max Bershad, Leon Pritzker, Joe Waksberg, Herman Fasteau, and George Minton made very...
