Online Panel Research

Preface xv

Acknowledgments xvii

About the Editors xix

About the Contributors xxiii

1 Online panel research: History, concepts, applications and a look at the future 1
Mario Callegaro, Reg Baker, Jelke Bethlehem, Anja S. Göritz, Jon A. Krosnick, and Paul J. Lavrakas

1.1 Introduction 1

1.2 Internet penetration and online panels 2

1.3 Definitions and terminology 2

1.4 A brief history of online panels 4

1.5 Development and maintenance of online panels 6

1.6 Types of studies for which online panels are used 15

1.7 Industry standards, professional associations' guidelines, and advisory groups 15

1.8 Data quality issues 17

1.9 Looking ahead to the future of online panels 17

2 A critical review of studies investigating the quality of data obtained with online panels based on probability and nonprobability samples 23
Mario Callegaro, Ana Villar, David Yeager, and Jon A. Krosnick

2.1 Introduction 23

2.2 Taxonomy of comparison studies 24

2.3 Accuracy metrics 27

2.4 Large-scale experiments on point estimates 28

2.5 Weighting adjustments 35

2.6 Predictive relationship studies 36

2.7 Experiment replicability studies 38

2.8 The special case of pre-election polls 42

2.9 Completion rates and accuracy 43

2.10 Multiple panel membership 43

2.11 Online panel studies when the offline population is less of a concern 46

2.12 Life of an online panel member 47

2.13 Summary and conclusion 48

Part I COVERAGE 55

Introduction to Part I 56
Mario Callegaro and Jon A. Krosnick

3 Assessing representativeness of a probability-based online panel in Germany 61
Bella Struminskaya, Lars Kaczmirek, Ines Schaurer, and Wolfgang Bandilla

3.1 Probability-based online panels 61

3.2 Description of the GESIS Online Panel Pilot 62

3.3 Assessing recruitment of the Online Panel Pilot 66

3.4 Assessing data quality: Comparison with external data 68

3.5 Results 74

3.6 Discussion and conclusion 80

4 Online panels and validity: Representativeness and attrition in the Finnish eOpinion panel 86
Kimmo Grönlund and Kim Strandberg

4.1 Introduction 86

4.2 Online panels: Overview of methodological considerations 87

4.3 Design and research questions 88

4.4 Data and methods 90

4.5 Findings 92

4.6 Conclusion 100

5 The untold story of multi-mode (online and mail) consumer panels: From optimal recruitment to retention and attrition 104
Allan L. McCutcheon, Kumar Rao, and Olena Kaminska

5.1 Introduction 104

5.2 Literature review 107

5.3 Methods 108

5.4 Results 115

5.5 Discussion and conclusion 124

Part II NONRESPONSE 127

Introduction to Part II 128
Jelke Bethlehem and Paul J. Lavrakas

6 Nonresponse and attrition in a probability-based online panel for the general population 135
Peter Lugtig, Marcel Das, and Annette Scherpenzeel

6.1 Introduction 135

6.2 Attrition in online panels versus offline panels 137

6.3 The LISS panel 139

6.4 Attrition modeling and results 142

6.5 Comparison of attrition and nonresponse bias 148

6.6 Discussion and conclusion 150

7 Determinants of the starting rate and the completion rate in online panel studies 154
Anja S. Göritz

7.1 Introduction 154

7.2 Dependent variables 155

7.3 Independent variables 156

7.4 Hypotheses 156

7.5 Method 163

7.6 Results 164

7.7 Discussion and conclusion 166

8 Motives for joining nonprobability online panels and their association with survey participation behavior 171
Florian Keusch, Bernad Batinic, and Wolfgang Mayerhofer

8.1 Introduction 171

8.2 Motives for survey participation and panel enrollment 173

8.3 Present study 176

8.4 Results 179

8.5 Conclusion 185

9 Informing panel members about study results: Effects of traditional and innovative forms of feedback on participation 192
Annette Scherpenzeel and Vera Toepoel

9.1 Introduction 192

9.2 Background 193

9.3 Method 196

9.4 Results 199

9.5 Discussion and conclusion 207

Part III MEASUREMENT ERROR 215

Introduction to Part III 216
Reg Baker and Mario Callegaro

10 Professional respondents in nonprobability online panels 219
D. Sunshine Hillygus, Natalie Jackson, and McKenzie Young

10.1 Introduction 219

10.2 Background 220

10.3 Professional respondents and data quality 221

10.4 Approaches to handling professional respondents 223

10.5 Research hypotheses 224

10.6 Data and methods 225

10.7 Results 226

10.8 Satisficing behavior 229

10.9 Discussion 232

11 The impact of speeding on data quality in nonprobability and freshly recruited probability-based online panels 238
Robert Greszki, Marco Meyer, and Harald Schoen

11.1 Introduction 238

11.2 Theoretical framework 239

11.3 Data and methodology 242

11.4 Response time as indicator of data quality 243

11.5 How to measure "speeding"? 246

11.6 Does speeding matter? 251

11.7 Conclusion 257

Part IV WEIGHTING ADJUSTMENTS 263

Introduction to Part IV 264
Jelke Bethlehem and Mario Callegaro

12 Improving web survey quality: Potentials and constraints of propensity score adjustments 273
Stephanie Steinmetz, Annamaria Bianchi, Kea Tijdens, and Silvia Biffignandi

12.1 Introduction 273

12.2 Survey quality and sources of error in nonprobability web surveys 274

12.3 Data, bias description, and PSA 277

12.4 Results 284

12.5 Potentials and constraints of PSA to improve nonprobability web survey quality: Conclusion 286

13 Estimating the effects of nonresponses in online panels through imputation 299
Weiyu Zhang

13.1 Introduction 299

13.2 Method 302

13.3 Measurements 303

13.4 Findings 303

13.5 Discussion and conclusion 308

Part V NONRESPONSE AND MEASUREMENT ERROR 311

Introduction to Part V 312
Anja S. Göritz and Jon A. Krosnick

14 The relationship between nonresponse strategies and measurement error: Comparing online panel surveys to traditional surveys 313
Neil Malhotra, Joanne M. Miller, and Justin Wedeking

14.1 Introduction 313

14.2 Previous research and theoretical overview 314

14.3 Does interview mode moderate the relationship between nonresponse strategies and data quality? 317

14.4 Data 318

14.5 Measures 320

14.6 Results 324

14.7 Discussion and conclusion 332

15 Nonresponse and measurement error in an online panel: Does additional effort to recruit reluctant respondents result in poorer quality data? 337
Caroline Roberts, Nick Allum, and Patrick Sturgis

15.1 Introduction 337

15.2 Understanding the relation between nonresponse and measurement error 338

15.3 Response propensity and measurement error in panel surveys 341

15.4 The present study 342

15.5 Data 343

15.6 Analytical strategy 344

15.7 Results 350

15.8 Discussion and conclusion 357

Part VI SPECIAL DOMAINS 363

Introduction to Part VI 364
Reg Baker and Anja S. Göritz

16 An empirical test of the impact of smartphones on panel-based online data collection 367
Frank Drewes

16.1 Introduction 367

16.2 Method 369

16.3 Results 371

16.4 Discussion and conclusion 385

17 Internet and mobile ratings panels 387
Philip M. Napoli, Paul J. Lavrakas, and Mario Callegaro

17.1 Introduction 387

17.2 History and development of Internet ratings panels 388

17.3 Recruitment and panel cooperation 390

17.4 Compliance and panel attrition 394

17.5 Measurement issues 396

17.6 Long tail and panel size 398

17.7 Accuracy and validation studies 400

17.8 Statistical adjustment and modeling 401

17.9 Representative research 402

17.10 The future of Internet audience measurement 403

Part VII OPERATIONAL ISSUES IN ONLINE PANELS 409

Introduction to Part VII 410
Paul J. Lavrakas and Anja S. Göritz

18 Online panel software 413
Tim Macer

18.1 Introduction 413

18.2 What does online panel software do? 414

18.3 Survey of software providers 415

18.4 A typology of panel research software 416

18.5 Support for the different panel software typologies 417

18.6 The panel database 418

18.7 Panel recruitment and profile data 421

18.8 Panel administration 423

18.9 Member portal 425

18.10 Sample administration 428

18.11 Data capture, data linkage and interoperability 430

18.12 Diagnostics and active panel management 433

18.13 Conclusion and further work 436

19 Validating respondents' identity in online samples: The impact of efforts to eliminate fraudulent respondents 441
Reg Baker, Chuck Miller, Dinaz Kachhi, Keith Lange, Lisa Wilding-Brown, and Jacob Tucker

19.1 Introduction 441

19.2 The 2011 study 443

19.3 The 2012 study 444

19.4 Results 446

19.5 Discussion 449

19.6 Conclusion 450

References 451

Appendix 19.A 452

Index 457