Methodology

Increasing Precision in Survey Experiments Without Introducing Bias

Scott Clifford University of Houston
,
Geoffrey Sheagley University of Georgia
,
Spencer Piston Boston University

Abstract

The use of survey experiments has surged in political science as a method for estimating causal effects. By far, the most common design is the between-subjects design in which the outcome is only measured posttreatment. This design relies heavily on recruiting a large number of subjects to achieve adequate statistical power. Alternative designs that involve repeated measurement of the dependent variable promise greater precision, but are rarely used out of fears that these designs will bias treatment effects (e.g., due to consistency pressures). Across six studies, we assess this conventional wisdom by testing experimental designs against each other. Our results demonstrate that repeated measures designs substantially increase precision, while introducing little to no bias. These designs also offer new insights into the nature of treatment effects. We conclude by encouraging researchers to adopt repeated measures designs and providing guidelines for when and how to use them.

Content

Thumbnail image of CSP_Designs_May30_2020.pdf
cloud_download

Supplementary material

Thumbnail image of CSP_Designs_Appendix_May30_2020.pdf
cloud_download
Appendix
Appendix for “Increasing Precision in Survey Experiments Without Introducing Bias”

Comments

Log in or register with APSA to comment open_in_new
Comments are not moderated before they are posted, but they can be removed by the site moderators if they are found to be in contravention of our Commenting Policy open_in_new – please read this policy before you post. Comments should be used for scholarly discussion of the content in question. You can find more information about how to use the commenting feature here open_in_new .
This site is protected by reCAPTCHA and the Google Privacy Policy open_in_new and Terms of Service open_in_new apply.