Analyze the Attentive & Bypass Bias: Mock Vignette Checks in Survey Experiments

Authors

Abstract

Respondent inattentiveness threatens to undermine causal inferences in survey-based experiments. Unfortunately, existing attention checks may induce bias while diagnosing potential problems. As an alternative, we propose “mock vignette checks” (MVCs), which are objective questions that follow short policy-related passages. Importantly, all subjects view the same vignette before the focal experiment, resulting in a common set of pre-treatment attentiveness measures. Thus, interacting MVCs with treatment indicators permits unbiased hypothesis tests despite substantial inattentiveness. In replications of several experiments with national samples, we find that MVC performance is significantly predictive of stronger treatment effects, and slightly outperforms rival measures of attentiveness, without significantly altering treatment effects. Finally, the MVCs tested here are reliable, interchangeable, and largely uncorrelated with political and socio-demographic variables.

Version notes

This is the version that has been peer-reviewed and conditionally accepted at Political Science & Methods. The file contains both the manuscript and Supplemental Appendix. The manuscript and Supplemental Appendix have been expanded to include additional discussion and analyses, including an analysis of how the Mock Vignette technique compares to alternative techniques.

Content

Comments

Log in or register with APSA to comment
Comments are not moderated before they are posted, but they can be removed by the site moderators if they are found to be in contravention of our Commenting and Discussion Policy [opens in a new tab] – please read this policy before you post. Comments should be used for scholarly discussion of the content in question. You can find more information about how to use the commenting feature here [opens in a new tab] .