Analyze the Attentive & Bypass Bias: Mock Vignette Checks in Survey Experiments



Respondent inattentiveness threatens to undermine experimental studies. In response, researchers incorporate measures of attentiveness into their analyses, yet often in a way that risks introducing post-treatment bias. We propose a design-based technique—mock vignettes (MVs)—to overcome these interrelated challenges. MVs feature content substantively similar to that of experimental vignettes in political science, and are followed by factual questions (mock vignette checks [MVCs]) that gauge respondents’ attentiveness to the MV. Crucially, the same MV is viewed by all respondents prior to the experiment. Across five separate studies, we find that MVC performance is significantly associated with (1) stronger treatment effects, and 2) other common measures of attentiveness. Researchers can therefore use MVC performance to re-estimate treatment effects, allowing for hypothesis tests that are more robust to respondent inattentiveness and yet also safeguarded against post-treatment bias. Lastly, our study offers researchers a set of empirically-validated MVs for their own experiments.


Thumbnail image of Kane_Velez_Barabas_Analyze the Attentive and Bypass Bias.pdf

Supplementary material

Thumbnail image of Kane Velez Barabas_Analyze the Attentive and Bypass Bias_SUPPLEMENTAL APPENDIX.pdf
Supplemental Appendix
This is the Supplemental Appendix for "Analyze the Attentive & Bypass Bias: Mock Vignette Checks in Survey Experiments". This appendix contains all wording for mock vignettes and mock vignette checks, as well as various analytics for these items. This appendix also contains a variety of additional analyses, including tests of MVCs' validity, demographic predictors of MVC performance, and detection of significant treatment effects among attentive subsamples.


Log in or register with APSA to comment
Comments are not moderated before they are posted, but they can be removed by the site moderators if they are found to be in contravention of our Commenting and Discussion Policy [opens in a new tab] – please read this policy before you post. Comments should be used for scholarly discussion of the content in question. You can find more information about how to use the commenting feature here [opens in a new tab] .