Analyze the Attentive & Bypass Bias: Mock Vignette Checks in Survey Experiments

13 October 2020, Version 1
This content is an early or alternative research output and has not been peer-reviewed at the time of posting.


Respondent inattentiveness threatens to undermine experimental studies. In response, researchers incorporate measures of attentiveness into their analyses, yet often in a way that risks introducing post-treatment bias. We propose a design-based technique—mock vignettes (MVs)—to overcome these interrelated challenges. MVs feature content substantively similar to that of experimental vignettes in political science, and are followed by factual questions (mock vignette checks [MVCs]) that gauge respondents’ attentiveness to the MV. Crucially, the same MV is viewed by all respondents prior to the experiment. Across five separate studies, we find that MVC performance is significantly associated with (1) stronger treatment effects, and 2) other common measures of attentiveness. Researchers can therefore use MVC performance to re-estimate treatment effects, allowing for hypothesis tests that are more robust to respondent inattentiveness and yet also safeguarded against post-treatment bias. Lastly, our study offers researchers a set of empirically-validated MVs for their own experiments.


post-treatment bias
survey experiments

Supplementary materials

Supplemental Appendix
This is the Supplemental Appendix for "Analyze the Attentive & Bypass Bias: Mock Vignette Checks in Survey Experiments". This appendix contains all wording for mock vignettes and mock vignette checks, as well as various analytics for these items. This appendix also contains a variety of additional analyses, including tests of MVCs' validity, demographic predictors of MVC performance, and detection of significant treatment effects among attentive subsamples.


Comments are not moderated before they are posted, but they can be removed by the site moderators if they are found to be in contravention of our Commenting Policy [opens in a new tab] - please read this policy before you post. Comments should be used for scholarly discussion of the content in question. You can find more information about how to use the commenting feature here [opens in a new tab] .
This site is protected by reCAPTCHA and the Google Privacy Policy [opens in a new tab] and Terms of Service [opens in a new tab] apply.