Methodology

Methodology

Analyze the Attentive & Bypass Bias: Mock Vignette Checks in Survey Experiments

John V. Kane New York University
,
Yamil R. Velez Columbia University
,
Jason Barabas Dartmouth College
Abstract
Respondent inattentiveness threatens to undermine experimental studies. In response, researchers incorporate measures of attentiveness into their analyses, yet often in a way that risks introducing post-treatment bias. We propose a design-based technique—mock vignettes (MVs)—to overcome these interrelated challenges. MVs feature content substantively similar to that of experimental vignettes in political science, and are followed by factual questions (mock vignette checks [MVCs]) that gauge respondents’ attentiveness to the MV. Crucially, the same MV is viewed by all respondents prior to the experiment. Across five separate studies, we find that MVC performance is significantly associated with (1) stronger treatment effects, and 2) other common measures of attentiveness. Researchers can therefore use MVC performance to re-estimate treatment effects, allowing for hypothesis tests that are more robust to respondent inattentiveness and yet also safeguarded against post-treatment bias. Lastly, our study offers researchers a set of empirically-validated MVs for their own experiments.
Content
Thumbnail image of content item
Supplementary material
Thumbnail image of content item
Supplemental Appendix
This is the Supplemental Appendix for "Analyze the Attentive & Bypass Bias: Mock Vignette Checks in Survey Experiments". This appendix contains all wording for mock vignettes and mock vignette checks, as well as various analytics for these items. This appendix also contains a variety of additional analyses, including tests of MVCs' validity, demographic predictors of MVC performance, and detection of significant treatment effects among attentive subsamples.
Comments
Log in using your APSA account or Register to comment
Comments are not moderated before they are posted, but they can be removed by the site moderators if they are found to be in contravention of our Commenting Policy – please read this policy before you post. Comments should be used for scholarly discussion of the content in question. You can find more information about how to use the commenting feature here .
This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.