MORE THAN MEETS THE ITT: HOW TO GET THE MOST OUT OF (APPARENTLY) FAILED EXPERIMENTS

Abstract

For a variety of reasons, experiments can yield non-significant and/or substantively weak intention-to-treat (ITT) effects. Such results can often be disappointing for researchers. In contrast to the extant literature on field experiments, there exists little explicit guidance on how researchers can more rigorously investigate weak ITT estimates in survey-based experiments. This essay highlights seven “pitfalls” that can lead survey experiments to yield null findings. While some of these pitfalls may be relatively familiar to researchers doing survey-experimental work, two such “pitfalls”—(1) respondent inattentiveness, and (2) failed manipulation of the independent variable—have only recently received substantial scholarly focus. This essay documents each pitfall and draws upon recent research to show how various “checks” or other techniques can be employed to detect and address them. Researchers are provided with specific strategies, and a checklist, for troubleshooting, diagnosing, and more thoroughly investigating their survey-experimental data before concluding that an experiment has “failed”.

Content

Comments

Log in or register with APSA to comment
Comments are not moderated before they are posted, but they can be removed by the site moderators if they are found to be in contravention of our Commenting and Discussion Policy [opens in a new tab] – please read this policy before you post. Comments should be used for scholarly discussion of the content in question. You can find more information about how to use the commenting feature here [opens in a new tab] .