MORE THAN MEETS THE ITT: HOW TO GET THE MOST OUT OF (APPARENTLY) FAILED EXPERIMENTS

04 January 2023, Version 1
This content is an early or alternative research output and has not been peer-reviewed at the time of posting.

Abstract

For a variety of reasons, experiments can yield non-significant and/or substantively weak intention-to-treat (ITT) effects. Such results can often be disappointing for researchers. In contrast to the extant literature on field experiments, there exists little explicit guidance on how researchers can more rigorously investigate weak ITT estimates in survey-based experiments. This essay highlights seven “pitfalls” that can lead survey experiments to yield null findings. While some of these pitfalls may be relatively familiar to researchers doing survey-experimental work, two such “pitfalls”—(1) respondent inattentiveness, and (2) failed manipulation of the independent variable—have only recently received substantial scholarly focus. This essay documents each pitfall and draws upon recent research to show how various “checks” or other techniques can be employed to detect and address them. Researchers are provided with specific strategies, and a checklist, for troubleshooting, diagnosing, and more thoroughly investigating their survey-experimental data before concluding that an experiment has “failed”.

Keywords

noncompliance
inattentiveness
survey experiment
null results

Comments

Comments are not moderated before they are posted, but they can be removed by the site moderators if they are found to be in contravention of our Commenting Policy [opens in a new tab] - please read this policy before you post. Comments should be used for scholarly discussion of the content in question. You can find more information about how to use the commenting feature here [opens in a new tab] .
This site is protected by reCAPTCHA and the Google Privacy Policy [opens in a new tab] and Terms of Service [opens in a new tab] apply.