Abstract
For a variety of reasons, experiments can yield non-significant and/or substantively weak intention-to-treat (ITT) effects. Such results can often be disappointing for researchers. In contrast to the extant literature on field experiments, there exists little explicit guidance on how researchers can more rigorously investigate weak ITT estimates in survey-based experiments. This essay highlights seven “pitfalls” that can lead survey experiments to yield null findings. While some of these pitfalls may be relatively familiar to researchers doing survey-experimental work, two such “pitfalls”—(1) respondent inattentiveness, and (2) failed manipulation of the independent variable—have only recently received substantial scholarly focus. This essay documents each pitfall and draws upon recent research to show how various “checks” or other techniques can be employed to detect and address them. Researchers are provided with specific strategies, and a checklist, for troubleshooting, diagnosing, and more thoroughly investigating their survey-experimental data before concluding that an experiment has “failed”.
Content
