A Difference-in-Differences Approach for Estimating Survey Mode Effects

Two common designs for identifying survey mode effects are cross-sectional approaches and experiments. But cross-sectional designs risk a combination of omitted variable bias and post-treatment bias when conditioned on respondent characteristics that are themselves mode sensitive. In theory, experiments obviate these biases, but only if the experiment occurs in tightly-controlled settings that avoid differential uptake. Considering the costliness of such experiments, I propose a difference-in-differences approach for estimating mode effects. Leveraging mixed-mode panel surveys, mode effects can be identified by comparing changes in responses for panelists who switch modes across waves to those who remain in the same modes. Difference-in-differences offers a cost-free alternative to experiments and potentially large bias reduction gains vis-à-vis cross-sectional designs. I apply DD by estimating the effects of completing live interviews vs. web surveys on racial attitudes and political knowledge in the 2016-2020 ANES and cognitive functioning measures in the 1992-2020 Health and Retirement Study.