A Difference-in-Differences Approach for Estimating Survey Mode Effects

26 June 2023, Version 1
This content is an early or alternative research output and has not been peer-reviewed at the time of posting.

Abstract

Two common designs for identifying survey mode effects are cross-sectional approaches and experiments. But cross-sectional designs risk a combination of omitted variable bias and post-treatment bias when conditioned on respondent characteristics that are themselves mode sensitive. In theory, experiments obviate these biases, but only if the experiment occurs in tightly-controlled settings that avoid differential uptake. Considering the costliness of such experiments, I propose a difference-in-differences approach for estimating mode effects. Leveraging mixed-mode panel surveys, mode effects can be identified by comparing changes in responses for panelists who switch modes across waves to those who remain in the same modes. Difference-in-differences offers a cost-free alternative to experiments and potentially large bias reduction gains vis-à-vis cross-sectional designs. I apply DD by estimating the effects of completing live interviews vs. web surveys on racial attitudes and political knowledge in the 2016-2020 ANES and cognitive functioning measures in the 1992-2020 Health and Retirement Study.

Keywords

Difference-in-Differences
Mode Effects
Survey Methodology
Public Opinion

Comments

Comments are not moderated before they are posted, but they can be removed by the site moderators if they are found to be in contravention of our Commenting Policy [opens in a new tab] - please read this policy before you post. Comments should be used for scholarly discussion of the content in question. You can find more information about how to use the commenting feature here [opens in a new tab] .
This site is protected by reCAPTCHA and the Google Privacy Policy [opens in a new tab] and Terms of Service [opens in a new tab] apply.