Responding as Expected? The Effects of Survey Mode on Estimates of Sensitive Attitudes in Self-Completion and Face-to-Face Interviews of the European Social Survey

Conducting face-to-face surveys is increasingly challenging. The evolving technological and social landscape of survey data collection is prompting researchers to seek more sustainable solutions through new data collection techniques and a rethinking of traditional survey methods. Recent examples include the Push-to-Web (PtW) survey experiments initiated by the European Social Survey (ESS) research consortium, which use probability sampling to reach respondents via postal invitation and combine web- and paper-based self-completion questionnaires. The literature indicates that the data collection method may strongly influence who participates in a survey and how people answer questions. These two phenomena are captured by the concept of the mode effect. This paper evaluates the PtW surveys in Hungary from the perspective of mode effects by pairing two PtW surveys with face-to-face ESS surveys closest in time. The evaluation examines sample compositions and responses to two culturally and socially sensitive questions for which the survey mode is considered a highly relevant factor given Hungary’s specific political context: attitudes toward (1) gay and lesbian people and (2) immigrants. Responses to these questions from both surveys are assessed using one-, two-, and multidimensional (GLM) analyses. The results show consistent patterns across both survey pairs. (1) PtW surveys yield similar response rates to face-to-face surveys but tend to appeal to more highly educated and older segments of the population. (2) Despite post-stratification to correct sample composition, the results still reflect different attitudes toward gay and lesbian people and immigrants across the two survey modes. (3) The survey mode has an independent impact on sensitive measurements: in GLM models, the survey mode significantly affects how respondents report their attitudes and overrides the original demographic correlations.