Conferences

Despite increases in the transparency of elements of survey design in recent years, the process of questionnaire development has somehow bucked this trend. And the detailed decision making process for handling pilot data often remains a mystery . In the European Social Survey (ESS), we have tried to rectify such omissions both to improve transparency and to improve communication and decision making. We have therefore developed a coordinated approach to the post pilot decision making process, triangulating different sources of evidence on the performance of survey measures.

Half of the ESS questionnaire consists of ‘rotating modules’ on specific topics awarded to multinational teams. These modules go through a protracted period of design and development which aim to achieve reliability and validity, as well as attaining equivalence across countries and languages. This involves a wide range of quantitative and qualitative methods. Based on all of this input and the advice of substantive and methodological experts, items or scales are then adopted, re-designed or discarded.

The work on the rotating modules is carried out jointly by a selected team of academics in collaboration with a subset of experts from the Central Coordinating Team (CCT) of the ESS. During Round 4 we improved documentation of the questionnaire design process by asking the Question Design Teams and the CCT to document the process via a pre-designed template. Following a two nation pilot, a series of different types of evidence were entered into the template then all considered simultaneously. This included evidence from a range of methods including cognitive interviewing, feedback from the translation process, factor analysis to test expected relationships and anticipated scales in the data, non response analysis, split ballot analysis, other quantitative analysis to examine distributions in the data as well as evidence from interviewer debrief feedback.

This paper provides a methodological investigation into the use of these different evaluation and testing methods. In particular the paper will look for evidence of whether the data obtained using these different methods was comparable. Key questions include:

1) What evidence was there that different pre-testing techniques found the same errors?

2) What kinds of data did the different pretesting methods provide and how useful were these to the question designers?

3) How beneficial was it to present all of the evidence in a single template to facilitate multi method informed decision making?

References:

Converse J and Presser S (1986), Survey Questions. Handcrafting the Standardized Questionnaire, Sage: Beverly Hills.