These notes should be read alongside the general advice on understanding quantitative data and qualitative data, as surveys allow us to collect both types.

Advantages

• Surveys provide an efficient way of collecting consistent data from large or small groups of people.
• Surveys are flexible and there are lots of options for asking different kind of questions. A questionnaire can collect a mix of qualitative data (through free text questions) and quantitative data (through closed questions that provide pre-coded response options).
• If there is an adequate sample, results can be generalised to the broader population that we are trying to represent or understand. This can’t be easily done through interviews or focus groups.
• Surveys collect data for all respondents in the same structure. This allows simple analysis of patterns and trends. For example, responses can be compared to see if there are differences in opinions based on students’ age or gender.
• Surveys are less personal and ‘hands on’ than focus groups and interviews. This can be an advantage when privacy and confidentiality is important.

Relationship to other methods

Surveys can be used as stand-alone data collection tools for collecting self-report data. In an evaluation, they are often integrated with other approaches as part of the evaluation design. For example:

• Initial interviews or focus groups might identify behaviours or attitudes that are of interest to participants and the evaluation. A survey would then allow us to estimate how common these behaviours are, and look for patterns or trends.
• Conversely, once we have analysed our survey data we might do follow-up interviews with selected survey respondents whose comments in the survey were unusual, unclear or particularly insightful.
• In some cases, the core purpose of a survey might be to enable focused sampling for in-depth qualitative inquiry. This is only possible with surveys that are not anonymous.
• If the survey responses don’t need to be anonymous, we may be able to integrate them in a spreadsheet with other data, such as administrative records, independent assessments or structured observation. This is a form of data triangulation that provides more options for analysis.

Limitations, and how to manage them

The success of a survey depends on asking good quality questions that suit the purpose of the evaluation. Unlike in an interview or focus group, the meaning of a question can’t be clarified part way through a self-complete survey, and once a survey has ‘gone into field’ it’s impractical to introduce an extra line of inquiry.

It is essential that questions make sense to the respondents, and that we aren’t forgetting to ask about something important. It’s very important to check a survey questionnaire with a few respondents before sending it out to everyone. ‘Piloting’ the survey first is the only way to be sure that respondents understand our questions in the same way that we intended them.

The best way of capturing data about behaviour is to observe it systematically and record it at the time. Asking people to tell us what they do (that is, to ‘self-report’) is less reliable and may be less efficient than collecting or referring to administrative data.

As with any self-report data, surveys rely on people expressing their opinions or describing their own experiences, attitudes or behaviours. This relies on people’s memory, as well as their willingness to disclose information or discuss a given situation. It also runs the risk of social desirability bias, which is the natural human desire not to ‘say the wrong thing’. This means we may need to verify or clarify things people say in a survey, by following up afterwards or checking other data sources. Individual follow-up is only possible with surveys that are not anonymous and where we have respondents’ consent and contact details.

People tend to fill out surveys quickly and not everyone will give a full response to an open-ended question, even if they have something important to say. If we need a personal approach that allows people to speak freely in their own words or we want to discuss a complex topic in detail, we may be better off undertaking interviews or focus groups.

Open-ended questions in surveys can take a long time to analyse, so we need to prioritise them and analyse the responses carefully. (Follow the link to Step 5: Organise and interpret the data.)

There are lots of reasons why people don’t respond to surveys: some people dislike doing them; others won’t get the invitation; others will simply forget. There are ways of lifting survey response rates, but we can’t expect to get a response from everyone unless we have a way of mandating it. (Follow the link to Step 4: Invite people to participate.)

If our sample is small or unrepresentative, we need to be very careful about how we interpret the findings.