How Might Response Rate Affect Your Survey?

Look at the example below to see the effect that non-response bias might have on survey results. Let’s say you conduct three surveys of the same sample of ten students hoping to learn the percentage that received A’s in a recent class:

Survey 1:

Only two of the ten students respond. They both received A’s, so your survey reports 100% of the students received A’s.

2-resp-

DEFINITIONS:

Non-Response Bias:
The error resulting from distinct differences between the people who responded to a survey versus the people who did not respond.

Response Rate:
The percentage of people invited to respond to a survey that actually returned a usable survey.

Sample:
The group of people invited to take part in a survey.

Survey 2:

Four of the ten students respond, including the two who received A’s, so according to this survey, 50% of the students received A’s.

4-resp

Survey 3:

Six of the ten students respond. With two receiving A’s, this survey reports that 33% of the students received A’s.

6-resp

So, what impact did non-response bias have on our three surveys?

Actual Sample:

In reality three of the ten students (30%) received A’s.

All-10

What does it mean?

Here’s the breakdown of the potential magnitude of non-response bias for the three surveys:

Survey 1 = 70 Percentage Points Off

Survey 2 = 20 Percentage Points Off

Survey 3 = 3 Percentage Points Off

These surveys illustrate how non-response bias can affect how well data represents the population being surveyed. If someone relied on Survey 1 data for decision-making, they’d be way off. Survey 2 is much closer to the actual, but is still not a very good estimate. Survey 3, though, with a higher response rate, gives a pretty close estimate of the true value—only 3 points off.

Non-response bias can become an issue when there are distinct differences between the people who respond to a survey and the people who don’t respond. For example, students receiving A’s may feel more confident sharing their grade than the one who received an F, even if the survey is anonymous.

Keep this example in mind the next time you’re using survey results and be aware of the response rate.

Download White Paper