Lousy List, Lousy Survey?

We all know that a mail survey’s mailing list is the lifeline of a research project. The mailing list, or survey sample, is meant to represent your population of interest—the group of people you wish to learn something about.

Recently, we implemented a mail survey that was intended to represent a group of professionals. Based on previous history surveying this group, the length of the questionnaire and the use of two survey mailings (one initial mailing followed by a mailing to non-respondents), we expected a response rate in the neighborhood of 25% to 30%. However, at the close of fieldwork, the final response rate exceeded everyone’s expectations. It came in at 37%!

In the world of marketing research, exceeding response rate expectations makes everyone ecstatic. But in this case the data analyst discovered that almost all the survey respondents were 55 years of age and older. This couldn’t be right—it didn’t make sense!

Upon investigation, it was discovered that the client’s list management company did not select a random sample from the entire group of professionals in their database. Instead, they selected professionals based on how long their records were in the database.

Unfortunately the entire mail survey had to be done again so that the results would truly represent this particular group of professionals. And, if you’re curious, on round two the response rate came in a bit less than expected at 22%.

A bad survey mailing list happens on occasion. In this case, it generated a false sense of excitement over the response rate only to become a waste of time and money. Don’t let this happen to you. Make sure to check and double-check your survey list, be it a mail or online survey, before your project starts, just to be safe.