Viewpoint: Getting back in the frame

Trevor Sharot

When I entered the market research business in the 1970s, there was (in the UK) a simple choice as to how to draw a sample for a survey of the general population. If a random sample was required, then the Register of Electors was used as the sampling frame. This was generally the case when quality was of prime importance, as in most social and governmental research. The same rigorous standards were applied to ongoing surveys, such as omnibus and media and consumer panels. Panels were an interesting case, for which quality was again very important, since they needed to be properly projectable to the population; the investment in a sampling frame and associated processes could be amortised over time. For much other work, including most customised surveys, a quota sample was considered more appropriate, given the usual time and budget constraints. Quota sampling was used on the understanding that such a sample came without the ‘warrantee’ that a random sample offered, such as freedom from selection bias (though non-response was always an issue) and the ability to compute standard errors. On the other hand, there was a considerable body of evidence that quota samples could deliver sufficiently accurate data most of the time (and even a random sample gets it wrong 5% of the time, working at 95% confidence level). At least the quotas and weighting targets were derived from the census and high-quality random samples such as the NRS.

At that time, telephone penetration was already high enough in the United States to be valid for telephone number sampling, and once Mitofsky (1970) and Waksberg (1978) had described two-stage random-digit dialling, the method gained wide acceptance. In the United Kingdom, penetration lagged behind, as evidenced by Collins (1987), but was soon considered adequate. Moreover, another alternative, the Postal Address File (PAF), was starting to be evaluated and was available on CD by the early 1980s. Despite the absence of information on household membership, its virtually complete household coverage gave it an advantage over the Register of Electors, which was exhibiting reducing coverage over time, especially among young, mobile and immigrant households.

In short, the UK became spoilt for choice if random sampling was required, and was well equipped for conducting respectable quota surveys. Meanwhile, other countries arrived at their own solutions. In the early 1960s, British American Tobacco had encouraged Newell Grenfell to set up Survey Research Malaysia to provide random samples there, spawning the Survey Research Group across Asia (now part of Nielsen). To this day Nielsen Malaysia continues to maintain schematic maps of every block and dwelling in the country. And, everywhere, random sampling was used to select sampling points from a file of government administrative districts, even if random route or quota sampling was employed for the final stage of selection. This gave proper geographical and social coverage, while also providing control of clustering and its inherent dangers (Harris 1977).

Fast forward to the noughties. For the majority of surveys, sampling points are no longer selected from administrative lists and households are no longer selected from registers, nor even by random routes. In fact, households are no longer selected at all. Instead, people are invited to take part in surveys online via a website or email.

The pool of available households represents an unknown proportion of the population. It is not known which member of the household is actually responding, or even for sure where the household is located, only what IP address the computer is identified by. Active internet users have probably been asked frequently to participate in surveys in the form of feedback questionnaires after an encounter with a vendor. They will have no way of distinguishing these from general population surveys.

As for response, an unknown number never cooperate, or have tired of doing so. However, response and non-response can no longer be identified so overall response rates are unknown, let alone the breakdown into ineligibles, refusals and non-contacts. There is, of course, no possibility of making repeated attempts to elicit higher response as in the ‘recalls’ of the past. It is known that a minority of the public have turned themselves into frequent, so-called professional, respondents in order to benefit from the rewards on offer.

Further, with no interviewer being involved in the process, the ‘human contract’ has been lost, probing has become mechanical rather than a skill, and back-checking performed by supervisors has all but disappeared, while casual confessions via forums, blogs and tweets reveal that some respondents simply straight-line their responses, with a few curved balls thrown in to dampen the suspicions of the editing software.

The main response of the industry, apart from making motherhood statements, has been to develop software of increasing sophistication to identify ‘professional’ and ‘inventive’ respondents. The most pressing issue – unknown coverage of the multiplicity of proprietary lists being used as sampling frames – has merely resulted in exchanges between vendors along the lines of ‘mine is bigger than yours’, with not a shred of supporting evidence.

Market research has a serious problem, and clients are being misled as to how serious it is. What is needed is a decent sampling frame of known coverage that everyone can evaluate and use, or at least purchase. Without this there can be no representative samples, no response rates and no measures of sample quality. The industry must organise itself around the need for this frame and decide what system of collaboration is needed to create it. With such a joint effort, funding should be no problem and would be used far more efficiently than with vendors competing with each other. Their effort can return to conducting and analysing research better than their competitors do.

Now here is the clincher. In the past, the time and cost for conducting door-to-door random samples was much greater than for quota samples. On the web, this is no longer the case. With a decent frame in place, accurate and targeted selection, measurement of response, chasing and elimination of multiple responses become straightforward: instantaneous tasks executed by software, and at no greater cost – perhaps less – than now. Also, with better identification of households, some part of the ‘human contract’ can be re-established and better control exercised over the survey process itself.

Would the Market Research Society like to play its part in this?

References
Collins, M. (1987) The problems of non-coverage and unlisted numbers in telephone surveys in Britain. Journal of the Royal Statistical Society, Series A, 150, pp. 241–253. doi:10.2307/2981475
Harris, P. (1977) The effect of clustering on costs and sampling errors of random samples. Journal of the Market Research Society, 19(3), pp. 112–122.
Mitofsky, W. (1970) Sampling of telephone households. Unpublished CBS memorandum.
Waksberg, J. (1978) Sampling methods for random digit dialing. Journal of the American Statistical Association, 73, pp. 40–46. doi:10.2307/2286513

Trevor Sharot is Managing Director of Red Research, a consultancy specialising in media research, sampling, methodology and advanced analysis

International Journal of Market Research 52(2), 2010

Would you like to respond to this Viewpoint? Or perhaps you have an idea for another? Responses and new submissions are welcome. They should be emailed to the IJMR, where they’ll be considered for publication.

What's New - Membership - Company Partner Service - Members' Area - Code/Guidelines - Qualifications - Training - Awards
Events - Networking - Publications - Media Info - Market Research - Search - Site Map - A-Z Directory - Contact Us - Home