In the “Feedback” section of the holiday New Scientist (p. 88):
The sign-up questionnaire on the Captain Cash money-saving website run by the London-based Sunday tabloid News of the World asks applicants to select from the following: “In a typical month, I buy the NOTW: never/less than once/1-2 times/3-4 times.”
Michael Barraclough is still trying to work out which of the first two options to tick.
You can see what the questionnaire was trying to tap: the average frequency with which someone buys the NOTW; that’s what “in a typical month” is supposed to convey. So the options were supposed to be understood as:
I never buy the NOTW;
/ some months, I don’t buy it, but some months I buy one copy;
/ in a month, I buy one or two copies;
/ in a month, I buy three or four copies.
This is still problematic (for reasons I’ll discuss in a moment), but the first thing to note is that ordinary people are not at all good at estimating such frequencies; the task requires averaging over a number of months, and people are unlikely to hold such data in memory. They can report what they did in the most recent month, or what they intend to do in a typical month, but that’s not what the survey wants to find out; it’s supposed to gauge readership. Readership can be estimated by indirect and somewhat complex methods, but self-report surveys are the cheap and easy way to go.
Suppose, however, that we reformulate the questionnaire in terms of averages (that is, means):
I never buy the NOTW;
/ the average number of times a month I buy the NOTW is greater than 0 but less than 1;
/ the average number of times is greater than 1 but less than 3;
/ the average number of times is 3 or greater.
There are other possible reformulations, but this one seems to me to come close to the intentions of the original questionnaire. In particular, the second clause (an average greater than 0 but less than 1) looks like a good fit to the original second clause (less than once in a typical month), and not so good a fit to my first reformulation above (no copies in some months, one copy in some months) — because the second formulation allows for possibilities other than averaging some number of 0s with some number of 1s (three 0-copy months with one 2-copy month, for instance.
Another complexity is that though most calendar months have four Sundays, some have five, and very occasionally there’s a February with only three [added: not really so; see Kivi Shapiro's comment below], so that the averages will depend on which particular months are in the mix. (Of course, you could try reformulating things with month referring to four-week periods beginning with a Sunday, but ordinary people just won’t cope with that.) [added: for a possible solution, see Chris Hansen's comment below]
These are small points, involving minor effects, and quite possibly my first reformulation (which is at least comprehensible to ordinary people) will do for practical purposes.
There is a larger point here, about how difficult it is to construct surveys that tap what you want them to.