Do we get meaningful information from the results of surveys we read about in the news?
This week a couple of press releases on what we do not eat have crossed my path.
In the first, according to data from Nielsen, over four in ten Britons are now on restricted diets, such as ‘sugar conscious’, low fat, low sodium, low carb, veggie, wheat/gluten-free, dairy free or halal.
The researchers claim 49% try to avoid antibiotics / hormones, 45% additives, 42% sugar, 41% GM foods, 40% products in packaging made using BPA (which some campaign groups speculate is linked to cancer), 39% MSG, 37% saturated / transfats, 35% sodium and … it goes on.
One in five households has a food-sensitive member, they say, and the most common ingredients avoided for this reason are grains (43%), eggs (38%), lactose/dairy (36%) and gluten (30%).
I asked and received fuller results and data, and on closer analysis some questionable figures present themselves.
* Of those avoiding organic foods, 13% do so because ” … someone in my household has a medical condition that prohibits consumption … ”
* 11% of responders “try to include” in their diet food products contained in a package made with BPA.
* 18% of avoiders of superfoods (chia, goji, acai) do so because of a medical condition, and 35% because they believe them harmful.
I know of no condition requiring the avoidance of so-called superfoods or organic foods, and I have to question whether many people even know what BPA is, let alone make a deliberate effort to expose themselves to it.
“The Rise of the ‘Varitarian'”
That was the title of another release I received, covering research by Censuswide and commissioned by Zizzi restaurant chain – to coincide with their new autumn menu that includes a new vegan lentil ragu, and more vegan gelati and wines. The release claimed that the average diner has tried at least three different diets this year.
It’s not clear to me what an ‘average diner’ is. Again, I was offered the detailed results of the survey of 2,000 respondents. It seems the average number of diets tried by diners in the last year was in fact 0.88. 69% tried none; 24% tried one, two or three; 7% tried four or more.
Other figures in the press release do not to my mind reconcile with the spreadsheet of data I’ve seen, so I won’t quote them, as I suspect they were misinterpreted. However, one interesting statistic concerned the number of dietary preferences on a night out among friends: around 40% of respondents said at least one needs to be accommodated within their immediate social circle – a figure food service providers need to heed if they wish to capture this potentially large market.
But again there are inconsistencies. Variations of virtually the same question to the same pool of people gave widely differing results: “Which diets do you normally have to accommodate in your friendship group?” (36% said none) and “On an average Saturday night meal out with a group of friends, how many dietary preferences do you have to accommodate?” (50% said none).
Can surveys and responders be trusted?
Not fully, no, I don’t think.
Results claiming to represent the whole country need to include a representative sample of people from the whole country – with each person having an equal chance of being selected for survey.
Email or internet surveyors have a limited pool of people – not merely those with online access, but also those whose email addresses they have and/or who have registered or agreed to take part in surveys.
Usually, even those who have agreed in principle to take part then self-select further and volunteer to take part in a particular survey – perhaps responding to a request or advertisement or influenced by a friend to do so. This results in a further skewering of results, of course. People with an interest in food and special diets, in some form, are more likely to complete a survey about food and special diets, particularly if they are incentivised with the offer of vouchers, for example, in return for their time.
Which brings in another problem issue – rewards. Some of those completing any survey offering a reward may be completing it primarily to receive that reward – and therefore I think it’s fair to question the accuracy of the responses they might give. After all, survey completion is the aim – not correct and honest survey completion. It’s hard not to speculate that this may lead to such peculiarities in results described above – although I know that some market research companies do have protocols in place to weed out ‘career responders’.
Nevertheless there are, in my view, simply too many biases involved in online surveys.
Does it matter?
Yes – not least because newspapers and their journalists are not digging deeper to look at the real story, which may be more interesting than the one presented.
But it’s a symptom of a bigger problem in media these days, that to get your name in the papers only requires that you engineer some noise and provide easy copy. For instance, you can do this by fabricating an enormous pisstake – as did the White Moose Cafe recently, whatever the fallout might be.
Or you can hire PRs to push gushing case studies to sometimes passive journalists who know how effortlessly these can be turned into eye-catching stories or sold to the papers. This sort of thing regularly happens with York Test and their IgG testing services for food intolerance.
Or you can pull together a survey, which doesn’t cost that much, I imagine, and put out a press release based on it with some quotable figures and stats. Doesn’t really matter how accurately representative the numbers are – because it gives the media something to fill their pages with regardless, and any waffle about gluten, or sugar, or clean eating, or vegans, or additives, or diets, will probably generate some coverage, especially online, where it serves as superior clickbait in this age of public obsession with diet.
To my mind what all this stuff tells us is that there are a lot of people with vested interests in getting – and keeping – free from in the news, irrespective of the value of the information provided. But how much of that helps those who actually need to live free from lives, I wonder?