Survey Limitations
By Jim Prevor, Editor-in-Chief, Produce Business
It is well known that one problem with research is getting accurate numbers — numbers that represent real behavior rather than numbers that reflect what consumers think best to say. Yet perhaps even more challenging is drawing the right policy implications from a given set of facts.
In this article, we see highlighted an interesting result from the Gallup-Healthways survey: “Americans who say they have easy access to affordable fruits and vegetables are more likely to report having eaten healthy the day before and to have consumed at least five servings of fruits and vegetables at least four days per week than those who say they do not have easy access.”
It is easy to leap from that survey result to a policy recommendation that we could help the cause of healthy eating by increasing the availability of outlets that sell “affordable fruits and vegetables.”
Yet, though that policy choice is a possible deduction from the research, it is not the only possible one or even a particularly likely one. This is because a survey is different from a controlled test. If we took a large group of people who were selected to be representative of the American population based on race, ethnicity, religion, income, psychological attitudes, eating habits, etc., and then randomly assigned them to live in two different areas, one with easy access to affordable fresh produce and one without, we actually have no idea what would happen to their diets.
Perhaps those who currently eat lots of fresh produce would go out of their way to get it or switch to, say, frozen produce with little health impact. Or, perhaps the very presence of these consumers in the community would lead local retailers to carry more fresh produce, or the presence of a high produce- consumption demographic in the area would encourage everything from menu changes at local restaurants to new retail entrants in the market.
In other words, people to some extent self-select where they live, and a vegan who will only eat organic fresh produce is unlikely to move into an area where such items are difficult to acquire. Those who don’t care… well, they won’t care.
Imagine a survey that asked people whether they had easy access to an Orthodox Jewish Synagogue in their neighborhood. We can predict that in neighborhoods with such access, there will be a much higher percentage of Orthodox Jews than in neighborhoods where there is no such access. To leap from that fact to a conclusion that if we merely open more Orthodox synagogues, many more people will become Orthodox Jews has the phenomenon backward. The Orthodox Jews move to be near Orthodox synagogues — the presence of such houses of worship almost certainly does not lead to large numbers of conversions.
Another danger in reading survey research is to look at one variable and then confuse coincidence with causation. So, although when the Gallup-Healthway’s research team identifies an interesting correlation, even if it is music to the collective ears of the produce industry, we need to exercise caution. Here is the quote:
“Consumption of fresh fruits and vegetables also plays an important role in obesity reduction. The 10 states with the greatest and most consistent diet of fruits and vegetables carry an average obesity rate (defined as a body-mass index that is equal to or greater than 30) of 24.6 percent, nearly two full percentage points lower than the national 2009 rate of 26.5 percent. The 10 states with the least robust produce diet, however, carry an average obesity rate of 28 percent, over three points higher than what is found for the 10 best.”
We would like to believe that fresh produce consumption is the key here, but don’t think the study proves this. To name just one alternative perspective, perhaps high produce consumption correlates with some unknown third variable, say high education or high income or an aggressive attitude toward health, and perhaps these characteristics lead both to high produce consumption and many other things: An aggressive exercise program, frequent check-ups, discipline in taking recommended medications and following doctor’s orders, avoidance of saturated fats, etc. Perhaps it is one or several of these attributes that lead to lower obesity levels and, sacré bleu, if we didn’t change any of these items and just persuaded people to eat more fresh produce, they might actually gain weight due to consuming more calories.
Finally, asking consumers questions about the number of servings they ate or if they ate healthily is inherently problematic, because we just have no idea what consumers mean by these terms and how their interpretations of these words have changed over time.
The Gallup-Healthways partnership does exceptional work. Both organizations are careful to include both cell phone and landlines; they use random-digit dialing and offer Spanish-speaking interviewers. They include Alaska and Hawaii and have a methodology that includes many callback attempts to reach respondents who were unavailable on the initial call. In many ways, they are the gold standard in telephone research of this type. The issue is that survey research is inherently limiting. It often tells us more about what people think is best to say than what they do, and typically provides too little information to create clear policy options.