How framing of survey questions makes a great difference

IN an episode titled The Grand Design of the BBC’s TV series Yes, Prime Minister, British PM Jim Hacker attempted to impose conscription; his private secretary, Bernard Woolley, had a survey done that showed support for the idea among the British. But Sir Humphrey Appleby, the Cabinet Secretary, showed Woolley how public surveys could reach contradictory conclusions by asking a series of leading questions and posing the central query in a particular manner.

This February, around four decades after the series was aired, market research organisation Ipsos carried out an intriguing survey in the UK. A representative quota sample of 2,158 British citizens, aged between 16 and 75 years, were questioned; half of the respondents were asked ‘Sample A’ questions, reflecting a positive view about national service. The other half saw ‘Sample B’, reflecting a negative view. A comparison of the results demonstrated that there were differences in the responses obtained on the degree of support for the reintroduction of national service in Britain, depending on how a question was posed, and the questions that come before it. In Sample A, 38 per cent were against reintroduction of conscription and 45 per cent favoured it. In Sample B, in contrast, 38 per cent supported it and 48 per cent opposed it.

One of the most important steps in ensuring a survey’s success is drafting the questionnaire. For creating a questionnaire that overcomes the biases and shortcomings of human cognition and yields relevant, reliable results, it’s essential to comprehend how people think and respond to questions.

In psychology, single questions are highly significant, yet they are often contentious. Daniel Kahneman, an Israeli-American psychologist and a pioneer of behavioural economics, provided an intriguing example with a fictional woman named Linda, 41, who studied philosophy, participated in anti-nuclear protests and was involved in political activities. “How likely is she to be a bank teller? Or how likely is she to be a bank teller who’s also active in the feminist movement?” asked Kahneman, recipient of the 2002 Nobel Prize in economics. It defies sense that 90 per cent of the respondents said she was more likely to be a feminist bank teller. However, to be a feminist bank teller, she also had to be a bank teller, which was the first option. But if the question is posed in this manner, the majority of the people will be misled. Thus, it’s not only the question you ask — it’s also how you phrase it that matters.

Consider another example from 1980s’ Britain. In a 1986 Gallup poll, 40 per cent of the respondents indicated that having nuclear weapons made them feel ‘safe’, while 50 per cent said they didn’t (the remaining had no definite opinion). However, these percentages changed when a different pollster substituted the word ‘safer’ for ‘safe’: 50 per cent of the respondents claimed that nuclear weapons made them feel safer, while 36 per cent said these made them feel less safe.

In June 1969, Gallup conducted a survey to find out how the US public felt about the ‘faster’ or ‘slower’ withdrawal of troops from Vietnam, following then President Richard Nixon’s order to withdraw 25,000 troops in three months. Although ‘same as now’ was not listed as an option, interviewers accepted respondents’ answers if they stated it out of the blue. As for the respondents, 42 per cent claimed it was ‘faster’, 16 per cent said it was ‘slower’ and 29 per cent said it was ‘same as now’. “In general, do you feel the pace at which the President is withdrawing troops is too fast, too slow or about right?” was the question Harris used in a similar survey shortly after the Gallup poll. In response, 49 per cent of the participants opined that the pace was about right, 29 per cent wanted a faster withdrawal and 6 per cent thought it was too fast. But why the disparity in these two polls? Well, they were different — Gallup didn’t include an explicit middle category.

In actuality, people react differently to gains or allowances than to losses or prohibitions. Kahneman and Amos Tversky popularised the concept of ‘framing’. In a 1981 publication, they modelled a fictitious outbreak of an unusual Asian disease that was predicted to claim 600 lives. Two alternative plans to combat the outbreak were put forth. If Program A was implemented, 200 lives would be saved. In Programme B, there was a one-third chance that 600 people would be saved and a two-third chance that none would be. In their survey, 72 per cent of the respondents were risk-averse and would rather save 200 lives than take a chance on saving a higher number. Subsequently, a second poll was carried out to choose between Programmes C and D, which had identical outcomes but different descriptions. The adoption of Programme C would result in 400 deaths. Programme D had a two-third chance of 600 deaths and a one-third chance that no one would die. Despite the fact that C and D were quantitatively equal to A and B, 78 per cent of the respondents now became risk-seekers and would rather gamble than accept the certain loss of 400 lives.

In his 2011 book Thinking, Fast and Slow, Kahneman tried to pinpoint the causes of such uncertainty. He clarified how people’s thought processes were a result of the combination of two distinct systems. The gut reaction or ‘System 1’ is “fast, intuitive and emotional”. ‘System 2’ is described as “slower, more deliberative, and more logical”. System 2 necessitates effort and concentration, whereas System 1 functions naturally and without conscious thinking, which typically comes into play initially. Only when it’s deemed necessary is System 2 activated.

“There you are, Bernard. The perfectly balanced sample,” Sir Humphrey said after demonstrating how different question formats might produce contradictory survey results. However, when they come across the findings of a public opinion survey, people seldom think about whether the questions were filtered, whether a ‘middle category’ was included among the alternatives, whether the questions were framed in terms of gains or losses, how the questions were posed and what were the questions that came before any particular one, and other related issues. Unfortunately, most people just assume that it’s Sir Humphrey’s ‘perfectly balanced sample’.