How accurate is the TI report ?
IN the 1936 US presidential election, Alfred Landon and Franklin Roosevelt were in the race for President.
Literary Digest, one of the most popular magazines of the time, conducted a large-scale survey to predict the election results. The Digest wrote letters to 10 million people seeking their opinion about the presidential vote.
About 2.5 million people responded and based on these results, the Digest predicted that Landon would easily win the election securing 57% of the votes. George Gallup was then the head of the American Institute of Public Opinion (AIPO).
Gallup conducted a poll of 50,000 people and based on its results, he predicted that Roosevelt would win the election securing 56% of the votes.
Gallup also conducted an auxiliary survey of 3,000 people and examined the results of the Literary Digest poll based on this survey.
Gallup concludes that the predictions of the Literary Digest are off by about 12%. When the actual election results came out, Roosevelt won with 62% of the votes.
The election results were very close to Gallup’s, while a huge poll by the Literary Digest had failed miserably. Roosevelt won the election easily, and Gallup prediction about error in Literary Digest’s poll was also correct.
It was such a surprising result; a survey based on a sample of only 50,000 people was correct and a survey based on a sample of 2.5 million was incorrect. Soon Gallup’s popularity skyrocketed and the Literary Digest went bankrupt and within two years the Digest was out of business.
The question is, how is it possible for such a large survey to give inaccurate results while a small survey produces accurate results? The dilemma could be understood looking at the sampling procedures of the two surveys.
For its survey, Literary Digest collected data from telephone directories and club memberships and wrote letters to all people in these lists.
Majority of respondents were selected from telephone directories. In those days, the telephone was a luxury in the United States that was only available to the upper middle classes.
The poor and lower middle class rarely had access to the telephone. Landon’s policies were more popular among the higher income classes, in contrast Roosevelt’s policies were more popular among the poor and lower middle class.
The proportion of high-income people in the Literary Digest survey was high and, therefore, their results showed London’s popularity among the rich.
Gallup, on the other hand, compiled a list of features that could influence voting patterns. For example, income, urban and rural, female and male, literate and illiterate and other such important features which have influence on election patterns were selected.
Gallup allocated quotas to each group in proportion to their population and selected the people according to their quota. Gallup staff interviewed selected people themselves and compiled the results based on which it became clear that Roosevelt would win.
Gallup also conducted a small auxiliary survey of 3,000 people using the Literary Digest’s method, and estimated how different the results of the Literary Digest would be from the actual results.
Based on this survey, Gallup gained the popularity that it has maintained till date and this legacy is also the basis of Gallup Pakistan. Gallup later improved the survey methodology and conducted Gallup polls before every US presidential election which has been popular for better predictions than any other organization.
Gallup’s popularity continues to this day and even today Gallup’s surveys are very meaningful to professionals.
It is clear from the above story that a large survey based on inappropriate methodology does not guarantee accuracy of results whereas a small survey based on correct method can give very accurate results.
In a good survey, every section of society should be represented in proportion to their population.
Transparency International and other international organizations, like Gallup, scientifically select samples (survey participants), so even though the samples are small, fairly accurate results can be obtained.
The fact is that Gallup had obtained the results of the Literary Digest based on a survey of 3000 people, which shows that if the method is correct, a survey comprising 3000 participants could be enough and samples of several million of samples are not required.
In short, even a small survey can make accurate predictions if the respondents are scientifically chosen and a large survey can give incorrect results, if the individuals are not selected correctly.
For example, if an organization interviews one million people from Karachi alone and concludes that MQM’s popularity is 35%, then these results may be correct for Karachi, but not for the whole of Pakistan.
Similarly, if the survey is conducted through the internet, the chances of getting responses from less educated and more engaged people will be very low. The proportion of younger people will be higher and the proportion of older people will be lesser than their population share.
Therefore, the results of this survey will not be representative of the total population of the country. Respondents for a good survey should be selected in such a way that every individual from the relevant population should have equal chances of being selected.
The accuracy of Transparency International’s election results also depends on the method adapted to select the sample. It should also be noted that Transparency International does not measure actual corruption, rather, it measures public sentiment regarding corruption.
Thus, the report of Transparency International shows how much the people agree with the claims of the present government to eradicate corruption. It is difficult to gauge the true extent of corruption from this report, however, the mood of the people can be gauged in a much better way.
—The writer is Director, Kashmir Institute of Economics, Azad Jammu and Kashmir University.
