Read the Small Print

This last Sunday I picked up the Indian Express while waiting for a train, and was baffled when I read the following headline, "People Say No to Israel-type Action Against Pakistan." The article details the results of a "six-question poll of over 600 residents of Chennai, Bangalore, Hyderabad and Kochi..." which indicated that 72% of those polled believed Israel was not "correct in attacking Gaza, first by airstrikes then by invasion, to sort out Hamas."

My initial reaction was twofold. 1) I was happy that Indian people seemed to be so averse to the use of force 2) I was skeptical of the polls methodology. It did not seem appropriate that the headline referred to the general "People" when it was only referring to those living in major cities. Plus, I thought it peculiar that so many of the people polled were aware of the attacks and the organization Hamas (a very small number of people responded "Don't know/Can't say" to any of the questions).

My detective skills led to to small print in the bottom right section of the article. It reads as follows:

"Methodology: Research organisation C fore conducted the survey on behalf of The New Indian Express in four cities of south India, Bangalore, Hyderabad, Chennai and Kochi, on 5-7 January 2009. A questionnaire was adminstered to a statistically selected sample of 608, and only those who were aware of the of the Israel atack on Gaza were selected. Person belonging to different socio-economic categories were interviewed. All figures are in percentages."

I don't think I am being a stickler in thinking that the fact that "only those who were aware" of the attacks were interviewed was probably worth mentioning in the body of the article and perhaps even in the headline. I believe this article in incredibly misleading because of a substantial and essentially unacknowledged selection bias. Regardless of education and economic status, the type of person who is informed about a certain international event is fundamentally different than their counterpart who is not informed. This is not to suggest that they should have conducted a poll that included a completely random cross section of the Indian population (as this would likely lead to the uninteresting result that most Indians are not aware of the organization Hamas), only that methodology deserves a greater space when presenting data.

For an article on how selection bias also affects microfinance impact research see Dean Karlan's excellent "Microfinance Impact Assessment: The Perils of Using New Members as Control Group." The article explains the faulty assumptions behind assuming that the difference between "veterans" of MFIs and "new members", even controlling for other variables, is the true impact of MF.