"It is the peculiar and perpetual error of the human understanding to be more moved and excited by affirmatives than by negatives."--Francis Bacon (1561-1626)***
I recently read this quote from the father of the scientific method in an article on why the value of a certain baseball player was misjudged. Obviously, a baseball player’s value not being correctly understood is not an issue of great importance, yet this article led me to begin thinking about other situations where affirmation bias can truly have a poisonous effect.
I am a reader of “The Economist” and have been for many years. I enjoy the crisp and concise nature of the articles and what I believe is excellent and thorough international coverage. Any reader of the “The Economist” will agree that the magazine takes a pro-capitalism, pro-business and relatively libertarian stance on most issues. The magazine was even created with the intent to espouse the benefits of free trade.
In the August 2nd 2007 edition, the magazine ran an article entitled “In Praise of Usury” which reports some of the findings from Dean Karlan and Joanathan Zinman’s research “Expanding Credit Access: Using Randomized Supply Decisions to Estimate the Impacts.” “The Economist” article argues that perhaps analysts are wrong to be critical of “pricey, short-term and profitable” lending. The article uses Karlan and Zinman’s research, which was on the impact of a South African moneylender expanding credit to marginal clients, as proof that this sort of lending can be profitable for the lender and welfare-improving for the consumer. I recently read Karland and Zinman’s paper to see whether “The Economist” used this research fairly, or whether this was an example of affirmation bias or cherry picking what the magazines editors and readers wanted to hear. In Italics are some sections from the article, and in bold is what I believe is the more balanced reality from reading the article:
Did these profits [to the moneylender] come at the expense of the poor? On the contrary. Despite the demanding terms on offer, those reconsidered for a loan seemed to prosper. Six to twelve months later, they were less likely to go hungry, and their chances of being in poverty fell by 19%. Not coincidentally, they were also more likely to have kept their jobs, perhaps because the credit helped them to overcome emergencies that might otherwise have forced them to abandon their posts. About a fifth of them, for example, spent their loan on transport, such as buying or repairing a car that they might have needed to get to work.
I think the author's use of the word “prosper” is a bit strong. Although all of the statistics in this paragraph are accurate, the authors do not mention that the study found no effect on monthly consumption. The research paper also mentions that their statistics show “no effect on avoiding shocks on margins other than employment.” There is nothing egregious here, but definitely some cherry picking.
The results were not all as happy: the authors found some evidence of higher stress, especially among female borrowers. But people also reported more control over their lives and a more positive outlook. Perhaps the easier access to credit allowed them to take a longer-term perspective, even if “longer term” is measured in months or weeks rather than the more conventional notion of decades.
Contrary to the fears of the credit snobs, the readier access to credit did not tempt the new customers into a debt trap. Over 15-27 months, those reconsidered for a loan were more likely to have a formal credit score. And this score suffered no harm as a result of their easier borrowing.
Overall, the study suggests that profit-seeking lenders do not deserve the fate Dante reserved for them. Far from tempting the poor into unpayable debt, they help them keep their jobs, put food on the table, and build up a credit history. The authors show that poor people can make good use of borrowed money, even if they sometimes struggle to demonstrate this creditworthiness to lenders. If not hell, that is a kind of purgatory.
Again, there is nothing factually wrong here or even ridiculously hyperbolic, it is only the use of a tone which suggest that this was definitive research, and not the “low-powered” test the researchers claim to have done. Karlan and Zinman write in their paper that “We find no statistically significant evidence that consumers make themselves worse off by borrowing, as some behavioral models would predict. However, our standard errors are large and do not rule out economically meaningful negative impacts on some outcomes.” The researchers also mention that they were only able to observe impact over 6-12 months timeframe.
This exercise leads me to a greater question. What should we expect from mass media when reporting on data based research? Obviously, we cannot ask mass media authors to describe every nuance of study design and statistical interpretation, but I do believe we should expect intellectual honesty and an effort not to only take the information from research that affirms what the author or editor already believed.
I suppose the lesson is that it is essential to be vigorous about bearing in mind not just the agenda of a publisher or author, but their intellectual history, when they report statistical evidence. There is no greater bias than the bias towards wanting to be proved right.
*** This entry was inspired by Dan Fox’s article on Baseball Prospectus, “What Would Bacon Do?” which uses the same quote at the top of the article.
I recently read this quote from the father of the scientific method in an article on why the value of a certain baseball player was misjudged. Obviously, a baseball player’s value not being correctly understood is not an issue of great importance, yet this article led me to begin thinking about other situations where affirmation bias can truly have a poisonous effect.
I am a reader of “The Economist” and have been for many years. I enjoy the crisp and concise nature of the articles and what I believe is excellent and thorough international coverage. Any reader of the “The Economist” will agree that the magazine takes a pro-capitalism, pro-business and relatively libertarian stance on most issues. The magazine was even created with the intent to espouse the benefits of free trade.
In the August 2nd 2007 edition, the magazine ran an article entitled “In Praise of Usury” which reports some of the findings from Dean Karlan and Joanathan Zinman’s research “Expanding Credit Access: Using Randomized Supply Decisions to Estimate the Impacts.” “The Economist” article argues that perhaps analysts are wrong to be critical of “pricey, short-term and profitable” lending. The article uses Karlan and Zinman’s research, which was on the impact of a South African moneylender expanding credit to marginal clients, as proof that this sort of lending can be profitable for the lender and welfare-improving for the consumer. I recently read Karland and Zinman’s paper to see whether “The Economist” used this research fairly, or whether this was an example of affirmation bias or cherry picking what the magazines editors and readers wanted to hear. In Italics are some sections from the article, and in bold is what I believe is the more balanced reality from reading the article:
Did these profits [to the moneylender] come at the expense of the poor? On the contrary. Despite the demanding terms on offer, those reconsidered for a loan seemed to prosper. Six to twelve months later, they were less likely to go hungry, and their chances of being in poverty fell by 19%. Not coincidentally, they were also more likely to have kept their jobs, perhaps because the credit helped them to overcome emergencies that might otherwise have forced them to abandon their posts. About a fifth of them, for example, spent their loan on transport, such as buying or repairing a car that they might have needed to get to work.
I think the author's use of the word “prosper” is a bit strong. Although all of the statistics in this paragraph are accurate, the authors do not mention that the study found no effect on monthly consumption. The research paper also mentions that their statistics show “no effect on avoiding shocks on margins other than employment.” There is nothing egregious here, but definitely some cherry picking.
The results were not all as happy: the authors found some evidence of higher stress, especially among female borrowers. But people also reported more control over their lives and a more positive outlook. Perhaps the easier access to credit allowed them to take a longer-term perspective, even if “longer term” is measured in months or weeks rather than the more conventional notion of decades.
Contrary to the fears of the credit snobs, the readier access to credit did not tempt the new customers into a debt trap. Over 15-27 months, those reconsidered for a loan were more likely to have a formal credit score. And this score suffered no harm as a result of their easier borrowing.
Overall, the study suggests that profit-seeking lenders do not deserve the fate Dante reserved for them. Far from tempting the poor into unpayable debt, they help them keep their jobs, put food on the table, and build up a credit history. The authors show that poor people can make good use of borrowed money, even if they sometimes struggle to demonstrate this creditworthiness to lenders. If not hell, that is a kind of purgatory.
Again, there is nothing factually wrong here or even ridiculously hyperbolic, it is only the use of a tone which suggest that this was definitive research, and not the “low-powered” test the researchers claim to have done. Karlan and Zinman write in their paper that “We find no statistically significant evidence that consumers make themselves worse off by borrowing, as some behavioral models would predict. However, our standard errors are large and do not rule out economically meaningful negative impacts on some outcomes.” The researchers also mention that they were only able to observe impact over 6-12 months timeframe.
This exercise leads me to a greater question. What should we expect from mass media when reporting on data based research? Obviously, we cannot ask mass media authors to describe every nuance of study design and statistical interpretation, but I do believe we should expect intellectual honesty and an effort not to only take the information from research that affirms what the author or editor already believed.
I suppose the lesson is that it is essential to be vigorous about bearing in mind not just the agenda of a publisher or author, but their intellectual history, when they report statistical evidence. There is no greater bias than the bias towards wanting to be proved right.
*** This entry was inspired by Dan Fox’s article on Baseball Prospectus, “What Would Bacon Do?” which uses the same quote at the top of the article.
2 comments:
Baseball!! wow. hey have you read Moneyball.
of course, i love it. bible of exploiting market inefficiencies.
Post a Comment