Quasi-Experimental Designs and Quasi-Valid Results

Performing a randomized evaluation of a social program can be a real pain. In fact, randomized evaluations are probably the worst method for evaluating the impact of a social program out there – except for all the other methods available.

The problem with the other methods, in the case of microfinance and many other programs where there are likely to be a lot of unobserved differences between participants and non-participants, is that no matter how sophisticated the econometrics used it is extremely difficult to account for these unobserved differences. In addition, “publication bias” (the bias of journals to only publish papers with interesting, as yet unseen, results) and the strong public demand for positive results can lead to systematic bias in what ends up in the public domain.

Perhaps the most famous example of a non-randomized impact evaluation of microfinance is Pitt and Khandker’s evaluation of the microfinance programs of BRAC, Grameen Bank, and BRDB. Pitt and Khandker used the rules for who can be a client of these institutions as a sort of technique for controlling for the unobservable differences between clients and non-clients and found that the extension of microfinance had a significant positive on households (and more so in the case that the loan was offered to a woman).

In a new paper, David Roodman and Jonathan Morduch find that the Pitt and Khandkher results don’t hold up to scrutiny. Not only did their specification tests reveal that the methodology used by Pitt and Khandker is likely faulty, but they were unable to even replicate the key result of the paper using the original methodology of the authors. Roodman and Morduch take pains to say that they don’t claim that microfinance has no impact on poverty – only that the Pitt and Khandker research provides no evidence that it does.