When does rigorous impact evaluation make a difference?
01 September 2011
Different
methods to estimate impacts are clearly suitable in different settings. But there is sharp
disagreement about exactly when high standards of rigour in impact estimation are a luxury and when
they are a necessity. Advocates of more rigorous impact evaluation argue that it can improve
incentives for development agencies by increasing transparency and avoid the waste of scarce
resources on attractive but ineffective projects. Advocates of more indirect and heuristic impact
evaluation methods argue that high demands for rigor are often better suited to academics than
practitioners, focus inordinate attention on easily quantifiable outcomes, take too long to yield
results, and divert scarce resources away from interventions already known to work well. The Center
for Global Development's Working Paper 225 (October 2010) seeks to contribute to this debate by
dissecting the case for and against rigorous impact evaluation in one concrete and high-profile
setting: the Millennium Village Project (MVP), a large, experimental intervention which aims to
spark local economic development in fourteen village clusters across Africa.
Comparing trends at the MVP intervention sites in Kenya, Ghana, and Nigeria to trends in the surrounding areas, the authors show how initial estimates of the project‘s effects change substantially if more rigorous impact evaluation methods than those used in the project‘s mid-term evaluation report are employed. The paper describes how weaknesses in the future evaluation design also limit its prospects of producing credible estimates of the project‘s effects. These weaknesses include the subjective choice of treated villages, the subjective choice of comparison villages, the lack of baseline data on the comparison villages, the small sample size, and the short time horizon.
The authors conclude with a discussion of how the initial phase of the project could have been designed—and future phases could be designed— to permit greater accuracy and clarity in assessing its impact. They note, however, that neither approach constitutes a rigorous impact evaluation of the MVP, which is impossible to perform due to weaknesses in the evaluation design of the project’s initial phase. These weaknesses include the subjective choice of intervention sites, the subjective choice of comparison sites, the lack of baseline data on comparison sites, the small sample size, and the short time horizon.
Full citation: Michael A. Clemens and Gabriel Demombynes. 2010. “When Does Rigorous Impact Evaluation Make a Difference? The Case of the Millennium Villages.” CGD Working Paper 174. Washington, D.C.: Center for Global Development.
