A systematic review of the United Nations Children’s Fund (UNICEF) estimated that 15 percent of all its reports included impact assessments, but noted that “many evaluations were unable to properly assess impact because of methodological shortcomings” (Victora 1995). A review of 127 studies of 258 community health financing programs found that only two studies were able to derive robust conclusions about the impact on access to health services (ILO 2002).
International consensus is growing that more and better impact evaluations are needed. Only by applying scientific rigor can development programs show that they produce results and offer a good return on the investment. The World Bank has made considerable investments in evaluating the impact of many of its development programs. Other donors like the UK and the Netherlands are joining forces. However, impact evaluations are expensive and not all programs need them. Here is the question: How to decide which programs should have an impact evaluation, who should decide and when? Let us know what you think!
Current practices and evaluation standards were discussed recently at a conference at The World Bank entitled “Making Smart Policy: Using Impact Evaluation for Policy Making” (January 15 and 16, 2008). Read more at http://web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EXTPOVERTY/0,,contentMDK:21567244~menuPK:336998~pagePK:148956~piPK:216618~theSitePK:336992,00.html
The goal of this conference was to show the power of impact evaluation for making policy. Much of the discussion focused on how impact evaluation can be a critical tool for identifying what works and what does not. Panelists and participants raised a number of issues, ranging from reasons why evaluations often do not inform decision making to the involvement – or not – of stakeholders.
While many issues were raised, some important questions had no satisfying answer:
How to decide which programs should have an impact evaluation?
What are or should be the criteria for making this decision? Evaluations can be decided based on a number of program characteristics, such as overall costs, innovative or exploratory nature, potential for great cost-effectiveness, or high visibility or political priority. If it’s the first, program costs, then we miss many small but highly cost effective programs implemented by NGOs. Each characteristic has it’s pros and cons.
How would you advise your organization and others to go about deciding when to evaluate program impact?
Who should decide?
Should the program implementers decide? Management? Some independent oversight body? There is clearly a conflict of interest for some of the deciders.
When should the decision about an impact evaluation be made?
For many of us the answer may be obvious, as early as possible in the planning of a program. However, in reality it may happen while the program is implemented. This limits what the evaluation can say about program impact. It may also indicate that evaluations are decided when programs seem to go well. Would we want programs evaluated when they do not seem to do so well? A rhetorical question! Even if less performing programs are evaluated, will we ever read about it?
I would like to hear your opinion about this. If you consider this an important topic, feel free to suggest whether the IH Section should deal with it. As we are planning the IH program at the annual meeting in San Diego, should we have a panel on Impact Evaluation? Don’t forget to submit an abstract and send us any other suggestions that you might have.
APHA Annual Meeting http://www.apha.org/meetings/highlights/
Abstract submissions: http://apha.confex.com/apha/136am/oasys.epl
Please note: the IH deadline is February 16.
Many thanks,
Eckhard
IH CONNECT 
