By Susan McIntyre, MSW, MPH
Director of Evaluation of FHI's Contraceptive Technology Research
Program
McIntyre, who has worked with the Contraceptive Technology
Research Program for much of her FHI career, now works with colleagues
to assess and document the program's impact.
Evaluating the impact of health research is an essential component of
efforts to put research into practice. Donors and policy-makers are
often not satisfied with research results alone. Increasingly, they are
asking, "So what? What is the relevance of the research for
improving health services and outcomes?"
This issue should ideally be considered when research questions are
being formulated to ensure that the research will be relevant to
practice. Certainly, it must be considered once research results are
available and related promising interventions to improve health are
being selected for implementation. The first challenge then is to
identify interventions that "yield good value for money."
Subsequently, as interventions or health research results are selected
and moved into practice, the challenge is to determine whether they are
enhancing the short-term performance of specific programs. Finally, and
perhaps of greatest importance to donors and policy-makers, is the need
to determine whether research results and the selected interventions
have a longer-term, broader impact.
The Handbook of Indicators for Family Planning Program Evaluation,1
a publication produced by the Evaluation Project of the U.S. Agency for
International Development (USAID), distinguishes short-term performance
evaluation from long-term, or impact evaluation. In performance
evaluation, specific indicators may be gathered on program resources,
activities, or results. In contrast, impact evaluation generally uses
population-based data to measure the intermediate outcome (effect) or
long-term impact of an effort.
Contraceptive prevalence is an example of an intermediate outcome of
many family planning programs. In contrast, a change in fertility rates
is an example of a program's long-term impact. A change in fertility
rate, however, can take many years to occur. Furthermore, it is
difficult to identify clearly the factors that produced such change. For
these reasons, even longer-term evaluations tend to focus on
intermediate outcomes rather than impact.
Research on a checklist to rule out pregnancy among nonmenstruating
family planning clients (see Research Involving FHI
Collaboration Produces Change), serves as an example of looking at
intermediate outcomes. A study conducted by FHI in Kenya demonstrated
that the checklist correctly identified more than 99 percent of 1,629
nonmenstruating women visiting family planning clinics as not being
pregnant.2 When providers had training and good access to the
checklists, refusals by providers to give contraceptives to
nonmenstruating clients declined from 34 percent to 4 percent.3
Would introducing the checklist to rule out pregnancy have a
long-term impact on fertility rates in Kenya? This is far more difficult
to demonstrate. Nonetheless, the checklist's simplicity, low cost, and
short-term effect on provision of contraceptives in Kenya qualify it as
an intervention likely to yield good value for money. Follow-up research
in each of the settings where the checklist has been disseminated and
adopted would clarify whether its effects there were similar to those
observed in Kenya. Indeed, such research conducted by FHI in Senegal and
Guatemala has yielded preliminary data showing marked declines in
refusals to provide contraceptives to nonmenstruating family planning
clients in clinics after the checklist became available there. But
because the costs of replicating such research may outweigh its
benefits, FHI has also chosen to simply distribute a follow-up
questionnaire to those who requested the checklist for use in their
programs. This will help document the extent to which the checklist has
been used, accepted, and shared. It should also clarify the checklist's
usefulness in these settings.
The findings generated by this questionnaire will not determine the
long-term, broader impact of the checklist. But relatively simple
findings like these are often what most helps program planners and
policy-makers assess the value of research findings, ultimately
clarifying the answer to the question of "So what?" and
facilitating movement of research into practice.
References
- Bertrand J, Magnani R, Knowles J. Handbook of Indicators for
Family Planning Program Evaluation. Chapel Hill, NC: The
Evaluation Project, 1994.
- Stanback J, Qureshi Z, Sekadde-Kigondu C, et al. Checklist for
ruling out pregnancy among family-planning clients in primary care
[letter]. Lancet 1999;354(9178):566-67.
- Stanback J, Brechin SJG, Lynam P, et al. The Effectiveness of
National Dissemination of Updated Reproductive Health/Family
Planning Guidelines in Kenya. Final Report. Research Triangle
Park, NC: Family Health International, 2001.
Copyright