Cost-Impact:

An Example from a Multi-Country Literacy Program

At InformEd International, we have seen that economic analyses [1] have been growing in popularity, as changes to the global development landscape necessitate a closer examination of international investments. Evidence-based decision-making has become an integral part of development program planning, and international NGOs are looking for the highest impact interventions for their limited program budgets. The clarity provided by economic analysis allows organizations to spend their money carefully and diligently while improving the transparency of investments.

In 2019, a large, international NGO approached InformEd International to develop a comprehensive evaluation, including an economic analysis, examining a multi-country program promoting literacy among primary school-aged children. The program was adapted to the needs of the 12 different countries in which it took place but contained the same core program activities.

In trying to summarize the costing for a particular country's program, we sought a way to determine the costs of each unit of impact. In other words, how much did the program cost for each new student who was able to reach a high level of literacy? Importantly, these students who could read and comprehend what they were reading by the end of the program had to be students who would have likely not achieved that milestone without the program. We started brainstorming using cost-effectiveness analysis [2], but it did not quite meet the needs of the program.

We had detailed budgets and expense reports, with each area of spending categorized by a code corresponding to a specific area of the program logical framework and impact information collected and calculated as part of the multi-country program evaluation. However, we did not have another intervention to compare the literacy project to, which is typically required to draw conclusions from cost-effectiveness calculations. Instead, all students received a public education, while some received additional support through the literacy program. We developed a novel method based on the data that we had, which we named cost-to-impact analysis.

The method might be appropriate for you, so we've broken down our analysis into five simple steps:

1. Gather expenditure information for all countries

In our case, we gathered expense reports and budgets for every fiscal year in all countries. To assist with other aspects of the economic analysis, we grouped expenses by four core program components and overhead and administration. In doing so, we could isolate only the direct program expenses used for the cost-impact ratio calculations.

2. Decide on an impact measurement, and gather impact information for all programs or program sites

One must decide on a unit of impact to determine the cost per impact. The unit of impact should be a quantifiable value that represents one unit of change or impact and can be applied to impact across all program areas. For example, a program could be focused on the number of lives saved or the number of families who move above the poverty line. In our case, this was the number of students who became readers with comprehension (RWC) at the end of the program, who would have not otherwise been an RWC if they had access to only the public education system and not the program run by the NGO. We used evaluation and monitoring data gathered in program areas and in comparison areas to inform the impact unit measurements. Students' literacy level was determined using literacy assessments administered to students at the start and end of the program in each country. As the cost-impact analysis accompanied a more extensive program evaluation, this step did not require any additional data collection to provide a more profound insight into the program.

3. Develop a cost-impact ratio

Cost-impact ratios, adapted from cost-effectiveness ratios, are determined by comparing the units of impact to the programs' cost. Direct program expenses are divided by the units of impact, determined in step two, resulting in the price per unit of impact:

Our analysis sought to answer how much was spent on each student who became Readers with Comprehension (RWC), who would not otherwise have reached that goal. To determine the denominator of the equation to the left, we subtracted the estimated number of students who became RWC in comparison areas from the number of students who became RWC in program areas. Expenses were then divided by the number of new RWC.

4. Analyze data and visualize results

We analyzed information in Excel, as the program's financial records were also kept in Excel files. After analyzing the data, visualizations can help to display and understand the results more thoroughly.

A simple bar graph can help to compare all the CIRs relative to the net cost per impact. This method of visualization is easy to interpret. The smaller the CIR, the lower the cost per impact. For example, the fictional country of Boscome has the lowest cost-per-impact, of $94 per student who became a new RWC, while the country of Monfinpo had the largest, of $2605 per student who became a RWC. The chart below uses the data that was included in our final cost-to-impact report, but the country names have been changed to fictional locations:

Plotting results on an x-y graph can help clarify the relative impacts and costs of each program or program site. Those points in the lower right of the graph represent the programs that have the highest impact for the lowest cost. Conversely, those in the upper left corner of the graph represent the programs with the lowest impact and highest costs. By examining this graph, we can see, for example, that the fictional country of Boscome, although not the country with the lowest direct program expenses, had the largest number of students who became new RWC, resulting in the lowest cost-per-impact.

5. Put the results into context

Although the cost-impact analysis alone provides useful information, it needs to be placed in a broader understanding of the cost, expense, and program information. It tells us the value per impact, but it does not tell us what specific program activities or adaptations lead to the value. The cost of a program is very closely tied to the program's length and coverage, and when implementation models vary depending on the program site, further exploration of the data is needed. We used a Value-For-Money framework [3] to help provide some of the necessary context by analyzing measures along the three dimensions of economy, efficiency, and effectiveness. We also examined how the ratio of spending on different core program components to overhead costs differed between countries and affected the program's results.

Like all analytic techniques, the cost-impact calculations have limitations. First, the accuracy and validity of the calculation depends on the quality of two sets of data (expense information, impact measurements). Expense reports are gathered as the program progresses, and sometimes lack accompanying information to help understand what may seem like errors or discrepancies. For example, while we were exploring the expense records of the program of interest, we discovered that there were some areas that required further knowledge to understand. For example, there were differences between countries in which logical framework code was used for some expense areas. Previous program managers were able to help provide the needed clarity to the expense reports, but without their guidance many unanswered questions would leave doubts of validity.

Despite the limitations, we believe that this method of determining the cost to impact of programs can be a useful technique to help programs and organizations allocate funding in a more informed way. We hope that its continued and expanded use will bring clarity and understanding to the different ways monetary costs can lead to impact.

[1] Centers for Disease Control and Prevention, Division for Heart Disease and Stroke Prevention, Evaluation and Program Effectiveness Team. Program Evaluation Tip Sheet: Economic Evaluation. Atlanta, GA. Accessed Jan 19, 2021.

[2] Brannen, C., Mahoney, M. & Tulloch, C. Estimating the Cost-Effectiveness of Education Programs: A Case Study. World Bank Strategic Impact Evaluation Fund (SIEF) & Abdul Latif Jameel Poverty Action Lab (J-PAL). Accessed Jan 19, 2021.

[3] Jackson, P. (2012). Value for Money and International Development: Deconstructing Myths to Promote a More Constructive Discussion. Organization for Economic Cooperation and Development (OECD) & Independent Development Evaluation.