Informed International recently supported a review of World Vision International’s Literacy Boost[1] pilot programme results from 2013 to 2017. The review analyzed reading assessment results from 11 randomized control trials within 10 countries (11 programmes) in sub-Saharan Africa and South Asia. The purpose of the review was two-fold: first, to apply a methodology enabling country comparisons of results; and second, to compare World Vision International’s impact to a newly established World Bank benchmark of quality programming. After a comprehensive literature review, the World Bank determined that within the international education sector, “Effect sizes of 0.25 standard deviations or larger are considered to be substantively important. Effect sizes at least this large are interpreted as a qualified positive (or negative) effect.”[2] This is the first time that the education sector globally has established benchmarks for judging the effectiveness of programming.
Informed’s review found that 8 of 11 implementation sites saw a substantial effect on students’ reading achievement because of the programme (see Figure 1 below). In the four countries with particularly strong effect sizes, students received the equivalent of an additional year of learning as a result of the Literacy Boost programme. While effect sizes in three implementation sites fell below the World Bank benchmark, it is important to note those still had small-sized effects.
Figure 1. Effect size results, by rank, of 4 reading outcomes in 11 programmes
Unlike the other reading outcomes, World Vision International’s goal measurement - an increase in reading with comprehension - is calculated using Cramer's V effect size. As a general guide, values greater than 0.10 indicate a small impact, values above 0.30 indicate a medium impact, and those greater than 0.50 indicate a large impact. Figure 2 below shows the results of this analysis for all implementation sites
Figure 2. Effect size results, by rank, of ‘reading with comprehension’ goal indicator in 11 programmes
As evaluators, we are most encouraged that implementation sites observed similar effect sizes across reading outcomes, showing that Literacy Boost doesn’t only impact struggling students or succeeding students, but rather has a positive impact on children along the path to becoming a reader.
As part of the reporting process, InformEd used Tableau to create a dashboard and accompanying story to summarize the results of the review. The story provides an overview of the conclusions drawn from the research, while the dashboard allows for sub-group filtering of the results (by country, region, and by effect size rank). To dig deeper into the performance of specific programme sites, see the story and dashboard below.
