Why do we measure impact? Well, in simple terms, it helps us do our jobs better.
Having a robust monitoring and evaluation system is like the current-location tracking dot on your GPS system. It allows us to see where we are compared to our planned route and destination. It allows us to redirect our efforts, correct our course or speed up.
“Impact implies changes in people’s lives. This might include changes in knowledge, skill, behavior, health or living conditions for children, adults, families or communities. Such changes are positive or negative long-term effects on identifiable population groups produced by a development intervention, directly or indirectly, intended or unintended." – United Nations, 2012
There are many types of information to take into account. Number of people we reach with certain programmes, meetings and events held, materials developed and shared, resources invested -- these can all give us ideas of how we are doing. To understand impact, it is important to look what changes with our efforts, measuring specific outcomes, like decline in child labour or increase in school enrollment, and social impact, which might be increased well-being or confidence of people we are working with.
SROI
Social Return on Investment is a methodology to try to understand the "social value" that is created by a certain project or activity. It enables us to assess the social value of our work through the eyes of the beneficiaries to improve programme design and build far-reaching and sustainable interventions. In doing so we are keeping farmers, families and children at the heart of what we do.
For example, a Village Savings and Loan Association can help parents in farming communities better manage their family finances and save for important cost, like school fees. Some traditional measurements of the benefits of a VSLA group might be the amount of money saved or the number of people participating.
However, when we look at social value, we ask the group members what changed in their lives. For VSLAs, we heard many responses, like improved confidence and sense of well being, families needs being met because of more stable financial situation, a sense of independence and hope, especially for women.
In 2020, we drew conclusions from the results of social return on investment studies (SROI) that were conducted in Uganda, Tanzania, and Malawi to streamline the Foundation’s programme approach. The new programme approach for addressing child labour rests on the three following pillars:
- Strengthening farmers' and households' incomes.
- Supporting quality education, vocational and skills training as alternatives to child labour.
- Strengthing local capacity to prevent, respond and remediate child labour.
Monitoring and Evaluation
Social Return on Investment complements our on-going tracking of our project work, to ensure we have a good sense of how we are doing. Regular contact with our implementing partners and quarterly progress reports against core indicators help us to know things are on track.
In addition to regular reporting, ECLT has a 3-step process for evaluation of our efforts on the ground. These include internal data collection and regular independently-verified assessments, both qualitative and quantitative, to measure change over time. Here is how the overall process works:
• Step 1: What is the current context?
A Baseline study is done by an external evaluator to take a snapshot of the status of children, households and the communities in general, in order to collect the initial value of a set of indicators. This information will serve to develop a solid programme strategy and ensure effective and sustainable impacts.
• Step 2: How well is the project delivering?
A mid-term assessment is carried out half-way through the project implementation to take stock of the project activities, outputs, project outcomes/results and project management. The assessment will determine if the project is likely to achieve its stated outputs and immediate objectives and will draw lessons from the experiences gained so far during the implementation. Based on this, the project can be adjusted to have a greater impact.
• Step 3: What has changed for the beneficiaries?
A final evaluation is commissioned is to get an external, independent assessment of the project achievements at outcome and impact levels. The final evaluation draws good practices and lessons learnt to optimise future programme design.
These 3 steps enable us to assess the relevance of the project (design and planning), the effectiveness (implementation), the efficiency, the impact, the sustainability. The ECLT evaluation cover the most critical areas of the project and adheres to the overarching evaluation criteria recognized by the UN System Evaluation Standards and Norms and the OECD/DAC Evaluation Quality Standards of relevance, effectiveness, efficiency, sustainability, and impact.
How we do it -- the evaluation process
ECLT follows a rigorous approach to evaluation to ensure independence, transparency, and credibility of the results.
1) Planning
Development of a timeline and provisional budget and Terms of Reference (ToR) to define the scope of work (sample of population to be covered) and the focus of the evaluation.
2) Selecting an independant evaluator
ECLT commissions independent technical experts for all its final evaluations. For that matter, the ECLT follows a consultative selection process of local and international consultants. The ECLT will form a panel of local stakeholders to participate in the review of the submitted technical proposals and grade the candidates following a specific criterion. As a result of this consultative process, ECLT decides which consultant or team of consultants will conduct the evaluation.
3) Implementation of evaluation
The Evaluation team will proceed to the field to meet and interview beneficiaries, non-beneficiaries, teachers, parents (including mothers), communities (leaders, committees), district level officials and other relevant stakeholders. Through focus group discussions and semi-structured interviews, quantitative and qualitative data is gathered translating into preliminary findings, lessons learned and recommendations.
4) Report
The evaluation report focuses at a minimum on the six principles listed above. Once the report has been finalised, ECLT disseminate copies of the final evaluation report, Executive Summary and/or Infographics to partners and stakeholders.
So, how did we do? What we learned from Independent Evaluations.
Tanzania UMOJA: In terms of impact, the project was successful in increasing household incomes, providing new decent work opportunities for youth, and improving school attendance and performance levels. The qualitative section of the study also revealed a number of unexpected positive outcomes, such as gains in confidence and a sense of empowerment amongst women, a decrease in bullying amongst students, and a decline of alcohol and drug abuse among youth who were involved in skills training groups. Check out the findings here.
Uganda REALISE project: The evaluation demonstrated that the project was relevant since it was implemented in a tobacco-growing district where child labour was highly prevalent and addressed the factors that enabled child labour by using a holistic approach. The project built strong partnerships with communities and district officials who developed child protection systems. These systems will remain sustainable if its members remain actively involved in the Child Protection Committees and if those committees have transportation & communication means to continue addressing child labour independently. Learn more about the outcomes of this phase of the project.
Mozambique REACT project: Local communities worked closely with school administrations to effectively identify and withdraw child laborers. After-school programs and apprenticeships kept children away from tobacco work and provided them with economic alternatives to tobacco growing. However, increased more collaboration and transfer of knowledge between tobacco companies and implementing partners was referred as a gap. Read more in this infographic!
Malawi CLEAR project: The evaluation demonstrated that the project had contributed to the reduction of child labour in tobacco growing and in other types of work, such as household chores and other agricultural activities, in the three districts covered. CLEAR Project’s design was well-suited to the social, cultural, and political context of Malawi, adequately satisfied the needs of stakeholders and the beneficiary population and was properly targeted. Check out more findings from the independent evaluation of our project in Malawi.