Impact Data Highlights
View our 2018 impact data from the Kenya empowerment program
How We Measure Impact
ZOE measures the results of the empowerment program for orphans and vulnerable children through gathering and analyzing data drawn directly from program participants across ZOE’s eight areas of intervention.
There are millions of orphans and vulnerable children lacking the basic necessities of life, with no hope that tomorrow can be better. ZOE has identified eight major life areas which hold these young people in life-threatening poverty.
Young participants move from the hopelessness of crushing poverty to meeting their needs across eight major life areas: food security, secure housing, health/hygiene, child-rights, education, income generation, community connections, and spiritual strength.
3 Years of Support is all it takes
*Numbers based on 2017 data
“Through my work, I can provide sufficient clothing, food, school expenses and other necessities for my family.”
Reported by 99.5% of 3-year ZOE Kenya graduates
Kenya Self-Sufficiency Index
The Self-Sufficiency Index (SSI) measures the level of achievement each family has reached at the time of the survey across seven of the eight areas of ZOE’s intervention. Spiritual Strength is not included in ZOE’s SSI measure, due to the nature of that life area. The SSI is an amalgamation of pertinent results in each life area.
2018 Kenya Impact Data Highlights
Download the 2018 impact data from the Kenya empowerment program
Purpose of the Impact Survey
• Number (ex: age)
• Multiple Choice
• Select all that apply
• 1:4 scale
• 1:5 scale
Data Validation and Analysis
Notes on Implementation
- For existing groups at the time the survey is launched, will start with their midpoint or graduate surveys:
- Goal is to survey 20-25% of groups (a lower percentage would be acceptable for the larger ZOE programs, while still yielding reliable results)
- At a minimum each program facilitator will conduct survey’s with at least one group in each year of the program.
- In some case, especially in countries with smaller programs, the percentage surveyed will be much greater than the goal. In all cases ZOE makes sure there are at least three groups surveyed, and at least 70 individual surveys at each level.
- For incoming groups:
- Chose the number of groups surveyed to be the same as the number of program facilitators in each country.
- Drawback – no way of knowing the total percentage surveyed until groups are all finalized. In countries with a very high number of kids/program facilitator, this will yield a smaller percentage covered by the survey.
Randomization of the Sample
- ZOE chooses the empowerment groups by using an Excel randomizer from the total list of empowerment groups with the following subgroups: countries, program year, and by geographic region when possible. In many countries, ZOE should have enough surveys to look at data sorting by Male/Female because the groups are of mixed genders, but there is not always an even distribution of Male vs Female heads of households in every country.
- The survey is given to all the heads of household [the youth acting as “parent” in a youth-headed household; usually the oldest or most capable sibling in the family unit] in the groups randomly selected. The survey questions cover the heads of household plus dependents.
- Utilizing the random number generating function in Excel to select the groups by selecting from their row number in our Excel files (for existing groups) or their position on a list (for new groups that are forming.) Exceptions to this are noted below:
- In the 2015 Rwanda groups there were two oversize groups, with 257 and 341 total members in each group. This is atypical of the program, and was not done in any other recent year. They were excluded from the survey.
- Six groups started meeting in Liberia in August 2016 but did not start getting resources until January. These groups do not follow the standard three-year cycle, and could skew the results if included.
Administering the Survey
- ZOE works hard to ensure these questions are understood across cultural differences by spending time with the test facilitators to ensure understanding.
- ZOE realizes that incoming group members from newly formed groups are the most likely to answer questions based upon what answers they hope will yield them the best results or what answer may hide their shame. To mitigate this possibility ZOE test facilitators:
- After the initial survey is given in the group setting, ZOE test facilitators follow up during home visits to verify the data.
- Reasons for extra care in assessing the validity of answers for the incoming groups:
- No trust in program yet
- Want to enter/stay in program, so may try to give answers they believe ZOE would want to hear (ex: age, number in household);
- May be ashamed of their level of poverty;
- Physically and mentally may be in poor condition, and not able to provide good answers.
- May not know correct answer (ex: ages)
- Understanding of questions may change over time (ex: nutrition, child rights)
- Heads of households in the program will be surveyed at the midpoint (1.5 yrs.) and at the end of the third year. These surveys will be administered in a group setting, with the heads of households filling them out but led and checked by a test facilitator
Research Design Categories
- Modified Cross Sectional Design (by Year)
- Interview at multiple stages of program: incoming, midpoint, and 3rd
- Longitudinal Design
- In general, the best way to show progress when same people are surveyed at different points in time.
- ZOE will maintain information on each household to link with other surveys.
- ZOE ensures the empowerment groups are randomly selected (using stratified sampling to make sure every country, every empowerment group year, and possibly urban/rural groups are represented).
- ZOE surveys every household in the selected empowerment groups.
- The goal has been to keep the survey a reasonable length with regards to the number of questions, the time needed to complete it, and the data entry cost in terms of staff hours. To accomplish this, the survey questions are limited to those that can be answered by a number or a selection from a list of possible responses. There are no open-ended questions on the survey. This puts some restrictions on the level of detail the survey can provide.
- Cultural differences and translation difficulties may cause some questions to be misunderstood, leading to inaccurate data, though every effort is made to mitigate this risk.
- Some of the information we wish to gather may be very difficult for the ZOE group members to provide. [i]
- After the Impact Survey was launched in January 2018, the survey data was analyzed to help refine the survey questions. Each new revision of the survey has attempted to correct challenges found while keeping as many of the survey questions as stable as possible. Data from groups that used earlier revisions of the survey has not been discarded since we will want to compare their original responses with the new surveys they complete as they continue in the program.
[i] One example of this is income data: a question on an early revision of the survey asked for the household weekly income, with ranges listed for the responses. The answers we received, however, considerably undercounted the household’s actual income. The surveys were only counting the money received from businesses with regular cash flow. Seasonal income from agriculture or animal husbandry was not included. In addition, no credit was given for anything they produced that was kept for their own consumption. Questions were added to measure indirect indicators (ex: household expenditures and assets, standard of living) instead. [See Kumar, Krishna. “Indicators for Measuring Changes in Income, Food Availability and Consumption, and the Natural Resource Base.” A.I.D. Program Design and Evaluation Methodology NO. 12. September 1989. (Document Order No. PN-AAX-223)]
- Due to the remote locations of the groups as well as other considerations, it is not feasible to randomly select individuals from the entire population of participants in a country to take the survey. Instead, whole empowerment groups are randomly[i] selected to participate. For the groups that are chosen, every family in the group is surveyed, with one survey completed per youth-led family.[ii]
- The sample size is chosen based on the number of program facilitators and the number of groups in each country. We strive to survey an average of 20-25% of the groups, beginning with those that started in 2015. For ZOE’s larger programs (10,000 and over enrolled annually) the percentage can be reduced to between 15% to 20% and while still maintaining reliable data.
[i] The random number generating function in Excel was used, with groups chosen based on their row number or position on a list.
[ii] A very small number of families leave their group before graduation, in which case they might not be surveyed. These cases include both positive reasons for leaving (early graduation, moving to a better location for their business) and negative ones (dropping out, expulsion due to not following rules), and are documented. In the 2015 Kenya groups that were surveyed, 96.5% of the families on the original name lists participated in the surveys at graduation.
Administering the Survey
- The program facilitators and/or communications assistants conduct the surveys. No problems have been observed so far with this structure, and it has been very helpful in getting any potential issues with the questions or the data resolved. In the future, to try to eliminate any possible bias, we may suggest that facilitators conduct surveys which are not their own empowerment groups. Alternatively, an outside group could be hired to conduct the surveys or at least to conduct an audit of the surveys.
- Surveys of incoming groups:
- This is the group that is the most challenging to survey. They are just joining the program, so they may have little trust in the ZOE program or their program facilitator. They may be in poor physical and/or mental condition, and less able to provide accurate answers. They may not know some of the answers, such as their correct age. They want to enter/stay in the program, so they may try to give answers they think ZOE staff would want to hear. Conversely, they may be ashamed of their level of poverty and attempt to inflate the level of their condition. Their understanding of some of the questions may change over time (ex: nutrition, child rights).
- To mitigate the challenges listed above, surveys for incoming groups are given in a two-step process:
- The heads of households take the survey as they prepare to join a ZOE empowerment group. This is done as soon as possible after beginning the program so that they remember clearly the details of their condition before ZOE. This survey is usually performed in a group setting.
- When the program facilitators conduct the home visits (as the group rosters are firming up) they go over the survey with the head of the household and make sure everything on it is accurate as of the time they entered the group. The test facilitator seeing the physical conditions of the home can assist in validating the data. Any sensitive questions that the program facilitator determine are better covered in a less public setting are completed at this point.
- Midpoint (1 ½ years) and graduate (at end of 3rd year) surveys:
- These surveys are administered by a survey facilitator in a group setting and completed by the heads of households. The facilitator strives to ensure proper understanding of the survey.
- In general, the data from this group is expected to have a high level of accuracy, except for the possible issues with misunderstanding the survey questions outlined above.