Impact Data Highlights

How We Measure Impact

Zoe Empowers measures the results of the empowerment program for orphans and vulnerable children through gathering and analyzing data drawn directly from program participants across eight areas of intervention.

THE PROBLEM

There are millions of orphans and vulnerable children lacking the basic necessities of life, with no hope that tomorrow can be better. Zoe Empowers has identified eight major life areas which hold these young people in life-threatening poverty.

OUR RESULTS

Young participants move from the hopelessness of crushing poverty to meeting their needs across eight major life areas: food security, secure housing, health/hygiene, child-rights, education, income generation, community connections, and spiritual strength.

3 Years of Support is all it takes

*Numbers based on 2017 data

Kenya Data Highlights

“Through my work, I can provide sufficient clothing, food, school expenses and other necessities for my family.”

Reported by 99.5% of 3-year Kenya graduates

Kenya Self-Sufficiency Index

The Self-Sufficiency Index (SSI) measures the level of achievement each family has reached at the time of the survey across seven of the eight areas of intervention. Spiritual Strength is not included in SSI measure, due to the nature of that life area. The SSI is an amalgamation of pertinent results in each life area.

Rwanda Data Highlights

“In the last year, I assisted others in my community without asking for payment.”

Reported by 96% of 3-year Rwanda graduates

Rwanda Self-Sufficiency Index

The Self-Sufficiency Index (SSI) measures the level of achievement each family has reached at the time of the survey across seven of the eight areas of intervention. Spiritual Strength is not included in SSI measure, due to the nature of that life area. The SSI is an amalgamation of pertinent results in each life area.

Survey Overview

Purpose of the Impact Survey
To gather and analyze data to assess how the empowerment program affects outcomes in eight areas of intervention: food security and nutrition, income generation, safe housing, health and hygiene, child rights, community connections, education, and spiritual strength. This information can be used to improve the program and can also be shared with supporters to better communicate the significance of their investment in the lives of these young people. The goal of the survey design is to acquire the most accurate and relevant data possible while using financial resources judiciously.
Survey Methodology
The impact survey is conducted at three points in the three-year program: at intake before the youth receives any benefits, at the midpoint, and at graduation. Empowerment groups at each level are randomly selected in proportion to the total number of groups in each country, with a goal of 20-25% of the groups surveyed (a lower percentage for the largest programs) and a minimum of three groups chosen at each level in every country. One survey is completed by every child-led family in the group, usually by the youth that is the head of household. Initially, the survey is designed to be cross-sectional, with the intake groups’ surveys providing baseline data to compare to the data from the midpoint and at graduation. The selected intake and midpoint groups will continue to be surveyed as they progress through the program, eventually yielding longitudinal data for the program.
Survey Limitations
We make every reasonable effort to ensure that the impact surveys return accurate, useful data that provides a true representation of the Zoe Empowers program. Our constraints include minimizing the time needed to administer the survey, enter the data, and analyze the results as well as the cost to conduct the survey.  All of the known or potential limitations of this study  we are aware  of are listed below.
Survey Questions
The survey questions measure the quality of life related to our eight areas of intervention. Additional questions collect background data on each person that we survey and their empowerment group. Some of this background information is gathered by the program facilitator and returned with the survey data. Questions measure the impact of the program at the family/household level, individual level, and community level. Most questions are relevant for participants at all levels of the program. The format of the survey was chosen to allow for as many questions as possible to be included without it becoming too burdensome to administer and to minimize the time needed for data entry. The types of survey questions/responses are listed below:
• Number (ex: age)
• Yes/No
• Multiple Choice
• Select all that apply
• 1:4 scale
• 1:5 scale
Data Validation and Analysis
The Data for Good program at SAS Institute Inc. is partnering with Zoe Empowers on this project.  They assist with data preparation, cleansing, and visualization, as well as the self-sufficiency index calculation.
Notes on Implementation
The surveys are translated into local languages by Zoe Empowers in-country staff when necessary, and are administered by one or more program facilitators in a group setting. Groups in their first year are given the surveys in two steps, completing as many questions as possible when the participants are just entering the group and then conducting follow-up interviews within the first few months during home visits to verify the information and complete the surveys.

Survey Methodologies

Sample Size
  • For existing groups at the time the survey is launched, will start with their midpoint or graduate surveys:
    • Goal is to survey 20-25% of groups (a lower percentage would be acceptable for the larger programs, while still yielding reliable results)
    • At a minimum each program facilitator will conduct survey’s with at least one group in each year of the program.
    • In some case, especially in countries with smaller programs, the percentage surveyed will be much greater than the goal. In all cases we make sure there are at least three groups surveyed, and at least 70 individual surveys at each level.
  • For incoming groups:
    • Chose the number of groups surveyed to be the same as the number of program facilitators in each country.
    • Drawback – no way of knowing the total percentage surveyed until groups are all finalized. In countries with a very high number of kids/program facilitator, this will yield a smaller percentage covered by the survey.
Randomization of the Sample
  • Zoe Empowers chooses the empowerment groups by using an Excel randomizer from the total list of empowerment groups with the following subgroups: countries, program year, and by geographic region when possible. In many countries, there are enough surveys to look at data sorting by Male/Female because the groups are of mixed genders, but there is not always an even distribution of Male vs Female heads of households in every country.
  • The survey is given to all the heads of household [the youth acting as “parent” in a youth-headed household; usually the oldest or most capable sibling in the family unit] in the groups randomly selected. The survey questions cover the heads of household plus dependents.
  • Utilizing the random number generating function in Excel to select the groups by selecting from their row number in our Excel files (for existing groups) or their position on a list (for new groups that are forming.) Exceptions to this are noted below:
    • In the 2015 Rwanda groups there were two oversize groups, with 257 and 341 total members in each group. This is atypical of the program, and was not done in any other recent year. They were excluded from the survey.
    • Six groups started meeting in Liberia in August 2016 but did not start getting resources until January. These groups do not follow the standard three-year cycle, and could skew the results if included.
Administering the Survey
  • Zoe Empowers works hard to ensure these questions are understood across cultural differences by spending time with the test facilitators to ensure understanding.
  • We realize that incoming group members from newly formed groups are the most likely to answer questions based upon what answers they hope will yield them the best results or what answer may hide their shame. To mitigate this possibility the test facilitators:
    • After the initial survey is given in the group setting, test facilitators follow up during home visits to verify the data.
    • Reasons for extra care in assessing the validity of answers for the incoming groups:
      • No trust in program yet
      • Want to enter/stay in program, so may try to give answers they believe facilitators would want to hear (ex: age, number in household);
      • May be ashamed of their level of poverty;
      • Physically and mentally may be in poor condition, and not able to provide good answers.
      • May not know correct answer (ex: ages)
      • Understanding of questions may change over time (ex: nutrition, child rights)
    • Heads of households in the program will be surveyed at the midpoint (1.5 yrs.) and at the end of the third year. These surveys will be administered in a group setting, with the heads of households filling them out but led and checked by a test facilitator
Research Design Categories
  • Modified Cross Sectional Design (by Year)
    • Interview at multiple stages of program: incoming, midpoint, and 3rd
  • Longitudinal Design
    • In general, the best way to show progress when same people are surveyed at different points in time.
    • Zoe Empowers will maintain information on each household to link with other surveys.
Avoiding Bias
  • Zoe Empowers ensures the empowerment groups are randomly selected (using stratified sampling to make sure every country, every empowerment group year, and possibly urban/rural groups are represented).
  • Surveys include every household in the selected empowerment groups.

Survey Limitations

Survey Questions
  • The goal has been to keep the survey a reasonable length with regards to the number of questions, the time needed to complete it, and the data entry cost in terms of staff hours. To accomplish this, the survey questions are limited to those that can be answered by a number or a selection from a list of possible responses. There are no open-ended questions on the survey. This puts some restrictions on the level of detail the survey can provide.
  • Cultural differences and translation difficulties may cause some questions to be misunderstood, leading to inaccurate data, though every effort is made to mitigate this risk.
  • Some of the information we wish to gather may be very difficult for the group members to provide. [i]
  • After the Impact Survey was launched in January 2018, the survey data was analyzed to help refine the survey questions. Each new revision of the survey has attempted to correct challenges found while keeping as many of the survey questions as stable as possible. Data from groups that used earlier revisions of the survey has not been discarded since we will want to compare their original responses with the new surveys they complete as they continue in the program.

 

[i] One example of this is income data:  a question on an early revision of the survey asked for the household weekly income, with ranges listed for the responses. The answers we received, however, considerably undercounted the household’s actual income. The surveys were only counting the money received from businesses with regular cash flow. Seasonal income from agriculture or animal husbandry was not included. In addition, no credit was given for anything they produced that was kept for their own consumption. Questions were added to measure indirect indicators (ex: household expenditures and assets, standard of living) instead. [See Kumar, Krishna. “Indicators for Measuring Changes in Income, Food Availability and Consumption, and the Natural Resource Base.”  A.I.D. Program Design and Evaluation Methodology NO. 12. September 1989. (Document Order No. PN-AAX-223)]

Sampling
  • Due to the remote locations of the groups as well as other considerations, it is not feasible to randomly select individuals from the entire population of participants in a country to take the survey. Instead, whole empowerment groups are randomly[i] selected to participate. For the groups that are chosen, every family in the group is surveyed, with one survey completed per youth-led family.[ii]
  • The sample size is chosen based on the number of program facilitators and the number of groups in each country. We strive to survey an average of 20-25% of the groups, beginning with those that started in 2015. For our larger programs (10,000 and over enrolled annually) the percentage can be reduced to between 15% to 20% and while still maintaining reliable data.

 

[i] The random number generating function in Excel was used, with groups chosen based on their row number or position on a list.

[ii] A very small number of families leave their group before graduation, in which case they might not be surveyed. These cases include both positive reasons for leaving (early graduation, moving to a better location for their business) and negative ones (dropping out, expulsion due to not following rules), and are documented. In the 2015 Kenya groups that were surveyed, 96.5% of the families on the original name lists participated in the surveys at graduation.

Administering the Survey
  • The program facilitators and/or communications assistants conduct the surveys. No problems have been observed so far with this structure, and it has been very helpful in getting any potential issues with the questions or the data resolved. In the future, to try to eliminate any possible bias, we may suggest that facilitators conduct surveys which are not their own empowerment groups. Alternatively, an outside group could be hired to conduct the surveys or at least to conduct an audit of the surveys.
  • Surveys of incoming groups:
    • This is the group that is the most challenging to survey. They are just joining the program, so they may have little trust in the Zoe Empowers program or their program facilitator. They may be in poor physical and/or mental condition, and less able to provide accurate answers. They may not know some of the answers, such as their correct age. They want to enter/stay in the program, so they may try to give answers they think the staff would want to hear. Conversely, they may be ashamed of their level of poverty and attempt to inflate the level of their condition. Their understanding of some of the questions may change over time (ex: nutrition, child rights).
    • To mitigate the challenges listed above, surveys for incoming groups are given in a two-step process:
      • The heads of households take the survey as they prepare to join an empowerment group. This is done as soon as possible after beginning the program so that they remember clearly the details of their condition before Zoe Empowers. This survey is usually performed in a group setting.
      • When the program facilitators conduct the home visits (as the group rosters are firming up) they go over the survey with the head of the household and make sure everything on it is accurate as of the time they entered the group. The test facilitator seeing the physical conditions of the home can assist in validating the data.   Any sensitive questions that the program facilitator determine are better covered in a less public setting are completed at this point.
    • Midpoint (1 ½ years) and graduate (at end of 3rd year) surveys:
      • These surveys are administered by a survey facilitator in a group setting and completed by the heads of households. The facilitator strives to ensure proper understanding of the survey.
      • In general, the data from this group is expected to have a high level of accuracy, except for the possible issues with misunderstanding the survey questions outlined above.
Entering the Data
The data is entered into Excel by the communications staff person or the program facilitator and sent to the program reporting coordinator in the United States. Great care taken to avoid data entry errors.  The original surveys are saved and/or scanned.
Data Cleansing and Analysis
The program reporting coordinator checks the data, following up with the in-country staff with any questions. The data is prepared and sent to a volunteer team at SAS Institute Inc. for further cleansing, analysis and data visualizations.

Impact multiplied by empowerment

Zoe Empowers equips orphans and vulnerable children to overcome extreme poverty in only three years. Chat with our team about how you can partner to transform lives.