Monitoring And Evaluation For Youth Workforce Development Projects

These two types of intermediate outcomes are presumed to be the means to an ultimate goal, such as 100-percent youthemployment. For the members of this PLP, outcomes to accomplish this goal vary from one project to another, depending on the project objectives.Impacts are the longer-term effects, whether positive or negative, that can be attributed to the program—such as youthemployment or self-employment, increased income, sustainable livelihoods, or improved community well-being.The table on the next page presents the activities, outputs, outcomes, and impact indicators of all six participating organizations in the Youth and Workforce Development PLP. Other youth enterprise and workforce development programsmay find these indicators useful to consider when developing their own programs. The appendix provides additionaldefinitions and explanations of monitoring, evaluation, and impact assessment.One of the greatest benefits of the causal model is that it simply illustrates and clarifies what needs to be done andhow each activity and its related output(s) will contribute to intermediate results. Since all programs have slightly different program objectives and assumptions about the causal chain, it is difficult to define a “one-size-fits-all” causalmodel. Compiling all of the PLP partners’ causal models here demonstrates that there is no one approach for achieving100-percent youth employment. While some programs focus on activities and outputs, such as training or educatingyoung people, others focus on providing financing or capacity building to local institutions, or identifying and analyzingmarket opportunities for youth. All these activities and outputs are valuable and, if well done, contribute to the causalchain that leads to longer-term impacts of gainful employment or self-employment.2 SEEP NETWORK PRACTITIONER LEARNING PROGRAM

Overview of Causal Chain for Six Youth-Workforce Development ProgramsPLPPartnersActivitiesWhat activities willthe program offerto foster youthemployment andentrepreneurship?OutputsIndividual outcomesSector outcomesImpactWhat will theactivity deliver?What will theseoutputs do forbeneficiaries?What will theseoutputs andoutcomes do forinstitutions orthe sector?Why are wedoing this?100% youthemploymentEDC, SC,IRC, FP,PartnerMKFTraining: marketdriven vocationalskills and life skillstraining- No. of youth who completed the training- % graduation (no. ofgraduates/no. of youthinitially registered)% of graduates productively engaged (employment, part-time job,formal and informaleducation, etc.)Greater alignment betweentraining (type, level, quantity) and market opportunities (training adjusted toincrease placement rate)POA,PartnerMKFMentoring- No. of youth mentored(beneficiaries)- No. of mentorsNo. of youth productively engaged (employment, part-time job,formal and informaleducation, etc.)Relationships establishedbetween the private sectorand NGOs or training institutions (measured as % ofretention of mentors)IRC, EDCActive recruitmentof girls- % of increase in thenumber of applicationsfrom girls- % of girls’ application- Use of selection criterion and special provisionsfor girls enrollment- % of girl graduates- % of girls productivelyengaged (employment,part-time job, formaland informal education,etc.)- No. of vocational training centers that follow acode of conduct for girls’empowerment- No. of girl-friendly actionsor policies implement by- Sustainablegovernment and NGOslivelihoodsPartnerMKFMicroloans- No. of business loans- % of business loansissued to young peoplebetween 18–30- Number of jobscreated for youth bymicrofinance clients- % of sustainable youthbusinesses after oneyear- % of youth enterprisestransferred from unregistered to registered status- % of young clients whoreturn for new businessloansSCSavings- No. of youth saving accounts- No. of youth who arepart of saving clubs orhave accounts% of youth who successfully increasedfinancial securityIncreased amount ofcapital available for futureeconomic growthEDC,MC, IRC,PartnerMKFMarket research (labor market researchto identify employment opportunitiesand skills needsAND/OR marketdevelopmentresearch to identifyself-employmentopportunities)- No. of youth engagedin conducting marketresearch- No. of businesses interviewed- No. of NGOs interviewed- No. of government officials interviewed- No. of consumer groupsinterviewed% of youth transitioninginto employment or selfemploymentEducation and trainingdriven by markets demand,i.e., more aligned withneeds of employers andneeds for local economicgrowth (measured as % ofplacement rate)IRC, SCElimination ofunsafe or exploitiveemployment and/orgrey economy- Establishment of standards- No. of trainers trained- No. of youth affected bytraining- No. of self-reportingnon-exploitative sourcesof income to complement household income- No. of adolescentsparticipatingNo. of institutions adoptingsafe practices, standards orcode of conductSC, EDC,IRCBuilding government capacity foryouth services andpolicies- Adoption of nationalyouth policy document- Number of programsexpanded with government funds% of youth engaged ingovernment programsLocal institutions trainedor given incentives toimplement youth-friendlypractices- % of employed youthin the community- % of selfemployedyouth in thecommunity- Increasedincome- Peace- JusticePRACTITIONER LEARNING PROGRAM SEEP NETWORK 3

2. The Challenge of Measuring: Selected PLP ExperiencesThis section describes the experiences of three PLP partners in measuring causal-model outcomes and impacts. Eachexperience highlights some difficulty in measuring or comparing activities, outcomes, or impacts, and offers the solutiondeveloped for the individual youth-workforce development program. The partners also explain the specific lessons theylearned from this experience.Save the Children: Measuring the IntangibleDeveloping life skills and financial literacy are the core program components of Save the Children’s Rural Youth Livelihoods program in upper Egypt. The causal model in this case reflects a causal link between life skills and financialliteracy training, increased income and new knowledge, and the ultimate impact sought by all members of this PLP: 100percent youth employment. The table below displays the Rural Youth Livelihoods program causal model.Rural Youth Livelihoods Program Causal ModelGoalEquip adolescents in rural Egypt to successfully navigate the transition to workand maintain decent means of living which enhance their well-being.Strategic ObjectiveIncreased use and adoption of positive economic practices and opportunitiesIntermediate Result 1Intermediate Result 2Intermediate Result 3Intermediate Result 4Increased availabilityand accessibility ofadolescent-friendlyservices andopportunitiesImproved quality ofadolescent-friendlyservices andopportunitiesEnhanced capabilities,skills, and knowledgeof adolescentsStrengthened enablingenvironment Non-formal life skills;financial and businesseducation and training Behavior shift by policymakers, adults, community, and private sectoractors (communityawareness raising andadvocacy) Linking and promotingawareness and access tofinancial (formal /in-formal savings) andnon-financial services(mapping, awarenessraising, partnerships) Market-drivencurriculum anddevelopment ofmaterials that arestrengthened andadapted for practicaland relevant learning Creating access tobusiness practicum(market-driven modelsof private-sectorengagement) Capacity building fornews/existing serviceproviders4 SEEP NETWORK PRACTITIONER LEARNING PROGRAM Support for adolescents(counseling andmentoring) to help themidentify livelihoodpathways Practice-based employment skills (smallbusiness or internships /apprenticeship)

Measuring Changes in Assets and BehaviorsThe key question for Save the Children (SC) was to determine whether the chosen program activities and outputs led tothe desired outcomes: improved financial/business management behaviors and practices. Measuring the ultimate resultwas possible in this program through a proxy indicator because the program included a financial support componentthat recorded how many young people started and maintained an income-generating activity (i.e. youth employment)between the start and end of the program. While this is not a direct measure of “the transition to work and means ofliving” since existing IGAs may or may not be generating profits, it is a reasonable proxy indicator for the ultimate resultsought by the program. However, measuring changes in assets accessible to youth was much more complex, given thatsocial and human assets are largely intangible. Additionally, changes in behavior take time and may not be obvious evenmonths after the program ends.To address this challenge, Save the Children decided to adapt and pilot a new composite index, the Youth LivelihoodsDevelopment Index (YLDI). The index comprises three self-administered self-assessment surveys designed for youth11–24 years of age: the Developmental Asset Profile (developed by the Search Institute), the Tangible Asset Profile,and the Livelihoods Competencies Profile. YLDI provides quantitative scores, measuring the presence or absence anddegree of various developmental, livelihood, and tangible assets.YLDI results provide program managers with overall Developmental Asset Profiles (DAP) of youth. The instrumentgenerates an aggregate score of an individual’s human, social, financial and physical livelihood asset levels, as well asmore detailed profiles by relevant characteristics (e.g., gender, age, grade level, or work experience). These profiles allowproject managers to design activities that support and maintain current assets and target interventions to improve thoseassets that are found to be inadequate. YLDI is also sensitive enough to track changes in developmental assets over time,making it useful for program design and evaluation.The following table shares sample results and indicators from the YLDI:YLDI Outcome/ResultYLDI IndicatorImproved Youth Livelihood Development (YLDI)% of youth with increased overall YLDI scoresIncreased awareness of youth livelihood developmentapproaches and responsesNo. of staff from government and NGOs that participate in Savethe Children engagement sessionsImproved financial capital among youth% of youth with increased financial capital scores on youthlivelihoods development indexIncreased community and parent support to youth livelihooddevelopment% of youth with increased support scale scores from the DAPLessons Learned in Measuring Intangible Outcomes Although measuring certain outcomes, such as behavior and attitudinal change, is difficult, there are tools thatcan quantify these changes over time. Save the Children worked with the Search Institute to adapt an alreadyexisting tool. Project managers should look for existing tools before investing in a new custom-made tool. Measuring outcomes is a tricky endeavor, and context matters. Save the Children translated the tool, adaptedit to the rural Egyptian context, and tested it with a small sample before using the index for the program. Theinitial test found that youth got bored with the long questionnaire and fabricated answers. After streamliningthe tool, a second test of the YLDI revealed greater reliability and consistency. Measuring activities and outputs (number of trainings held, number of youth trained) is usually easy and wellunderstood by staff. However, tools to measure intangible outcomes can be difficult for staff to administer. Thisis especially true if they do not understand what the tool is measuring or why measuring outcomes is important.Save the Children found that it had to orient staff on why collecting the data was important and how the datawas used after they were collected. The training program also helped increase staff comfort with the languageand questions in the YLDI tool.PRACTITIONER LEARNING PROGRAM SEEP NETWORK 5

Mercy Corps and Partner Microcredit Foundation: Measuring Which Activities MatterMostPartner Microcredit Foundation (Partner MKF) began in Bosnia and Herzegovina in 1997 as an economic development program of Mercy Corps. The microfinance institution today provides financial services to more than 60,000clients, 23 percent of whom are young people under 30. Two years ago, Partner MKF began research and developmentefforts to better serve the youth market. This is particularly important because national youth unemployment in Bosniaand Herzegovina is estimated at 60 percent, yet 89 percent of youth surveyed in a market study had never attendedbusiness training or engaged in economic activities.Mercy Corps works with microfinance institutions (MFIs), such as Partner MKF, to support the integration of complementary services to youth loan clients. For example, many youth-workforce development programs combine two ormore activities, such as loans, financial education, training, mentoring, or apprenticeships, to achieve the desired impact.With additional support from the World Bank, Partner MKF agreed to create a new youth loan product (to launch in2010), including business training and mentoring, and conduct a comparative cost-benefit analysis to determine whichcombination gave the biggest “bang for the buck.”Using Impact Evaluation to Compare Outcomes from Different ActivitiesImpact evaluation specifically assesses those changes in program outcomes that can be attributed to a particular program activity or activities. The central question for impact evaluation is what would have happened to those participantsreceiving the intervention had they not received it.For example, Partner MKF knew that access to credit is an important contributor to youth employment and livelihoodsoutcomes. However, much less was known about the specific effect of different loan conditions, business training, andmentoring on young people’s success in business. The impact evaluation will help Partner MKF, Mercy Corps, and theWorld Bank determine which of the following variations of activities would produce the greatest contribution to impacting young people’s success in business:1.2.3.4.Young people who receive the youth loan product with standard maturity and loan amountYoung people who receive the youth loan product with an extended loan term, if desiredYoung people who receive an extended loan term and a larger loan size, if desiredYoung clients who receive the original youth loan, plus free business training and mentoringThe impact evaluation methodology involves randomized treatment groups (program participants) and control groups(non-participants). It attempts to replicate as closely as possible the scientific experiment model, which measures directcontribution of activities to results by controlling for non-program influences. Each treatment group receives differentprogram activities, and the control group does not receive any particular support aside from the loan. Partner MKF andMercy Corps track indicators including: Number of employees, including of youth employeesWho makes financial decisionsWhere business is conducted (in the home; office; street; etc.)Employment outside the microenterpriseNumber, type and monetary value of business assets and inventoryBusiness expenditures and type, past and presentTotal sales for last three months, and last yearSales for the past month and last yearBiggest challenges/problems to the businessFinancial literacy (questions about interest, credit history, investments, etc)33. Source: Baseline study used in program (internal document).6 SEEP NETWORK PRACTITIONER LEARNING PROGRAM

One year after the implementation in 2010, Partner MKF and the World Bank will conduct an impact evaluation todetermine whether and to what degree each of the added activities above contributed to the success of youth businesses.This is done by analyzing the “double difference,” found by subtracting the achievement of the control group from theachievement of the treatment group to determine the net change.For example, assume that both the treatment (participants) and control (non-participants) groups start out at the beginning of the project with the same level of income. At the end of the project, the youth who received a loan had a 50percent increase in income, whereas non-participants’ income increased by 25 percent over the same time period. Animpact evaluation considers the double difference—that is, the increase in the control group subtracted from the increase in the treatment group, which is the net change, or impact, of the loan on income. Thus, while the loan increasedparticipants’ income by 50 percent, the real impact of the project is a 25 percent increase in youth income (50 minus 25).Subtracting the change the control group experienced helps account for factors outside the program.In addition, if all young people, including the control group, see their income increase by 50 percent during the one-yearperiod, one could not conclude that either the youth loan or mentoring component of the program had a positive effecton youth income. If, however, results found (with statistical significance) that young people in the group that receivedmentoring support in addition to the loan had greater financial success at the end than those who only received a loan,one could conclude that mentoring adds value to young entrepreneurs. The cost of implementation of the program, whowill pay, and what solution would be most sustainable, must then be taken into consideration.Lessons Learned in Comparing Program Activities The scientific, experimental model is extremely difficult to replicate, particularly by organizations with limitedfinancial and technical means. However, programs can separate youth into groups to try to reasonably observethe differential impact of different program activities on overall results. With existing successful programs, suchas youth microfinance, it is often easy to multiply the impact of the program on youth by including additionalprogram elements. Organizations and donors are often unsure if the investment in impact evaluation is worth it—that is, if thefunds would not be better spent just providing the program to more youth. Doing this sort of comparison testing is worth the investment if the organization plans to scale up or replicate the project, and wants to make surethat the methods used produce the greatest possible impact. For example, in this case, Partner MKF may integrate the mentoring component to other, non-youth clients, if the study proves that mentored clients are moresuccessful. Mercy Corps could also scale up

The PLP learning products are written by and for practitioners in the field of youth workforce development.2 See these other learning products in the Youth and Workforce Development PLP: "Strategies for Scale-Up in Youth Workforce Development Programs" "Market Assessments in Youth Workforce and Livelihoods Programs"