BETTER CRITERIA FOR BETTER EVALUATION
BETTER CRITERIA FOR BETTER EVALUATIONMegan Kennedy-Chouane OECDPresentation to Belgian Development Co-operation PartnersJanuary 2021@OECD EVALNET #EvalCriteria oe.cd/criteria
Exercise1. Write down at the top of your paper a short description of an intervention (apolicy, project, programme, strategy, etc.) that you might be interested inevaluating. Be sure to specify the objectives (what it aims to achieve).2. As we go through each definition, try to write down one or two questions that youcould ask through the lens of that criterion
First set out by the OECD Development AssistanceCommittee (DAC) in 1991, defined in 2002, thecriteria encourage a focus on effectiveness andresults (looking beyond inputs and activities).What aretheevaluationcriteria?Definition: “A principle or standard by whichsomething may be judged or decided” - OxfordLiving DictionaryThough originally developed for use in the contextof development co-operation, now widely usedand referenced, including for national and southsouth co-operation. Demand-driven spread.3
Why do criteria matter?“If they can get you asking the wrongquestions, they don't have to worry aboutanswers.”- Thomas Pynchon
Intervention used to refer to the subject of theevaluation. Encompasses all the different types ofefforts: project, programme, policy, strategy, thematicarea, an institution, financing mechanism, etc.The criteria can be used to evaluate international cooperation activities, as well as private sector, nongovernment actors, and national or local governmentsin domestic policy contexts.SOME POINTSON LANGUAGE& SCOPEBeneficiaries has specific meaning here. Defined as,“the individuals, groups, or organisations, whethertargeted or not, that benefit directly or indirectly,from the development intervention." Other terms,such as rights holders or affected people, also used.Notes are part of the definition, further detail indocument: oe.cd/criteria
Each criteria is a lens,giving a differentperspective on theintervention – both theimplementationprocess & the results
together, they provide amore complete picture.
RELEVANCEIs the intervention doingthe right things?
RELEVANCEThe extent to which the intervention objectives and design respond to beneficiaries’,global, country, and partner/institution needs, policies, and priorities, and continue todo so if circumstances change.Note: “Respond to” means that the objectives and design of the intervention aresensitive to the economic, environmental, equity, social, political economy, andcapacity conditions in which it takes place. “Partner/institution” includes government(national, regional, local), civil society organisations, private entities and internationalbodies involved in funding, implementing and/or overseeing the intervention.Relevance assessment involves looking at differences and trade-offs betweendifferent priorities or needs. It requires analysing any changes in the context to assessthe extent to which the intervention can be (or has been) adapted to remain relevant.
COHERENCEHow well doesthe intervention fit?
COHERENCEThe compatibility of the intervention with other interventions in a country, sector orinstitution.Note: The extent to which other interventions (particularly policies) support orundermine the intervention, and vice versa.Internal coherence addresses the synergies and interlinkages between theintervention and other interventions carried out by the sameinstitution/government, as well as the consistency of the intervention with therelevant international norms and standards to which that institution/governmentadheres.External coherence considers the consistency of the intervention with other actors’interventions in the same context. This includes complementarity, harmonisationand co-ordination with others, and the extent to which the intervention is addingvalue while avoiding duplication of effort.
COHERENCEWhat are the priorities for applying Coherence in your context?Internal coherence: Trade, migration, security – policycoherence for development; human rights norms andstandardsExternal coherence: Between Members states / external in thesame context?
EFFECTIVENESSIs the intervention achievingits objectives?
EFFECTIVENESSThe extent to which the intervention achieved, or is expected to achieve, itsobjectives, and its results, including any differential results across groups.Note: Analysis of effectiveness involves taking account of the relative importance ofthe objectives or results.
EFFICIENCYHow well areresources used?
EFFICIENCYThe extent to which the intervention delivers, or is likely to deliver, results in aneconomic and timely way.Note: “Economic” is the conversion of inputs (funds, expertise, natural resources,time, etc.) into outputs, outcomes and impacts, in the most cost-effective waypossible, as compared to feasible alternatives in the context. “Timely” delivery iswithin the intended timeframe, or a timeframe reasonably adjusted to the demandsof the evolving context. This may include assessing operational efficiency (how wellthe intervention was managed).
IMPACTWhat differenceis the intervention making?
IMPACTThe extent to which the intervention has generated or is expected to generatesignificant positive or negative, intended or unintended, higher-level effects.Note: Impact addresses the ultimate significance and potentially transformativeeffects of the intervention. It seeks to identify social, environmental and economiceffects of the intervention that are longer term or broader in scope than thosealready captured under the effectiveness criterion. Beyond the immediate results,this criterion seeks to capture the indirect, secondary and potential consequences ofthe intervention. It does so by examining the holistic and enduring changes in systemsor norms, and potential effects on people’s well-being, human rights, gender equality,and the environment.
SUSTAINABILITYWill thebenefits last?
SUSTAINABILITYThe extent to which the net benefits of the intervention continue, or are likely tocontinue.Note: Includes an examination of the financial, economic, social, environmental, andinstitutional capacities of the systems needed to sustain net benefits over time.Involves analyses of resilience, risks and potential trade-offs. Depending on the timingof the evaluation, this may involve analysing the actual flow of net benefits orestimating the likelihood of net benefits continuing over the medium and long-term.
Key principles for usePRINCIPLE ONE: THINK FIRSTThe criteria should be appliedthoughtfully to support high quality,useful evaluation.They should be contextualized tothe individual evaluation, theintervention being evaluated, andthe stakeholders involved.PRINCIPLE TWO: NO STRAIGHT JACKETUse of the criteria depends on thepurpose of the evaluation.Covered according to the needs ofthe relevant stakeholders and thecontext of the evaluation. More orless time and resources may bedevoted to each criterion
Thoughts on next stepsCriteria support critical thinking, help us ask the rightquestions.Supporting better evaluation also requires: paying attention to quality focusing on use building capacity
Operationalizing the criteriaResponses to the criteria update may include:- Incorporating the principles (and lessons on risks for use) topolicies, manuals, trainings- Using the new definitions, and guidance- Institution-specific language- Operational instructions on scope/focus (coherence, impact)- Other changes to evaluation systems (Ultimate goal is to improve interventions: to improve relevance,coherence, effectiveness, efficiency, impact, sustainability. Othersystems and functions also support this goal:- What changes are needed beyond evaluation?
QuestionsAnswersDiscussion
Thank you!oe.cd/criteria@OECD EVALNET #EvalCriteria
Evaluation @ EnabelSharing experience on the non-mechanicalapplication of the DAC CriteriaGéraldine LADRIERE – Operation Advisor ENABEL
Evaluation of Interventions @ EnabelWhenWhoHowMid-Term Review (MTR) andEnd-Term Review (ETR) forevery projectFramework contract withexternal evaluators(consultants)Standard Terms of Reference,based on the DAC criteria
ApproachGeneric evaluationquestions to assess the project’sperformancebased on the five DACcriteriausing a ‘standardevaluation grid’Including at least two transversalthemes (gender andenvironment)two horizontal themes(result-oriented steeringand monitoring)Specific evaluationquestionsTo assess crucial and/orparticular aspects of a project
Challenges1A standard evaluation gridcan make the work‘mechanical’, without reallylooking at what is relevant2Overlap between genericobjective, specific objective,transversal and horizontalthemes4DAC criteria - ‘does itwork?’While forgetting about thewhy, for whom and how5Long and redundant reports3Too many objectives canreduce the depth of areview
Better health care and health services in Rwanda 2015-2020 – 18 M Large project, many outputsFocus on one DAC criterion for every result R1. Quality assurance system is set up, integrated and functional in all hospitals - effectiveness R2. Mental health services are accessible at community and national level - sustainability R3. Urban health coverage is rationalized and extended in line with National Policy - relevance R4. Leadership and governance is reinforced - relevance R6. Asset management system is designed and operational – effectivenessConclusion : choice of criteria according to the results of the Intervention, allow more focusedand relevant evaluation – more evidence on results information - avoid duplication with specificquestions.5
Support to Health Ministry in Niger Institutional support (focus on ‘soft’)Why and how did change happen?DAC criteria Realistic Evaluation1.DAC criteria: were the changes achieved relevant, effective, efficient and sustainable ?2.Evaluators then focussed on a explanatory approach : Why did it worked (or not), forwhom, under what circumstances, in what way, over what period and for how long ?ConclusionCombining DAC criteria and Realistic Evaluation increased the learning function and thesteering function of the evaluation6
Belgian development agencyHoogstraat 1471000 Brusselsenabel.be
Efficiency analysisThursday, Jan. 21, 2021
Setup of the efficiency analysis Black box in previous evaluations:how do we capture the efficiencyof our work beyond farm level?(farm-FO-enabl.env.)Research question:Which outcomes/return caninvestors or donors expect uponinvesting x amount of money in aRikolto intervention?Open to any research methodthat is– Comprehensive: systemicperspective– Replicable: low-cost and internallyCase selection criteriaCoverage of efficiencyconsiderations from afood systemperspectiveDiversity in focus andgeographic locationAvailability and quality/ reliability of dataStrategic relevance ofthe subject (futureprogrammes andfundraising)Timeframe:Oct-Jan
Efficiency analysis: methodology and cases Light-touch SROI process that passesthrough all steps of a SROI process butskips fairly quickly over some steps bymaking good use of available data fromM&E and MTRComplementary multi-criteria efficiencyanalysisIndonesia:sustainable andinclusive valuechainBelgium: shortchain initiativeBurkina Faso:franchise model forparboiled riceNicaragua:sustainablelandscapemanagementDR Congo: specialtycoffee for export
Initial meta-findings Sparked a lot of interest and activeinvolvement among colleagues (refreshing!)before, during and after the analysis lessresistance than with regular evaluationsVery interesting for our colleagues in terms ofmethodology (valuation of intangible, nonmonetary benefits)Not considered an “evaluation”, more anopportunity to learn and for fundraising Important to foresee additional guidance onhow to use the findings in practice Art and science–Greater insight into actual benefitsas perceived by stakeholders and beneficiaries–Insight into which interventions of a projectare more efficient–– Tweak investments towards interventions withgreatest returnDid we invest enough in intervention x?Brings finance and programme colleaguescloser during an evaluation–Fin. can better attribute costs and understandsto which kind of results these costs contribute– Capturing unintended benefits is not easy, youneed to train your ear to capture them whencapturing information and stories from thestakeholdersBalancing the attribution (deadweight) is tricky,can be easy to overattributeCombine quantitative SROI with qualitativemulti-criteria efficiency analysis to better graspthe dynamics of the project and nuance theanalysisChallenge to replicate without an expert:––Enthusiasm of colleagues to dig their heels intothe analysisAbility to take a distance, to be self-critical andopen for blind spots
DEVELOPMENT EVALUATION CRITERIA21 JANUARY 2021
WHO ARE WE? The Union des Villes et Communes de Wallonie (UVCW) is the Associationof the Cities and Municipalities of Wallonia Since 2001, we have been devising and implementing the BelgianMunicipal Cooperation (MIC) Programme (in partnership with theAssociation of the municipalities from the Brussels-Capital Region Brulocalis) involving municipalities from Wallonia, Brussels and 5 Africancountries
WHAT WILL WE BE TALKING ABOUT? How have we made a differentiated use of the evaluation criteria taking into account the various aspects of our intervention and ofour evaluations? We will address this issue in two stepsA.B.What are the key elements at the origin of ourdifferentiated use of the evaluation criteria?At which levels have evaluation criteria been used indifferentiated way?
A. WHY THIS PARTICULAR USE OF THE CAD CRITERIA?A. What are the reasons for our particular use of the CAD criteria?a.our contextb.our vision of developmentc.our expectations based on this visiond.our organizational structuree.our potential for monitoring and evaluation
OUR CONTEXTa.What is our context of action? That is, what is imperative to us !oUVCW is a small organisation (around 60 FTE) which is not initiallydedicated to development cooperationHOWEVERWe have been devising and implementing the Belgian MunicipalCooperation Programme for 20 years as a partner of the Belgian CooperationoOur main partners are municipalities, that is non-professional (andvoluntary) actors of development cooperation which are also the beneficiaries (in the case of the African municipalities)
OUR CONTEXTo Within this institutional cooperation, we act both in the interest of the donor the African and Belgian municipalities (the latter being also our members) Dual responsibility that has led us to restructure our relationships considering the “Belgo-African partnerships” as our direct partners switching from a “cooperation” approach to a “collaboration” approach with a“development objective”
OUR VISIONb.What is our vision of development ? That is, how does our contextinfluence our vision?o Our interventions mainly aim at strengthening the capacities of the partner local institutionsBY empowering them in lead the of their own development
OUR VISIONo What do we mean by strengthening the capacities of the partner local institutions ? global and clustered approach with a special attention to the strategic priorities identified in the development paths of therecipient local authorities internal coherence between partnerships and external coherence withsupralocal authorities rigour in action (avoiding to skip steps and unwanted opportunityeffects) 1 common and unique outcome, logical framework and ToC per country
OUR VISIONo What do we mean by empowering local institutions in lead the of their own development? the recipient organisations are the headquarters UVCW takes on the co-leading the technical support the overall coordination with the donor, etc.
OUR VISION Our Programme is organised on a collective work basis collective programming collective (shared) coordination collective decision-making collective strategic and operational planning collective and individual (in parallel) implementation of the activities collective monitoring collective (internal) evaluationIn other words, our philosophy is let us share all that can be shared
OUR EXPECTATIONSc. What are our expectations about the collective approach? That is, why is ourvision of cooperation (collaboration) beneficial?Because a collective coordination allows (in particular)o to share the resources and the potentialso to optimize the management of a group of municipalities active within the samecountryo to support the sustainability of the benefits and of the virtuous practicesobtained thanks to the subsidieso to make collective auto-evaluationso to support the exploitation of pilot actions for dissemination and scaling uppurposes
OUR ORGANIZATIONAL STRUCTUREd. How are we organized to implement the MIC Programme? That is, how doesour organizational structure reflect our vision?o Who? overall coordination - UVCW local coordination - duo of local Coordinators from the partnermunicipalities national coordination - a local Coordinator co-opted among the recipientmunicipalities in each partner country coordination by activity-type - a local Coordinator co-opted among therecipient municipalities in each partner country to conduct the activity-type(All the Coordinators in the recipient municipalities are municipal employees)
OUR ORGANIZATIONAL STRUCTUREo What ? collective designing/updating (around twice a year) of the pluriannualstrategy per country collective proposal from the recipient municipalities collectively approved by all city-to-city partnerships collective operational planning detailed formulation (distribution of the activities-types between thelocal Coordinators within the recipient municipalities)a. analysisb. structuringc. organisation of the implementation(on a specific form for that purpose) check of the overall coherence and time schedulebudgetingcollective monitoring (every two months)
OUR POTENTIAL FOR MONITORING AND EVALUATIONe.What is our potential for the monitoring and evaluation of the MICProgramme? That is, how does our vision of development cooperationinfluence our evaluation practices?In terms of M&E, the complexity of the institutional organisation/structuration also meanso a multitude of evaluation subjectso but also a multiplicity of resources for qualitative internal evaluations
B. LEVELS AT WHICH THE EVALUATION CRITERIA ARE USED IN A DIFFERENTIATED WAY That is what has led uso to resort to qualitative internal evaluations, in particular to « collective autoevaluations »oand to identify 3 levels at which the evaluation criteria are used in adifferentiated way differentiation in the way to interpret the evaluation criteria differentiation for equity purposes differentiation for formative purposes
B. LEVELS AT WHICH THE EVALUATION CRITERIA ARE USED IN A DIFFERENTIATED WAYa.To what extent does our vision lead us to interpret the evaluation criteria in adifferentiated way?o During our collective auto-evaluations we had to wonder about theinterpretation of the evaluation criteriao In the context of an external evaluation the consultants have to agree onthe interpretation of the evaluation criteria to work in a coherent wayBUTthe same would be unrealistic in a context of auto-evaluation because theparticipants, who are also the evaluators, are numerous judge and jury not familiar with the evaluation practices
B. LEVELS AT WHICH THE EVALUATION CRITERIA ARE USED IN A DIFFERENTIATED WAY 2 optionso seek a consensusBUT how to reach it? what would be the « variance » from the « average » interpretationsand how to relate that?o accept the coexistence of several interpretations of the criteria, with adouble advantage no more difficulties
BETTER CRITERIA FOR BETTER EVALUATION Megan Kennedy-Chouane OECD Presentation to Belgian Development Co-operation Partners January 2021 . Supporting better evaluation also requires: paying attention to quality focusing on use building capacity . Operationalizing the criteria
Bruksanvisning för bilstereo . Bruksanvisning for bilstereo . Instrukcja obsługi samochodowego odtwarzacza stereo . Operating Instructions for Car Stereo . 610-104 . SV . Bruksanvisning i original
10 tips och tricks för att lyckas med ert sap-projekt 20 SAPSANYTT 2/2015 De flesta projektledare känner säkert till Cobb’s paradox. Martin Cobb verkade som CIO för sekretariatet för Treasury Board of Canada 1995 då han ställde frågan
service i Norge och Finland drivs inom ramen för ett enskilt företag (NRK. 1 och Yleisradio), fin ns det i Sverige tre: Ett för tv (Sveriges Television , SVT ), ett för radio (Sveriges Radio , SR ) och ett för utbildnings program (Sveriges Utbildningsradio, UR, vilket till följd av sin begränsade storlek inte återfinns bland de 25 största
Hotell För hotell anges de tre klasserna A/B, C och D. Det betyder att den "normala" standarden C är acceptabel men att motiven för en högre standard är starka. Ljudklass C motsvarar de tidigare normkraven för hotell, ljudklass A/B motsvarar kraven för moderna hotell med hög standard och ljudklass D kan användas vid
LÄS NOGGRANT FÖLJANDE VILLKOR FÖR APPLE DEVELOPER PROGRAM LICENCE . Apple Developer Program License Agreement Syfte Du vill använda Apple-mjukvara (enligt definitionen nedan) för att utveckla en eller flera Applikationer (enligt definitionen nedan) för Apple-märkta produkter. . Applikationer som utvecklas för iOS-produkter, Apple .
och krav. Maskinerna skriver ut upp till fyra tum breda etiketter med direkt termoteknik och termotransferteknik och är lämpliga för en lång rad användningsområden på vertikala marknader. TD-seriens professionella etikettskrivare för . skrivbordet. Brothers nya avancerade 4-tums etikettskrivare för skrivbordet är effektiva och enkla att
Den kanadensiska språkvetaren Jim Cummins har visat i sin forskning från år 1979 att det kan ta 1 till 3 år för att lära sig ett vardagsspråk och mellan 5 till 7 år för att behärska ett akademiskt språk.4 Han införde två begrepp för att beskriva elevernas språkliga kompetens: BI
**Godkänd av MAN för upp till 120 000 km och Mercedes Benz, Volvo och Renault för upp till 100 000 km i enlighet med deras specifikationer. Faktiskt oljebyte beror på motortyp, körförhållanden, servicehistorik, OBD och bränslekvalitet. Se alltid tillverkarens instruktionsbok. Art.Nr. 159CAC Art.Nr. 159CAA Art.Nr. 159CAB Art.Nr. 217B1B
