EVALUATION REPORT - UNDP
United Nations Development Programme andGovernment of IndonesiaMaking Aceh Safer through Disaster RiskReduction in Development (DRR-A)EVALUATION REPORTPrepared by Akbar Meirio
United Nations Development Programme andGovernment of IndonesiaMaking Aceh Safer through Disaster RiskReduction in Development (DRR-A)EVALUATION REPORTPrepared by Akbar Meirio1
Table of ContentsList of Acronyms and Abbrevia ons . 3Execu ve Summary . 61. Introduc on. 82. Brief descrip on of the interven on . 83. Evalua on Scope and Objec ves . 104. Evalua on Approach and Methodology . 115. Data analysis . 136. Findings . 13Effec veness . 13Efficiency . 31Relevance . 34Appropriateness . 35Sustainability . 37Impact . 427. Lessons learned . 438. Recommenda ons. 44Annex 1. The list of the documents consulted . 47Annex 2. List of informants and par cipants . 48Annex 3. Evalua on Matrix . 532
List of Acronyms and Abbrevia onsAPBAAnggaran Pendapatan dan Belanja Aceh (Aceh Provincial GovernmentBudget)APBDAnggaran Pendapatan dan Belanja Daerah (District Government Budget)BPPDBadan Pembinaan Pendidikan Dayah/Badan Dayah (Agency for theDevelopment of Dayah Educa on/Dayah Agency)BAPPEDABadan Perencanaan Pembangunan Daerah (Regional DevelopmentPlanning Agency)BAPPENASBadan Perencanaan Pembangunan Nasional (Na onal DevelopmentPlanning Agency)BNPBBadan Nasional Penganggulangan Bencana (Na onal DisasterManagement Agency)BPBDBadan Penganggulangan Bencana Daerah (District/Municipality DisasterManagement Agency)BPBABadan Penganggulangan Bencana Aceh (Aceh Provincial DisasterManagement Agency)CAPCommunity Ac on PlanCBDRRCommunity-based Disaster Risk Reduc onCPRUCrisis Preven on and Recovery UnitDIBAData dan Informasi Bencana Aceh (Aceh Disaster Data and Informa on)DIBIData dan Informasi Bencana Indonesia (Indonesia Disaster Data andInforma on)DIKTIDirektorat Pendidikan Tinggi (Directorate of Higher Educa on of theMinistry of Educa on)Dinas PUDinas Pekerjaan Umum (Public Works Department)DISDIKDinas Pendidikan (Regional Educa on Department)DISHUBKOMINTEL Dinas Perhubungan, Komunikasi, Informasi dan Telekomunikasi(Transporta on, Communica on, Informa on and Telecommunica onsDepartment)DIPADa ar Isian Pelaksanaan Anggaran (Budget Programme Implementa onForm)DPRADewan Perwakilan Rakyat Aceh (Aceh Parliament)DRMISDisaster Risk Management Informa on SystemDRRDisaster Risk Reduc onDRR-AAcronym employed to refer to the Making Aceh Safer through DisasterRisk Reduc on in Development projectESRIName of a geographic informa on system so ware companyFGDsFocus Group DiscussionsFJABForum Jurnalis Aceh Peduli Bencana (Aceh Journalist Forum for DisasterAwareness)3
FokusbariForum Komunitas Siaga Bencana Arul Item (Arul Item Village Forum forDisaster Preparedness)FORMASIBABForum Masyarakat Siaga Bencana Aceh Barat (Aceh Barat Forum ofCommuni es for Disaster Preparednes)FSBGForum Siaga Bencana Gampong (Village Forum for DisasterPreparedness)GoIGovernment of IndonesiaICBRRIntegrated Community-based Risk Reduc onIPARInternal Project Assurance ReportKIIsKey Informant InterviewsKMPBKelompok Masyarakat Penanggulangan Bencana (Community-basedDisaster Management Forum)LoALe er of AgreementM&EMonitoring and Evalua onMDFMul -Donor FundMoHAMinistry of Home AffairsMusrenbangMusyawarah Rencana Pembangunan (Development PlanningDiscussion)NGONon-Governmental Organiza onNPCNa onal Project CoordinatorPACCPublic Awareness Coordina on Commi eePMEUPlanning, Monitoring and Evalua on UnitPNPMProgram Nasional Pemberdayaan Masyarakat (Na onal Program onCommunity Empowerment)PPDProvincial Project DirectorQanunRegional Regula on in AcehRAD-PRBRencana Aksi Daerah – Pengurangan Risiko Bencana (LocalAc on Planfor Disaster Risk Reduc on)RAPIRadio Antar Penduduk Indonesia (Indonesia Inter-Popula on Radio)RenstraRencana Strategis (Strategic Plan)RenjaRencana Kerja (Work Plan)RKPDRencana Kerja Pemerintah Daerah (Local Government Work Plan)RKPGRencana Kerja Pembangunan Gampong (Village Development WorkPlan)RPJMRencana Pembangunan Jangka Menengah (Medium-Term DevelopmentPlan)RPJMGRencana Pembangunan Jangka Menengah Gampong (Village MediumTerm Development Plan)SARSearch and Rescue4
SC-DRRSafer Communi es through Disaster Risk Reduc on in DevelopmentSDElementary SchoolSekdaRegional SecretarySKPASatuan Kerja Perangkat Aceh (Aceh government’s local department oragencies)SOPStandard Opera ng ProcedureTDMRCTsunami Disaster Mi ga on Research CentreToRTerms of ReferenceToTTraining of TrainersUNUnited Na onsUNDPUnited Na ons Development ProgrammeUnsyiahUniversitas Syiah Kuala (Syiah Kuala University)WalhiWahana Lingkungan Hidup Indonesia (an Indonesian NGO whoseconcern is that of environmental protec on)5
Execu ve Summary1. This report presents the findings of the final evalua on of the Making Aceh Saferthrough Disaster Risk Reduc on in Development (DRR-A) Project. The findings aredrawn from analyses of documents relevant to the project, direct observa ons ofproject ac vi es and results, and reviews of the proceedings of interviews as wellas focus group discussions (FGDs) with project stakeholders and beneficiaries. Theyare, for the purpose of methodical presenta on, organized into the six categories ofeffec veness, efficiency, relevance, appropriateness, impact and sustainability.2. Effec venessDRR-A has made significant contribu ons towards the formula on and establishmentof regulatory instruments and ins tu onal arrangements for implemen ng disasterrisk reduc on in Aceh. A remaining problem is the s ll low capacity of BPBA to leaddisaster management in Aceh. The project has also implemented Community BasedDisaster Risk Reduc on (CBDRR) program in 10 villages located in 10 districts. The DRRac vity plan proposed through CBDRR, however, appears to be weakly incorporatedinto government planning and budgets.3. DRR-A has also improved to a significant degree the capacity of the Tsunami DisasterMi ga on Research Centre (TDMRC) to provide science-based products and services.The project has also contributed towards the establishment of a master’s degreeprogram in Disaster Management at the Universitas Syiah Kuala or Syiah KualaUniversity (Unsyiah.) Yet, DRR-A appears to have been less effec ve in improving thefinancial capacity of the TDMRC to generate revenue. Public awareness of disasterrisk reduc on was raised by DRR-A with campaigns using a wide range media, andthe integra on of disaster risk reduc on knowledge into elementary, junior and highschools as well as the Modern Dayah curriculum. The achievement of the integra onof DRR into the elementary school curriculum appears to be more advanced than injunior and high schools. The program in the Modern Dayah is very good; but therewas insufficient me to ensure that the integra on of DRR into the curriculum is fullyachieved.4. EfficiencyThe organiza onal structure of DRR-A was efficient enough to support the project’simplementa on, but it failed to ac vely involve the other two important na onalgovernment agencies: BAPPENAS and BNBP. Delays in the implementa on of manyDRR-A ac vi es compromised the project’s ability to produce be er quality outputs.These delays were principally caused by UNDP delays in transferring the funds to theproject and by the Government in channeling of DIPA funds. The evaluator also found6
that the highest spending by DRR-A on strengthening TDMRC was the least efficientspending to achieve the intended outputs of DRR-A.5. RelevanceThe design of the DRR-A project is consistent with efforts to support the implementa onof na onal policies and priori es and respond to the urgent need for the Aceh ProvincialGovernment to establish be er arrangements and environments to put disastermanagement into opera on in Aceh. The project’s relevance was also enhanced bythe flexibility of its design, allowing it to respond to changing priori es and needs.However, it should be noted that DRR-A was not effec ve in maximising synergy ofoutputs to improve results.6. AppropriatenessThe design of DRR-A was suitable for the implementa on of the project’s ac vi eswithin Aceh’s structural and cultural contexts. The acceptance of DRR-A by a widerange of stakeholders and actors, including community leaders and members, is strongevidence of the appropriateness of its design and the nature of its interven ons.However, DRR-A did not contain an appropriate exit strategy to end the projectsmoothly and help achieve more sustainable output.7. SustainabilityIt is highly likely that the regulatory framework and ins tu ons for disaster managementestablished by DRR-A will be sustained by the Government of Aceh. There are alsoindica ons that the community-based disaster risk reduc on (CBDRR) programmeestablished with the support of the project will con nue to be implemented by theVillage Disaster Risk Reduc on Forum which was also established with the supportof the project. However, at the local government level, the real commitment of thegovernment to con nue or replicate CBDRR Program appears to be minimal.8. Although it is most likely that the Tsunami Disaster Mi ga on Research Centre (TDMRC)will con nue to operate, it is not as likely that the Centre will be able to provide sciencebased informa on, services and knowledge to the extent and quality that it did whenits opera ons were supported by DRR-A. Limited funding and the Center’s lack of abilityto raise revenue may stand in the way of its capacity to sustain the quan ty and qualityof its outputs. With regard to the sustainability of public awareness efforts ini atedby DRR-A, it is hoped that Dishubkomintel’s plans to include disaster risk reduc onmessages in some of its regular programs will ensure the con nuity of campaign-typeac vi es. There is a strong possibility that the plan to integrate disaster risk reduc oninto school curricula will be put into effect. To this end, Disdik has proposed to the7
Government that programs be designed and budgets allocated to con nue this endeavorthrough APBA. The Government of Aceh’s formal commitment to finance this programthrough APBA has, however, not yet been obtained. Efforts to integrate disaster riskreduc on into Modern Dayah’s curriculum are facing an obstacle of a different nature,Badan Dayah is finding it difficult to con nue this program due to the absence of a legalbasis to support its proposal to implement the program through APBA.9. ImpactThe baseline and end-line surveys of the implementa on of DRR-A indicate that theproject has brought about posi ve impacts on the state of disaster preparedness ofthe people in Banda Aceh and the ten districts where the project was implemented.Another posi ve impact of the DRR-A project is the increased awareness of people andcommuni es of the need to ini ate efforts to mi gate some poten al hazards in Aceh.10. In addi on to presen ng the findings of the evalua on exercise, this report also offerslessons learned and recommenda ons to various stakeholders. It is hoped that therecommenda ons may be useful to those contempla ng follow-on ac vi es to sustainthe many benefits that the project successfully produced.1. Introduc on11. The “Making Aceh Safer through Disaster Risk Reduc on” (DRR-A) Project wasimplemented by UNDP Indonesia and the Ministry of Home Affairs of the Governmentof Indonesia from January 2009 to May 2012. Towards its comple on, an independentevalua on was commissioned to systema cally evaluate the project, learn from theexperience of the project’s implementa on, and to provide inputs for future DRR-typeprojects in both transi onal and development se ngs.12. The evalua on was conducted from 8th of May 2012 un l 7th of July 2012. It was carriedout through document reviews and field assessments. The results of the evalua onare presented in this report which consists of sec ons which recount the project’sinterven on; outline the scope and objec ves of the evalua on; describe the approachand methodology employed by the evalua on; explain the data analysis process, andoffer the evalua on’s findings, lessons learned, and recommenda ons.2. Brief descrip on of the interven on13. The “Making Aceh Safer through Disaster Risk Reduc on in Development” (DRR-A)Project was designed to make disaster risk reduc on a normal part of the developmentprocess established in core func ons of Aceh’s local government and their public and8
private partners, especially in Aceh’s local communi es where the most effec veand direct ac ons can be taken to reduce physical, economic and social vulnerabilityto disasters. The project supported the provincial government to reduce the risk ofdisasters through four substan ve key outputs:Output 1 – Ins tu onal arrangement and enabling environment established to facilitatea par cipatory and concerted implementa on of DRR measures;Output 2 – Demonstra on of gender sensi ve projects in selected loca ons to test andimprove measures for reducing risk from natural disasters;Output 3 – TDMRC-UNSYIAH strengthened to provide science-based informa on,services and knowledge assistance to the local government and other DRR proponentsin implemen ng their DRR ac vi es;Output 4 – DRR public awareness programmes implemented to promote a gendersensi ve “Culture of Safety” among the people and ins tu ons of Aceh.14. This project was financed by the Mul Donor Trust Fund (MDF). Its implementa onwas based on five strategic pillars:1) Gender mainstreaming: ensuring the different needs and interests of men andwomen are accommodated equally, with an emphasis on women’s empowerment,since women tend to be more vulnerable to disasters.2) Accommoda ve framework: the project serves as an accommoda ve frameworkwith windows for dis nct yet interconnected ini a ves. It is also accommoda vewith regard to the types of measures to be taken, adop ng the priori es set by theinterna onally accepted Hyogo Framework of Ac ons.3) Mutual reinforcement among project components: the project is implemented bypursuing the outputs and project component targets in a simultaneous manner.4) Building on exis ng ini a ves launched under other programmes, the componentsof the Disaster Risk Reduc on (DRR-A) project are designed to capitalize on therelevant ini a ves undertaken by the different stakeholders.5) Learning lessons from experience: since many ac vi es in this project entailbreaking new ground, learning from experiences through workshops, seminars, etc.is applied for improving performance as the project advances.15. The project was implemented by Ministry of Home Affairs as na onal implemen ngpartner (IP) who delegated the authority to the Aceh Government to implement theproject. At the provincial level the responsible par es for project opera ons included theAceh Disaster Management Agency (Badan Penanggulangan Bencana Aceh/BPBA), theTsunami Disaster Mi ga on Research Centre (TDMRC) at the University of Syiah Kuala,the Aceh Provincial Development Planning Agency (Badan Perencana PembangunanDaerah Aceh/BAPPEDA), the Organiza on Bureau (Biro Organisasi ), the Educa on9
Department (Dinas Pendidikan/DISDIK), the Islamic Boarding School Educa on Agency(Badan Pembinaan Pendidikan Dayah/BPPD) and Transporta on, Communica on,Informa on and the Telecommunica on Department (Dinas Perhubungan, Komunikasi,Informasi dan Telekomunikasi/DISHUBKOMINTEL) and 3 selected local NGOs.16. DRR-A has produced key outputs in disaster risk reduc on by suppor ng strategicdisaster management regulatory frameworks and planning, undertaking gendersensi ve Community-based Disaster Risk Reduc on (CBDRR) in ten villages (East Coast,West Coast, and Central Highlands) of ten districts in Aceh, enhancing the capacity ofTDMRC, and fostering a culture of safety in Aceh through public awareness and theeduca on system.3. Evalua on Scope and Objec ves17. The evalua on assessed the implementa on of the DRR-A project from its start in 2009to its end in 2012. The exercise took into account ma ers related to the implementa onof the project at provincial, district, and community levels, in rela on to all fourprogramma c outputs. The considered target groups included the beneficiaries ofthe project, namely: 1) Provincial Agencies and organiza ons (BPBA, Biro Organisasi,DISHUBKOMINTEL, DISDIK, BPPD, DRR Forum, PACC, etc); 2) District Agencies (BPBD,DISDIK, BAPPEDA, etc); 3) TDMRC; 4) Selected local NGOs (Bytra, IBU Founda on, andKarst Aceh); and 5) community beneficiaries (Village DRR Fora), and schools.18. In accordance with the Terms of Reference for the evalua on, and taking into accountUNDP’s evalua on guidelines, the evalua on assessed the project implementa on inAceh in terms of its effec veness, efficiency, appropriateness, relevance, impact andsustainability. The specific objec ves of the evalua on are:1. To assess the achievement of stated project outcomes and outputs, taking intoaccount the strengths and weakness of the project, and unexpected results.2. To determine the overall efficiency in the u liza on of resources in achieving results.3. To assess the appropriateness of the design of the project and the implementa onarrangements, including but not limited to the project modality, organiza onalstructure, and coordina on mechanisms set up to support the project;4. To assess the extent to which the project has contributed to the crea on of anenabling environment, and the extent to which this has helped shape effec vegovernment policies and programming on disaster management and risk reduc on;5. To assess the sustainability of results and provide recommenda ons for sustainingthe benefits of the project and how to improve sustainability in future ini a ves;6. To assess the approach to capacity development and whether ini a ves havecontributed to sustainability;10
7. To review the effec veness of the gender mainstreaming strategy and partnership strategy;8. To gain insights into the level of client sa sfac on with the project. The clients includecommunity and local government beneficiaries; na onal government partners and donor;9. To i
RPJMG Rencana Pembangunan Jangka Menengah Gampong (Village Medium-Term Development Plan) SAR Search and Rescue. 5 SC-DRR Safer Communi es through Disaster Risk Reduc on in Development SD Elementary School Sekda Regional Secretary SKPA Satuan Kerja Perangkat Aceh (Aceh government’s local department or
Visual Guide to the UNDP Standard Development Project. 1 2 SELECT IMPLEMENTING PARTNER Ensure participation of wide range of stakeholders Contact MPTF O ce if UNDP is an . UNDP Brand Manual UNDP Tagline Portal Communication Toolkit Success Story Template Photo Guidelines See in Atlas: Standard CDR report On-Demand Atlas
1.1.3. Structure of the updated Evaluation Guidelines The Evaluation Guidelines are organized in a number of easy-to-use sections that can be used in sequence or as individual pieces and are intended to be a living and continuously evolving document to support evaluation in UNDP.
3 Evaluation reference group: The evaluation commissioner and evaluation manager should consider establishing an evaluation reference group made up of key partners and stakeholders who can support the evaluation and give comments and direction at key stages in the evaluation process.
2.3 Key conclusions 23 3. IMPLICATIONS OF THE INTERFACE FOR UNDP 25 3.1 Implications for UNDP programming 25 3.2 Key recommendations for UNDP 29 ANNEX 1: CASE STUDY FINDINGS 31 1. Bolivia 31 2. Haiti 34 3. Indonesia (Aceh) 37 4. Kenya 42 5. Kyrgyzstan 45 6. Papua New Guinea 49 7. Sri Lanka 52 8. Sudan 56 9. Zimbabwe 61 ANNEX 2. GLOSSARY OF KEY .
Special conditions5 4.1 The advance payment to be made upon signature of the contract by both parties is contingent upon receipt and acceptance by UNDP of a bank guarantee 6for the full amount of the advance payment issued by a Bank and in a form acceptable to UNDP.7 4.2 The amounts of the
Indicators used by HQ and CO should be linked by design. Last year, UNDP Mauritius CO linked the indicators in the country programme in retrospect to the indicators developed based on the UNDP Strategic Plan. Some of the indicators matched, while the others didn’t, undermining the effectiveness of UNDP’s integrated M&E system.
UNDP Inclusive Growth Team, Gonzalo Pizarro and Justus Okoko from UNDP SDG Integration Team based in Amman, as well as suggestions from Giulio Quaggiotto from UNDP Strategic Innovation Team and . Photo: Asian Development Bank is.gd/LNU8l4 . Next acPr tices — Innovations in the COID-19 social protection responses and eyond 9 .
our offices and business units, UNDP uses standard bidding documents, available in English, Spanish and French. On average, UNDP buys approximately 3 billion (US) worth of goods and services every year to maintain operations in the field and to react to upcoming and recurring demands. While the needs vary, services account
