Incident Command System (Ics) Performance Evaluation Final Evaluation .

The country programs have targeted the right actors for capacity building from an operationalperspective; i.e., the appropriate first responder individuals and units within the nationalgovernment. But, from a policy perspective, the programs in Indonesia and Thailand currentlyhave to solicit the support of high level officials to obtain the buy-in from staff at theimplementing agencies. There is a “top-down” mentality, where operational staff will buy-in toICS (or any program) if senior-level officials support it. ICS is generally accepted within the leadership and operational staff of the implementing agencyand partner organizations. ICS has been fully accepted as a best practice within the nationalgovernment and other partner agencies, but there are some cultural barriers to overcome. The evaluation team identified evidence of knowledge retention of ICS principles, particularly inthe Philippines and Thailand. However, the inability to conduct the retained knowledge survey inThailand and Indonesia forced the evaluation team to collect qualitative data on retainedknowledge outcomes. ICS has been integrated in national disaster frameworks in the Philippines and Thailand, whileIndonesia is anticipating the inclusion of ICS in the anticipated National Disaster ResponseFramework. Institutionalization of ICS in local laws and plans will help ensure the sustainabilityof ICS, even after USAID funding concludes. Barriers to the utilization of ICS exist, and the reasons are country-specific. In Indonesia andThailand, a “top-down” structure of governance limits the progress of ICS without the supportof higher level officials in the government. In the Philippines, substantially more trainings havebeen conducted, but the program needs to address “next level” questions, such as reachingmore barangays (villages) or expanding the program’s focus beyond training to developingincident management teams (IMTs). In all three countries, ICS resources have been translated and adapted; doctrine and guidingdocuments have been provided to key implementers who received training. However, eachprogram will need to further provide these resources to scale up the program’s reach and meetcurrent demands, such as requests for training for local governments. Funding is available from the country governments to continue the use of ICS if external supportends. The funding will be able to support the delivery of trainings, either by the USFS or thelocal cadre. The current pace of training delivery in all countries does not meet the localdemands.RECOMMENDATIONSWith the findings and conclusions gathered, the evaluation team recommends the following courses ofaction for USAID/OFDA to consider:Recommendation 1: Accelerate Basic and Intermediate Training. USAID, USFS, and the partner nationgovernments have collaborated well to establish fully functioning ICS programs in each country with thefull-suite of courses and certified master trainers to provide all courses. However, basic andintermediate trainings must be accelerated to meet local demands.Recommendation 2: Strengthen MT Cadres in All Countries. Related to the first recommendation, theprogram should strengthen and expand the MT cadres in all countries. “Strengthen” has a local context.9

The cadre in Indonesia, for example, only has 49 members with an additional 25 MT to be added by theend of 2017 with three courses to still be delivered this fall. The requests for training are coming fromall 34 provinces. But strengthening the Indonesia program means improving retention, aside fromexpanding training. The Philippines has the largest cadre with 431 trainers, but ICS courses have notreached all local government units and the development of IMTs at the local levels should be considereda possible way of strengthening. The Thailand program has experienced significant attrition in the cadredue to promotions, retirements, and concurrent duties, so accelerating trainings should be aconsideration.Recommendation 3: Provide Implementation-Focused Support and Tracking. All programs must addresskey programmatic areas heavily influenced by local conditions. Indonesia must find a champion at thehighest levels of the government agencies involved in disaster response due to the “top-down” structureof the government. The Philippines has trained approximately 27,000 participants, but stakeholdersshould review the program objectives and determine if it should evolve from training delivery to theestablishment of local IMTs. As the Department of Disaster Prevention and Mitigation (DDPM)transitions ICS training responsibilities to the DDPM Academy in Thailand, changes are expected in theway courses are managed. However, the program’s focus should continue to be on the accelerateddelivery of trainings.Recommendation 4: Develop Intermediate-Term Performance Targets. So that country initiatives can beachieved and measured objectively, the local government agencies in all countries should develop aperformance monitoring plan over the next two years with indicators that have ambitious but achievableperformance targets. While a plan is not a requirement under the agreement with the local governmentagencies, Indonesia and the Philippines have built monitoring systems for trainings. The Indonesia ICSprogram has collaborated with the University of Indonesia to develop a database, while the program inthe Philippines has a database of an estimated 27,000 trainees that also tracks post-training performance.The Thailand program is in its early stages, and USFS should make a recommendation to the DDPM todevelop a system. These resources can be used to develop performance monitoring to track how inputsmeet higher level program objectives to build ICS capacity.Recommendation 5: Assist in Developing a Participant Database. Related to Recommendation 4 above,the ICS program in each country should encourage all local government agencies to develop or maintaina database that will track training recipients. Programs should have a resource that collects and reportsmeaningful data with which to measure effectiveness, capacity, or targeting of current training effort.Recommendation 6: Establish a performance management system within the country programs. Relatedto Recommendations 4 and 5 above, the local government agencies should establish a performancemanagement system and go through the complete cycle to: 1) develop a performance management plan,2) construct a country-level results framework, 3) define indicators to track performance, and 4)conduct periodic assessments of outcomes. A performance management system will allow the countryteam to effectively monitor activities and manage results. As stated in Recommendation 4, foundationalresources in Indonesia and the Philippines have been built with databases, and Thailand should develop asimilar system. These databases can serve as primary data sources for information across the differentlevels of the results framework to monitor the program.10

I.EVALUATION PURPOSEThe goal of this performance evaluation is to improve USAID/OFDA’s understanding of theperformance of the ICS program supported by USAID/OFDA in the EAP region. The evaluation focusedon the effectiveness and sustainability of the USAID/OFDA funded ICS programs in Indonesia, thePhilippines and Thailand. The findings of this evaluation will inform future programming decisions andadjustments to ongoing ICS programming.After five years of supporting the ICS programming in a variety of countries, USAID/OFDA seeks adeeper understanding of the successes and areas for improvement in this capacity-building model. Anevaluation of the performance of the ICS program will allow USAID/OFDA decision-makers to apply theinformation in future decisions related to DRR programming and the ICS program model specifically. Aswith many capacity-building initiatives, contextual factors can influence the results of the intervention.The introduction of ICS into a variety of countries—all with their own disaster risks, governmentstructures, and socio-economic factors—has led USAID/OFDA to seek more information about thefactors for success and the potential for sustainability.II.EVALUATION QUESTIONSThis evaluation has two objectives: understanding effectiveness and sustainability. Each objective hasseveral lines of inquiry that shall inform the evaluation design. Data was collected for all lines of inquiryin the Philippines, Indonesia, and Thailand, and was analyzed to compare results from the differentcontexts. These objectives and their respective lines of inquiry are listed below.Objective 1: Effectiveness1. To what extent has the ICS program built the capacity of national governments to respond todisasters effectively, particularly at the initial stages of the emergency?2. Which aspects of the ICS training program are most effective? Which aspects of the ICS trainingprogram are least effective?3. Is the ICS targeting actors within the national government that are best positioned to effectivelyimplement elements of the ICS system?4. What country-level factors influence the effectiveness of the ICS program?5. To what extent do training participants—including the different levels of the cascade trainingsystem—retain knowledge and skills transferred through the ICS trainings?Objective 2: Sustainability6. To what extent are countries able to integrate the ICS into their national frameworks or otherinstitutionalization models?7. What barriers to utilization of the ICS exist?8. How much adaptation (translation of core concepts and revision of key elements to better reflectlocal conditions) of the ICS is required before the materials can be utilized at the local governmentlevel?9. What factors contribute to sustainability of the ICS? Will existing investments remain viable withoutexternal donor support?11

III. PROJECT BACKGROUNDICS Program BackgroundCountries in the EAP region face extremely high risks from typhoons (hurricanes), monsoon rains,volcanoes, earthquakes, tsunamis, and other natural phenomena. To mitigate these risks, USAID/OFDAhas funded a range of DRR activities in the region. In the OFDA context, DRR is a broad term thatincludes initiatives to prevent or mitigate the impact of hazards like earthquakes, floods, droughts, andstorms. Investments in disaster risk reduction help make communities more resilient to disasters andbetter able to manage and mitigate their impacts, saving lives, and reducing suffering before, during, andafter events occur. While DRR programming by USAID/OFDA in the EAP region focuses on contextspecific activities designed to meet the particular DRR needs of each individual country, host countryhuman and institutional capacity building has been a consistent theme throughout. Developing acapability to respond to disasters more effectively using the ICS has been a key element in all of theseefforts.ICS is a standardized incident management approach, applicable to any disaster scenario, which isdesigned to improve coordination and communication among the various actors and agencies involved indisaster response. ICS was originally developed in the U.S. in the 1970s, following a series ofcatastrophic southern California fires, when analysis revealed that inadequate management, rather thanlack of resources or failure of tactics, was the main reason for ineffective response. ICS was intended toaddress these management weaknesses, which had caused problems with accountability, communication,planning, management structures, and integration of interagency requirements.ICS comprises a set of basic principles, an organizational structure, and an operational planning process,and includes a clear chain of command, common terminology, interoperable communications,standardized training, and consistent certification requirements for disaster response staff. ICS expandedthroughout the U.S. in the 1980s and 1990s, and has since the early 2000s become a primary componentof the United States’ overall, presidentially mandated National Incident Management System (NIMS),where it is codified in the Command and Management section. More recently, spearheaded by the U.S.,the ICS approach has also spread internationally.One of several components of USAID/OFDA’s DRR strategy in EAP focuses on increasing the capacityof national disaster management agencies to effectively respond to disasters. As part of this effort,USAID/OFDA has funded USFS ICS capacity building activities across the EAP, providing training inBrunei, Indonesia, the Philippines, Thailand, Vietnam, Mongolia, the Solomon Islands, Timor Leste, andVanuatu, recently expanding activities to China, Malaysia, and Palau, and supported the EmergencyOperations Center in Burma and regional activities.Indonesia Country Program BackgroundThe combined threat to humans from natural disasters and climate change is nowhere more severe thanin Indonesia. Volcanoes, earthquakes, and tsunamis have gained media attention, but floods, droughts,storms, landslides, and forest fires have posed the greatest threat to the country over the past threedecades; according to the Badan Nasional Penanggulangan Bencana (BNPB), the National DisasterManagement Agency, the incidence of natural disasters has increased by almost 350 percent during thattime. This increase is largely due to the effects of climate change, population growth, the acceleratedextraction of resources, and other factors, which have exacerbated the frequency and intensity ofdisasters. Natural disasters affect the lives and livelihoods of millions of Indonesians, and have asubstantial impact on the Indonesian economy. From 2004 to 2010, BNPB estimates that the economicimpact of natural disasters totaled approximately 14 billion.An estimated 70 percent of disasters in Indonesia are hydro-meteorological in nature; therefore, the12

incidence and intensity of disasters is expected to continue to grow due to climate change andassociated effects. Natural disasters and climate change impacts will increase vulnerabilities for tens ofmillions of Indonesians. There are 17,000 islands in Indonesia - approximately 6,000 of which areinhabited - with a majority of the 240 million-population living near the coast. Climate change will affecta large segment of the population, threatening livelihoods, economic growth and stability of thisimportant country and strategic ally to the United States.The ICS program in Indonesia was launched to build on the previous training and technical supportprovided by the United States Government (USG). During Phase 1 (2004-2006) of the Association ofSoutheast Asian Nations (ASEAN)-U.S. Disaster Management Cooperation Program funded by theDepartment of State (DOS), Government of Indonesia (GOI) disaster managers were introduced to ICSthrough a series of workshops, a regional Basic/Intermediate ICS course conducted in Hanoi, Vietnam,and an ICS hosted study tour to the U.S. that included two GOI officials. Following the Association ofSoutheast Asian Nations (ASEAN)-U.S. Phase I activities, the USFS conducted a Basic/Intermediate ICScourse for GOI disaster managers in Jakarta in 2008 through the Indian Ocean Tsunami Warning System(IOTWS). USAID funded this activity. From 2008 to present, the Department of Justice InternationalCriminal Investigation and Technical Assistance Program (ICITAP) has conducted a series ofBasic/Intermediate/Advanced ICS courses at the national and provincial level. The primary audience forthese trainings is the Indonesian National Police (INP); however, at the request of the USAID/IndonesiaMission, ICITAP has included BNPB and other organizations in trainings when space is available. ICITAPhas also provided direct training and technical support on ICS to BNPB staff and some personnel fromthe Badan Penanggulangan Bencana Daerahs (BPBDs) or Provincial Agencies for Disaster Management inresponse to requests for assistance.In 2008, the BNPB issued Regulation 10, which mandates that GOI response agencies use ICS. BNPBrequested USG assistance in integrating ICS and other relevant NIMS components into the newlyestablished Indonesian disaster management system, and ICS is a primary component of BNPB’scurriculum for GOI disaster managers. Regulation 10 was updated in 2016 by Regulation 3 to further setthe framework for ICS application and the basis for the provision of training courses.USAID/OFDA provided 450,000 and USAID/Indonesia provided 1.085 million to support theimplementation of ICS in Indonesia from FY 2012 to 2015. The funding supported seven programactivities, namely the establishment of an ICS Steering Committee, the development of a training planand the adaptation of course materials, the completion of ICS

INCIDENT COMMAND SYSTEM (ICS) PERFORMANCE EVALUATION FINAL EVALUATION REPORT A Performance Evaluation of the ICS Program in East Asia and Covering the Assistance Provided by USAID/OFDA from 2012 August 14, 2017 [Contract No. AID-OAA-I-15-00018; Task Order No. AID-OAA-TO-16-00037] DISCLAIMER