A Comprehensive Review And Analysis Of Maturity Assessment .

Proceedings of the World Congress on Engineering and Computer Science 2009 Vol IIWCECS 2009, October 20-22, 2009, San Francisco, USAA Comprehensive Review and Analysis of MaturityAssessment Approaches for Improved Decision Support toAchieve Efficient Defense AcquisitionNazanin Azizian, Dr. Shahram Sarkani, Dr. Thomas MazzuchiAbstract— Based on multiple reviews of major defenseacquisition programs, the Government Accountability Office(GAO) has consistently reported that the Department ofDefense (DoD) acquisition programs are experiencingdifficulties in terms of schedule slips, cancellations, and failureto meet performance objectives due to insufficiently maturetechnology, unstable design, and a lack of manufacturingmaturity. The GAO claimed that “maturing new technologybefore it is included in a product is perhaps the mostdeterminant of the success of the eventual product or weaponsystem.” As a result, the DoD adopted the TechnologyReadiness Level (TRL) metric as a systematic method to assesstechnology maturity. However, as a result of increasingcomplexity of defense systems and the lack of objectivity of thetool, the TRL metric is deficient in comprehensively providinginsight into the maturity of technology. Objective and robustmethods that can assess technology maturity accurately andprovide insight into risks that lead to cost overruns, scheduledelays, and performance degradation are imperative formaking well-informed procurement decisions.Realizing this challenge, numerous other models andmethods have been developed to efficiently supplement andaugment the TRL scale, as well as provide new means ofevaluating technology maturity and readiness. The workpresented in this paper has investigated the literature, andleading research and industrial practices for technologymaturity assessment techniques, which are then analyzed usingthe SWOT (Strength, Weakness, Opportunity, Strength) model.The right maturity assessment techniques at the right time canenable government agencies and contractors to produceproducts that are cheaper, better, and made faster by closingknowledge gaps at critical decision points. This paper providesa comprehensive review and analysis of the prominent maturityassessment techniques in order to provide a selection criterionfor decision makers to choose the best fit method for theirprogram.Index Terms—Technology Maturity, Technology Readiness,TRL.I. INTRODUCTIONSuccessful parallel development and integration ofcomplex systems result in successful programs. ComplexNazanin Azizian is a doctoral candidate at The George WashingtonUniversity, Department of Engineering Management and SystemsEngineering, Washington DC, 20052. She is also employed as a SystemsEngineer by Lockheed Martin, Undersea Systems and Sensors (USS) (phone:571-214-4055; e-mail: nazanin@gwmail.gwu.edu).Dr. Sarkani and Dr. Mazzuchi are professor and department chair,respectively, at The George Washington University, Department ofEngineering Management and Systems Engineering, Washington DC, 20052(e-mail: sarkani@gwu.edu; mazzu@gwu.edu ).ISBN:978-988-18210-2-7systems are comprised of multiple technologies and theirintegrations. Due to scale and complexity of systems,stakeholders want confidence that risk is minimal and theprobability of successful technology development andintegration is high before investing large sums of money.Confidence is achieved when it is ensured that the developedor improved technologies can meet system requirements. Asa result, quantitative assessment tools that can provide insighton whether a group of separate technologies at variousmaturity levels can be integrated into a complex system at alow risk is beneficial to the success of a program and can helpsupport decision making of stakeholders.Tetlay and John [1] call the 21st century “The SystemsCentury” due the increasing complexity and high-integrationof technological products. They contend that assessingsystem maturity and readiness during the life-cycle ofdevelopment is imperative to the overall success of thesystem. Tetlay and John [1] argue that in recent years highinterest is taken in metrics such as the Technology ReadinessLevel (TRL), System Readiness Level (SRL), ManufacturingReadiness Level (MRL), Integration Readiness Level (IRL)and other metrics as avenues to measure maturity andreadiness of systems and technologies. These metrics areused to assess the risk associated with the development andoperation of technologies and systems; therefore they are away of ensuring that unexpected will not occur.However, literature has revealed gaps in clearly specifyingwhether the objective of the metrics and methods is tomeasure maturity or readiness. In general, maturity andreadiness are used interchangeably. In addition, in most casesthe applicability of tools and methods toward technologyversus a system is vague. Literature does not distinguishbetween maturity and readiness, and rarely specifies whethera method has been designed for a system or a technology.This research will generalize maturity and readiness as oneentity and refer to them throughout this paperinterchangeably. The objective of this research is to developa comprehensive assessment review of methods and toolsused to evaluate the maturity and readiness of technology andsystems. Although a technology and a system are not thesame, many of the methods described in this paper do notdistinguish between them.Systematically measuring technology and system maturityis a multi-dimensional process that cannot be performedcomprehensively by a one-dimensional metric. Although theTRL metric has been endorsed by the government and manyindustries, it captures only a small part of the information thatstakeholders need to support their decisions. This paperpresents other maturity assessment methods that have beendeveloped other rectify this shortcomings of the TRL.WCECS 2009

Proceedings of the World Congress on Engineering and Computer Science 2009 Vol IIWCECS 2009, October 20-22, 2009, San Francisco, USAII. BACKGROUNDA. Problem StatementLiterature repeatedly denotes that acquisition programsexperience cost overruns, schedule slips, and performanceproblems [2-14]. Based on a review of major defenseacquisition programs, the Government Accountability Office(GAO) has consistently reported that the Department ofDefense (DoD) acquisition programs are experiencingdifficulties in terms of schedule slips, cancellations, andfailure to meet performance objectives as a result ofinsufficiently mature technology, unstable design, and a lackof manufacturing maturity. The GAO claimed that “maturingnew technology before it is included in a product is perhapsthe most determinant of the success of the eventual product orweapon system” [8]. More recently, based on assessment of72 Weapons Programs, the GAO reported in March 2008 [7]thatbefore entering production [7]. All in all, the GAOconcluded that DoD programs enter various phases ofacquisition and product development knowledge gaps thatresult in design, technology, and production risks.B. DoD Acquisition Lifecycle FameworkThe DoD has adopted Evolutionary acquisition as anstrategy to deliver an operational capability over severalincrements, where each increment is dependent on asufficiently defined technology maturity level. The objectiveof evolutionary acquisition is to quickly hand off a capabilityto a user in a manner in which the technology developmentphase successively continues until the required technologymaturity is achieved and prototypes of system componentsare produced. Each increment is comprised of a set ofobjectives, entrance, and exit criteria [15]. The stages of theDoD acquisition life-cycle are depicted in figure 2.“none of them had proceeded through system developmentmeeting the best practices standards for maturetechnologies, stable design, or mature productionprocesses by critical junctures of the program, each ofwhich are essential for achieving planned cost,schedule, and performance outcomes”Further, the GAO [7] reported that in the fiscal year 2007,the total acquisition cost of major defense programs haverisen 26% from the initial estimate, while development costsgrew by 40%, and programs have failed to deliver thepromised capabilities. The trends in cost and schedulegrowth in these defense programs over the years are depictedin figure 1.Analysis of DOD Major Defense Acquisition ProgramFiscal year 2008Fiscal Year2000 Portfolio2005 Portfolio2007 PortfolioNumber of Programs759195Total Planned Commitments 790 Billion 1.5 Trillion 1.6 TrillionCommitments Outstanding 380 Billion 887 Billion 858 BillionPortfolio PerformanceChange to total RDT&E costsfrom first estimate27%33%40%Change in total acquisition costfrom first estimate6%18%26%Estimated total acquisition costgrowth 42 Billion 202 Billion 295 BillionShare of programs with 25percent or more increase inprogram acquisition unit cost37%44%44%Average schedule delay indelivering initial capabilities16 Months17 Months21 MonthsFigure 1: GAO Assessments of Major DefenseAcquisition ProgramsOver the past 6 years the GAO has been reporting to theDoD that its weapons system acquisition programs aresuffering in the area of cost growth and schedule delays, andunfortunately in a 2008 report the GAO continued to statethat these cost and schedule problems have not been rectified.The resulting cost overrun and schedule delays is of nosurprise because no program followed the best practicesstandards for maturing technology, stabilizing design, andmaturing production process. In fact the GAO (2008)reported that 88% of the assessed programs began systemdevelopment without fully maturing critical technologies;96% of the programs had not demonstrated the stability oftheir designs before entering system demonstration phase;and no program had fully matured their production processesISBN:978-988-18210-2-7Figure 2: DoD Acquisition Life-CycleThe Milestone Decision Authority (MDA) in collaborationwith the appropriate stakeholders determines if sufficientknowledge is obtained at each phase of the acquisitionlife-cycle before proceeding to the next phase. It is importantto note that the initial phase of evolutionary acquisitions ispreceded by Material Development Decision phase. Anacquisition program can begin at any stage of the acquisitionlife-cycle, but it must meet the entrance criteria to enter thenext phase. Therefore, if a program that is conceived aftermilestone B of the acquisition process framework, atechnology readiness assessment must be conducted toensure that the technologies meet the requirement for theupcoming phase [4, 5, 16].C. Technology Readiness Assessment (TRA) ProcessThe DODI 5000.2 establishes the requirement for theperformance of Technology Readiness Assessment (TRA) inany defense acquisition program. The TRA is a systematic,metric-based process that evaluates the maturity of CriticalTechnology Elements (CTEs) of a system and isaccompanied by a report that identifies how the CTEs areselected, and why are considered critical. TRA is notintended to assess the quality of the system architecture,design, or integration, but only reveal the readiness of criticalWCECS 2009