Verification Of Adaptive Systems
DOT/FAA/TC-16/4Verification of Adaptive SystemsFederal Aviation AdministrationWilliam J. Hughes Technical CenterAviation Research DivisionAtlantic City International AirportNew Jersey 08405April 2016Final ReportThis document is available to the U.S. publicthrough the National Technical InformationServices (NTIS), Springfield, Virginia 22161.This document is also available from the Federal AviationAdministration William J. Hughes Technical Center atactlibrary.tc.faa.gov.U.S. Department of TransportationFederal Aviation Administration
NOTICEThis document is disseminated under the sponsorship of the U.S.Department of Transportation in the interest of information exchange. TheU.S. Government assumes no liability for the contents or use thereof. TheU.S. Government does not endorse products or manufacturers. Trade ormanufacturers’ names appear herein solely because they are consideredessential to the objective of this report. The findings and conclusions inthis report are those of the author(s) and do not necessarily represent theviews of the funding agency. This document does not constitute FAApolicy. Consult the FAA sponsoring organization listed on the TechnicalDocumentation page as to its use.This report is available at the Federal Aviation Administration William J.Hughes Technical Center’s Full-Text Technical Reports page:actlibrary.tc.faa.gov in Adobe Acrobat portable document format (PDF).
Technical Report Documentation Page1. Report No.2. Government Accession No.3. Recipient's Catalog No.DOT/FAA/TC-16/44. Title and Subtitle5. Report DateVERIFICATION OF ADAPTIVE SYSTEMSApril 20166. Performing Organization Code8. Performing Organization Report No.7. Author(s)Chris Wilkinson1, Jonathan Lynch1, Raj Bharadwaj1, and Kurt Woodham29. Performing Organization Name and Address10. Work Unit No. (TRAIS)1Honeywell International Inc.1985 Douglas Drive NGolden Valley, MN 55422-39352NASA Langley Research CenterMail Stop 130Hampton, VA 23681-219911. Contract or Grant No.IA1-1073, DFACT-10-X-0000812. Sponsoring Agency Name and Address13. Type of Report and Period CoveredFAA National Headquarters950 L'Enfant Plaza North, S.W.950 L’Enfant PlazaWashington, DC 20024Final Report14. Sponsoring Agency CodeAIR-13415. Supplementary NotesThe Federal Aviation Administration William J. Hughes Technical Center Aviation Research Division COR was Charles Kilgore.16. AbstractAdaptive software, which has the ability to change behavior at runtime in response to changes in the operational environment,system configuration, resource availability, or other factors, has been an active research topic, but with limited use to date in theaviation domain. Progress in adaptive flight control systems and plans for using adaptive systems in the Next Generation AirTransportation System, however, are compelling an examination of requirements for verification of these systems for commercialapplications. This report documents the findings of a two-phase research study of software assurance requirements for adaptivesystems, especially from the perspective of RTCA/DO-178B and C. Phase 1 of the study was conducted by NASA LangleyResearch Center and Phase 2 was conducted by Honeywell International Inc.17. Key Words18. Distribution StatementAdaptive systems, Neural networks, Machine learning,Verification, Assurance, Model-based development, FormalmethodsThis document is available to the U.S. public through theNational Technical Information Service (NTIS), Springfield,Virginia 22161. This document is also available from theFederal Aviation Administration William J. Hughes TechnicalCenter at actlibrary.tc.faa.gov.19. Security Classif. (of this report)UnclassifiedForm DOT F 1700.7 (8-72)20. Security Classif. (of this page)UnclassifiedReproduction of completed page authorized21. No. of Pages10422. Price
ACKNOWLEDGEMENTSResearch reported in this document was supported under Interagency Agreement IA1-1073,DTFACT-10-X-00008 between the Federal Aviation Administration (FAA) and NASA LangleyResearch Center (NASA-LaRC). The research was divided into two parts: Phase 1 performed byNASA-LaRC, and Phase 2 performed by Honeywell International Inc., under subcontract toNASA-LaRC. Completion of this research project would not have been possible without thesupport of Charles Kilgore, the FAA Contracting Officer’s Representative, and BarbaraLingberg, the FAA sponsor for the FAA’s Software and Digital Systems Research Program.iii
TABLE OF CONTENTSPageEXECUTIVE SUMMARYx1.INTRODUCTION11.11.223Overview of Phase 1 ActivitiesOverview of Phase 2 Activities2.TERMINOLOGY33.UNDERSTANDING ADAPTIVE APPROACHES63.13.23.33.44.The Feedback ProcessLife Cycle ContextLearning MethodRole of AdaptationUSE OF ADAPTIVE ALGORITHMS AND THEIR ROLE IN ADAPTIVESYSTEMS’ 0Fundamental Concepts in NNsExample Application of an NNObservations Concerning NNsGenetic AlgorithmsReflection/Autonomic ComputingSupporting Technologies111416161718ADAPTIVE CONTROLS AND THEIR USE IN ADAPTIVE SYSTEMS185.1Control System essRobustness of Non-Adaptive Control SystemsRobustness of Adaptive ControlAirworthiness Terminology Relevant to Stability/RobustnessAdaptive Control System Architecturesiv22
5.2.15.2.25.2.35.2.45.2.56.Model Reference Adaptive ControlModel Identification Adaptive ControlModel-Free Adaptive ControlAdaptive AugmentationL1 Adaptive Control2223242425INITIAL IDENTIFICATION OF SAFETY ISSUES256.16.26.3Impact of Life Cycle Context on SafetyImpact of the Role of Adaptation on SafetySafety Concerns for Adaptive etic AlgorithmsReflection/Autonomic ComputingAdaptive Controls7.INITIAL PHASE 2 ACTIVITIES308.CURRENT CERTIFICATION GUIDANCE AND STANDARDS338.1ARP-4754A Guidelines for Development of Civil Aircraft and Systems348.1.18.1.236388.28.39.Discussion of Recent Changes to ARP-4754Identified Textual Changes Within ARP-4754AARP-4761 Guidelines and Methods for Conducting the Safety AssessmentProcess on Civil Airborne Systems and EquipmentSoftware Design AssuranceADAPTIVE SYSTEM CERTIFICATION9.19.29.39.4Concerns Regarding the Feasibility of Applying DO-178B to Software DesignAssurance of Adaptive SystemsSuggested Approach to Software Design Assurance of Adaptive SystemsSystem-Level Approaches to the Certification of Adaptive SystemsDefining the System-Level Characteristics of an Adaptive System4040414142434410.ASPECTS OF TOOL QUALIFICATION FOR ADAPTIVE tions for Derivation and Validation of Adaptive System Safetyand Functional RequirementsRecommendations for Adaptive System Requirements VerificationFUTURE ADAPTIVE SYSTEMS CERTIFICATION RESEARCH NEEDSv484950
13.SUMMARY5014.REFERENCES52APPENDIX A—TERMINOLOGYAPPENDIX B—LITERATURE SEARCH RESULTSAPPENDIX C—DO-178B/C OBJECTIVES APPLIED TO ADAPTIVE SYSTEMSviA-1B-1C-1
LIST OF FIGURESFigurePage1Elements of the feedback process72Model of a neuron123NN topology124Gradient descent learning rules145Comparison of fuel measurement system approaches156Closed loop control system197Control system with gain scheduling218MRAC239MIAC2310Model-Free Adaptive Control2411Adaptive augmentation2512Adaptive system taxonomy3113Example of flight control architecture3214Certification process flow and applicable standards3415ARP-4754 and ARP-4754A sections mapping3716In/Out mapping of ARP-4754 and ARP-4754A37vii
LIST OF TABLESTablePage1Role of statistical techniques and representative applications112ARP-4754A invocation363DO-178B invocation414System safety objectives for adaptive systems45viii
LIST OF VVHMWCETAdvisory CircularArtificial intelligenceArtificial neural networkAir traffic managementCode of Federal RegulationsCertification review itemCertification SpecificationDesign Assurance LevelEuropean Aviation Safety AgencyEuropean Organization for Civil Aviation EquipmentFederal Aviation AdministrationFailure conditionFunctional development assurance levelFormal methodsHigh level requirementItem development assurance levelModel-based designModel Reference Adaptive ControlModel Identification Adaptive ControlNational Airspace SystemNational Aeronautics and Space Administration Langley Research CenterNext Generation Air Transportation SystemNeural networkSupplemental type certificateType certificateTechnical Standard OrderVerification and validationVehicle health managementWorst-case execution timeix
EXECUTIVE SUMMARYThis report documents the accomplishments of Phase 1 and Phase 2 of the Verification ofAdaptive Systems research task, under Interagency Agreement IA1-1073, DTFACT-10-X00008, between the Federal Aviation Administration and NASA Langley Research Center(NASA-LaRC). This research study addresses verification and safety assurance issues for the useof adaptive systems such as those planned for the Next Generation Air Transportation Systemand the National Airspace System air traffic control system. An adaptive system was defined as asystem that changes behavior based on an active feedback process in the presence of changes inthe system or its environment. Requirements for system safety assurance are based on thegeneral concept that correct behavior of a system can be specified, predicted, and verified priorto operation. Consequently, any proposed use of adaptive systems that violates that conceptraises issues that must be evaluated. The goal of this research was to conduct a preliminaryexamination of what is necessary to provide sufficient assurance that an adaptive system is safelyused in an aircraft product from a software perspective.The research in both phases of this effort targeted understanding the applicability of existingsoftware assurance requirements, especially those in RTCA/DO-178B, “Software Considerationsin Airborne Systems and Equipment Certification,” and the recently released update to DO178C, with its corresponding supplements, for adaptive systems. Work for Phase 1 wasperformed by NASA-LaRC, and the Phase 2 work was performed by Honeywell InternationalInc., under subcontract to NASA-LaRC.The Phase 1 effort focused on understanding the latest current technology in machine learningand the mechanisms that could cause an aircraft system to adapt or change behavior, in responseto change in its environment. Understanding the mechanisms used for adapting is essential tounderstanding the impact on software assurance. Much of the work in the initial phase consistedof gathering information on the broad field of machine learning, how machine learning is used toenable a system to adapt (especially with respect to feedback processes), and where machinelearning is being used in various domains (with particular interest in controls applications).Research is reported in five areas for Phase 1: 1) terminology; 2) understanding adaptiveapproaches; 3) the use of adaptive algorithms and their role in adaptive systems’ evaluation; 4)adaptive controls and their use in adaptive systems; and 5) initial identification of safety issues.In Phase 2, the disparate information on different types of adaptive systems developed underPhase 1 was condensed into a useful taxonomy of adaptive systems. As evident from thetaxonomy, the wide range of factors relevant to adaption makes it clear that the applicability ofthe DO-178C objectives will likely differ depending on the type of adaptive system. Therefore,determining the applicability of the current software assurance process is extremely difficult forthe general case (that is, for adaptive systems in general), but possible for a specific adaptivesystem. Consequently, the Honeywell team examined an exemplar adaptive system andevaluated how the output of that controller can be predicted and verified in compliance withsystem safety and assurance standards. A significant product of this evaluation is a table,provided in appendix C, that describes the impact of the exemplar adaptive system on each DO178C objective. In addition, a number of system-level objectives were identified that may benecessary to ensure that adequate verification of an adaptive system is possible. The importancex
of considering adaptive systems starting at the system level is discussed, along withrecommendations for follow-on work in AS safety and verification requirements.xi
1. INTRODUCTIONIn the “Decadal Survey of Civil Aeronautics: Foundation for the Future” [1], the NationalResearch Council identified intelligent and adaptive systems as one of the five common threadsfor the “51 high-priority R&T challenges.” In general, adaptive systems are defined as those thathave the ability to change behavior in response to changes in their operational environment,system configuration, resource availability, or other factors. Adaptive systems have been usedeffectively in a number of application domains, from industrial plant control to missile guidance,though they have not been used in civil aviation. However, that is expected to change. Thedecadal survey explicitly identified adaptive systems technologies to be the key enablers forintelligent flight controls; advanced guidance and adaptive air traffic management (ATM)systems; and for health management techniques to extend life and improve maintenance.Adaptive flight and engine control systems have been researched for decades and are attractivefor several reasons. There are adaptive systems that have the ability to detect, anticipate, andprevent failures and reconfigure various aircraft systems (e.g., displays or controls) in response;some that simply improve or optimize performance in a changing operational environment; andothers that can detect performance degradation due to failure or damage. Expected growth in airtraffic is another reason to research the potential. The Next Generation Air TransportationSystem (NextGen) Integrated Work Plan [2], for example, describes “net-enabled adaptivecontrol of ground, airborne and satellite weather observation sensors in real time” as an enablingcapability to meet needs for improved weather observations. Adaptive systems are also beingproposed for management of human machine interactions on aircraft and ATM systems tomitigate safety incidents due to failures at the human machine interface. In this case, theemphasis is on the system behavior that adapts to the current context (e.g., tasks, user state,system configuration, environmental states, etc.).The use of advanced computational techniques, such as those that underlie adaptive systems, isnot a new topic in the aviation domain. In 1994, the Federal Aviation Administration (FAA)published a chapter in their Digital Systems Validation Handbook titled “Artificial Intelligencewith Applications for Aircraft” [3]. Artificial intelligence (AI) is a broad and rapidly expandingfield of technology “devoted to computer programs that will mimic the product of humanproblem solving, perception, and thought” [3]. The handbook chapter provided an overview ofAI technology, focusing on expert systems, and identified potential certification issues foraviation systems that would use those technologies. At that time, expert systems were intended toautomate procedures that were already known and serve as assistants or advisors instead ofprimary decision tools. Today, expert systems are safely used in that capacity in aviationapplications.Adaptive systems, however, have succeeded expert systems as the next AI technology foraviation applications. Adaptive technologies, such as neural networks (NN), can be introducedinto the design of a system to achieve a goal such as enhancing performance or efficiency;maintaining desirable behavioral traits, such as robustness; or responding to changes in thesystem or its environment. Research supported by Eurocontrol investigated an NN-based systemfor automatic recognition and diagnosis of safety-critical, non-nominal events in ATM forimproving safety monitoring for the Single European Sky ATM Research initiative [4].1
Adaptive systems learn as they execute, thereby exhibiting behavior that can be less predictablethan traditional avionics systems. Because requirements for system safety assurance are based onthe concept that correct behavior of a system can be specified, predicted, and verified, any use ofadaptive systems in civil applications poses challenges in assuring safety by means of traditionalsafety assurance methods and procedures. This includes understanding the impact of adaptationon system requirements and design and software implementation and verification, becauseadaptation is ultimately realized through software.The primary aim of the Verification of Adaptive Systems task was to develop an understandingof the ramifications of adaptive systems on software assurance. The task also aimed, to the extentit was possible, to develop a rational and practical approach for the assurance of flight softwarethat uses adaptive techniques, potentially including approaches targeted at the system level. Thisreport documents the results of the two phases of research activity to accomplish those aims.1.1 OVERVIEW OF PHASE 1 ACTIVITIESWork on Phase 1 was performed by NASA Langley Research Center. Phase 1 research focusedon developing an understanding of the state-of-the-art in adaptive systems technology, especiallymachine learning, and how adaptive technology is used or proposed to be used in aviationapplications, including controls. The following four objectives were defined for Phase 1:Objective 1:Objective 2:Objective 3:Objective 4:Provide definitions of terminology associated with verifying adaptive systems in asafety-critical airborne environment (e.g., adaptive system, NN, adaptivesoftware, AI, and deterministic).Describe contrasting characteristics of adaptive systems and deterministicsystems, including relative benefits, strengths, and weaknesses.Investigate the differences between an adaptive approach to system developmentand a deterministic approach, and their effects on system and softwareverification.Identify safety issues when an adaptive, nondeterministic system approach is usedand propose mitigation techniques to address these in a safety-critical airborneenvironment.The intent of Phase 1 was to lay the groundwork necessary to identify the differences betweenconventional and adaptive systems from both a requirements and design perspective, andsubsequently identify any unique software safety considerations that would not be addressedusing existing assurance processes, especially DO-178B [5]. Much of the Phase 1 effort involvedgathering information about machine learning and the current uses of adaptive systems inindustry, and trying to develop a cogent terminology set associated with the use of machinelearning in aviation applications.Sections 2–6 of this report document the results of the Phase 1 effort. Section 2 provides anoverview of terminology issues for adaptive systems. Appendix A lists terms and definitionsrelevant to adaptive systems. Section 3 describes fundamental aspects of adaptive approaches,including strengths and weaknesses, with special emphasis on feedback processes. In section 4,adaptive algorithms are discussed, including NNs, genetic algorithms, and reflectiveprogramming. Section 5 presents different approaches to adaptive control. Section 6 then2
provides an initial assessment of safety issues for adaptive systems. Section 7 contains asummary of the Phase 1 work in preparation for Phase 2. Appendix B provides the results of theliterature search as a bibliography.1.2 OVERVIEW OF PHASE 2 ACTIVITIESWork in Phase 2 was performed by Honeywell International Inc. The Honeywell team startedwith the foundational work in Phase 1, then focused Phase 2 activities on determining the extentto which existing guidance in RTCA
adaptive controls and their use in adaptive systems; and 5) initial identification of safety issues. In Phase 2, the disparate information on different types of adaptive systems developed under Phase 1 was condensed into a useful taxonomy of adaptive systems.
Sybase Adaptive Server Enterprise 11.9.x-12.5. DOCUMENT ID: 39995-01-1250-01 LAST REVISED: May 2002 . Adaptive Server Enterprise, Adaptive Server Enterprise Monitor, Adaptive Server Enterprise Replication, Adaptive Server Everywhere, Adaptive Se
Summer Adaptive Supercross 2012 - 5TH PLACE Winter Adaptive Boardercross 2011 - GOLD Winter Adaptive Snocross 2010 - GOLD Summer Adaptive Supercross 2010 - GOLD Winter Adaptive Snocross 2009 - SILVER Summer Adaptive Supercross 2003 - 2008 Compete in Pro Snocross UNIQUE AWARDS 2014 - TEN OUTSTANDING YOUNG AMERICANS Jaycees 2014 - TOP 20 FINALIST,
Chapter Two first discusses the need for an adaptive filter. Next, it presents adap-tation laws, principles of adaptive linear FIR filters, and principles of adaptive IIR filters. Then, it conducts a survey of adaptive nonlinear filters and a survey of applica-tions of adaptive nonlinear filters. This chapter furnishes the reader with the necessary
Highlights A large thermal comfort database validated the ASHRAE 55-2017 adaptive model Adaptive comfort is driven more by exposure to indoor climate, than outdoors Air movement and clothing account for approximately 1/3 of the adaptive effect Analyses supports the applicability of adaptive standards to mixed-mode buildings Air conditioning practice should implement adaptive comfort in dynamic .
Characteristics of Complex Adaptive Systems Complex Adaptive Systems A complex adaptive system is a system made up of many individual parts or agents. The individual parts, or agents, in a complex adaptive system follow
new approaches for verification and validation. 1.1. Role of Verification and Validation Verification tests are aimed at "'building the system right," and validation tests are aimed at "building the right system." Thus, verification examines issues such as ensuring that the knowledge in the system is rep-
Design vs. Verification Verification may take up to 70% of total development time of modern systems ! This ratio is ever increasing Some industrial sources show 1:3 head-count ratio between design and verification engineers Verification plays a key role to reduce design time and increase productivity 10 IC Design Flow and Verification
The Project Gutenberg EBook of First Course in the Theory of Equations, by Leonard Eugene Dickson This eBook is for the use of anyone anywhere at no cost and with almost no restrictions whatsoever. You may copy it, give it away or re-use it under the terms of the Project Gutenberg License included with this eBook or online at www.gutenberg.org Title: First Course in the Theory of Equations .
