Test and Evaluation of Autonomous Systemsin a Model Based Engineering ContextRaytheonMichael NolanUSAF AFRLAaron FifarekJonathan Hoffman3 March 2016Copyright 2016. Unpublished Work. Raytheon Company.
Agenda MotivationTrust and Certification ProcessBackgroundFormal AnalysisRequirements AnalysisArchitectureModel TraceabilitySysML Representation of Autonomous System andAutonomous System Development Basic example of Autonomous Systems T&E in MBE context SummaryApproved for Public Release. Case Number: 88ABW-2015-59592
MotivationIntroduction, Discovery, and Cost of Software lopment70% of faults are introduced3.5% faults are found1x estimated nominalcost for fault removalArchitectureDevelopmentOpportunity to find faults asthey are introduced whencosts are tionImplementation20% of faults are introduced16% faults are foundNIST Planning report 02-3, The Economic Impacts of5x estimated nominalInadequate Infrastructure for Software Testing, May 2002.cost for fault removalD. Galin, Software Quality Assurance: From Theory to20.5% faults arefound300-1000x estimatednominalcost for fault removal10% of faults are introduced59.5% faults are found20-80x estimated nominalcost for fault removalRework and certification is70% of SW cost.4Implementation, Pearson/Addison-Wesley (2004)3. B.W. Boehm, Software Engineering Economics, Prentice Hall(1981)Approved for Public Release. Case Number: 88ABW-2015-59593
Trust and CertificationProducts / ProcessCompositionally Verified Systems of SystemsValidationDesignRequirementFormalization &AnalysisAnalytical Proof SynthesisSimulation Requirements Testing Architecture FORMALIZEDModeling,SAFETY ASSESSMENTAssuranceSimulation, Test NewAutonomyNeedValidator Models& EvaluationHAZARD ogyPathsArchitectureFormalization &AnalysisREQUIREMENTSRun TimeAssuranceSystem Design and Safety Requirements(ARP 4761, ARP 4754/A, MIL-HDBK-882E)Testable Requirements & Verification Plans(DO-178C/254, MIL-HDBK-516)Approved for Public Release. Case Number: 88ABW-2015-59594
Formal AnalysisFormal Methods refers to mathematically rigorous techniques and tools for thespecification, design and verification of software and hardware systems.- Langley Formal Methods (http://shemesh.larc.nasa.gov/fm/fm-what.html) What is Formal Analysis?– Analysis performed on mathematically precise models utilizing elegantComputer Science algorithms and tools Model-Checking Theorem Proving Why do we want to do it?– We can exhaustively search the behavior of models to prove or disprovedesired properties– Removal of ambiguity due to required mathematical rigor– Can identify unintended and unspecified behaviorsApproved for Public Release. Case Number: 88ABW-2015-59595
AnalysisAdvantage of Model CheckingTesting Checks Only the Values We SelectEven Small Systems Have Trillions(of Trillions) of Possible Tests!Model Checker Tries Every Possible Value!Finds every exception to theproperty being checked!Approved for Public Release. Case Number: 88ABW-2015-59596
Requirements Development & AnalysisPrecise, structured standards to automate requirementevaluation for testability, traceability, and de-conflictionApproved for Public Release. Case Number: 88ABW-2015-59597
Formal Requirements Analysis Natural language requirements are difficult to process logically andmathematically especially if they are not written with a formal basis–“The flight control function that performs the automatic avoidance maneuvershall be of a level of redundancy equivalent to the primary flight control system” What is the formal definition of this constraint on the system? Not a trivial definition on the systemFormal Methods refers to mathematically rigorous techniques and tools for thespecification, design and verification of software and hardware systems.- Langley Formal Methods oral logic definitions are not obvious to write for most individuals and takes years ofpractice to master effectivelyWhat does that mean?There may be logical basis butit’s not accessible to others.Approved for Public Release. Case Number: 88ABW-2015-59598
Formal Requirements Analysis Our Approach – Pattern Implementation– Constrain natural language to patterns which contain a scope and a predicate– Enforces the formal basis necessary to ensure mathematical rigor Can requirements be defined and verified compositionally?PropertyPatternsClassesApproved for Public Release. Case Number: nceBounded ExistenceOrderPrecedenceResponseChain PrecedenceChain Response9
ArchitectureGuarantee appropriate decisions with traceable evidenceduring the system architectural designApproved for Public Release. Case Number: 88ABW-2015-595910
Architecture: AADL and AGREE The Architecture Analysis & Design Language (AADL)– Developed by SAE– Architecture modeling notation with well-defined semantics Assume Guarantee REasoning Environment (AGREE)plugins– Developed by University of Minnesota and Rockwell Collins– Part of the DARPA High-Assurance Cyber Military Systems on: somethinga system assumes aboutit’s environment (inputs)ImplementationGuarantee: what you canassume about the systemand the performance ofthe system (outputs)1. Kathleen Fisher, “Using Formal Methods to Enable More Secure Vehicles:Tufts University”, 16 September, 2014 DARPA's HACMS Program, s/sites/28/2014/05/HACMS-Fisher.pdf [cited 27 Jul. 2015].Approved for Public Release. Case Number: 88ABW-2015-595911
AGREEAssume Guarantee REasoning Environment Assume-Guarantee Contract -Verifiable set of Assumptions and Guarantees thatabstracts the behavior of a system component implementation AssumptionsConstraints over whata component expects to seefrom its environment GuaranteesConstraints over how acomponent behaves inresponse to its environmentApproved for Public Release. Case Number: 88ABW-2015-595912
Compositional Verification A series of techniques to allow for systemsto be decomposed into less complexmodules to be enforce a hierarchicalstructure that can be leveraged forcompositional techniques Systems can be hierarchically organized 1– Requirements vs. architectural designmust be a matter of perspective– Need better support for N-leveldecompositions for requirementsand architectural design1. Whalen, Michael W., et al. “Your “What” Is My “How”: Iteration and Hierarchy in System Design.”Software, IEEE 30.2 (2013): 54-60.Approved for Public Release. Case Number: 88ABW-2015-595913
Model DevelopmentCumulative Evidence Through Research,Developmental, and Operational TestApproved for Public Release. Case Number: 88ABW-2015-595914
Introduce Simulink and SLDV Uses formal methods to find violations ofdesign properties and assumptions Formal Analysis techniques from:– Prover Plug-In– Polyspace formal analysis engine from MathWorksApproved for Public Release. Case Number: 88ABW-2015-595915
SLDV AnalysisProperty ModelPropertyModelPropertyBlocksApproved for Public Release. Case Number: 88ABW-2015-595916
Requirements TraceabilityRequirement - SpeAR PropertyArchitecture - AGREE GuaranteeModeling - Simulink Design Verifier PropertyApproved for Public Release. Case Number: 88ABW-2015-595917
Model Lifecycle Management PerspectiveMLM autonomy perspectivestarts with MLM framework«analysis model»rev1«arch model»rev1BobMary«cad model»rev1
SysML Representation of Autonomous System andAutonomous System Development---Building on the MLMframeworkNominal autonomous systemmodeled in SysML (Rhapsodyexample)UML Test Protocol or similar utility is usedEnables efficient pairing of requirements, teststraps, procedures, reports, and other artifacts witheach member of a product familyModels are executable within modelingenvironment at chosen level of fidelity3/7/201619
Basic example of Autonomous Systems T&E inMBE context Basic Machine Learning algorithm hosted in Simulink Data sets for nominal autonomous system developed Simulink components integrated within Rhapsody (SysML) Model executed in the SysML environment SysML test utilities placed around test and test results– IBM Test Conductor or potentially RQM wrapper Systems trained with different data sets behaved differently MBE considerations– Configuration management, Data management– Flexibility, product family architecture support– Training Data is paired with the autonomous system Ability to trace system development back to the training data set usedAutonomous systems development requiresadditional MBSE considerations3/7/201620
Summary Discovery of critical flaws early in the design process can save timeand money Formal requirement traceability throughout design process Composability for reuse and modular verification Autonomous systems development requires additional MBSEconsiderationsApproved for Public Release. Case Number: 88ABW-2015-595921
Future Directions of Work Continued research in the Development Process– Requirements Realizability arguments could identify early conflictsNatural language masking of formal representations– Architecture Abstraction of different compositional levels across different teams– Modeling Bounding nonlinear behavior within discrete defined systems Assurance Case Construction– Utilize the artifacts from the Development Process to provide evidence of behavior Move the formulation forward with these artifacts Implementing the Development Process on more complex systems– Testing the scalability of the techniques– Designing challenges that approach the complexity of Air Force domain systems– Potentially build on MBSE – autonomy structure Run-time Assurance for nonlinear autonomy– If we can’t formally prove or test can we bound?– How can we safely bound a system?Approved for Public Release. Case Number: 88ABW-2015-595922
firstname.lastname@example.org@raytheon.comCopyright 2016. Unpublished Work. Raytheon Company.
(ARP 4761, ARP 4754/A, MIL-HDBK-882E) Testable Requirements & Verification Plans (DO-178C/254, MIL-HDBK-516) Certified Assurance Case Compositionally Verified Syst
Page 2 Autonomous Systems Working Group Charter n Autonomous systems are here today.How do we envision autonomous systems of the future? n Our purpose is to explore the 'what ifs' of future autonomous systems'. - 10 years from now what are the emerging applications / autonomous platform of interest? - What are common needs/requirements across different autonomous
Autonomous Differential Equations 1. A differential equation of the form y0 F(y) is autonomous. 2. That is, if the right side does not depend on x, the equation is autonomous. 3. Autonomous equations are separable, but ugly integrals and expressions that cannot be solved for y make qualitative analysis sensible. 4.
2 Research and development case study: Robotics and autonomous systems research Introduction This case study on robotics and autonomous systems research is one of a series that we have developed to support and complement our published report on research and development. Our examination of robotics and autonomous systems research
systems to autonomous systems. Firstly, the autonomy and autonomous systems in different ﬁelds are summarized. The article classiﬁes and summarizes the architecture of typical automated systems and infer three suggestions for building an autonomous system architecture: extensibility, evolvability, and collaborability.
programming interfaces and utilizes JAUS-compliant perception and navigation components. REV Autonomous Navigation System The REV is equipped with sensors and algorithms for autonomous navigation, in addition to the teleoperation and semi-autonomous controls. There are two main autonomous navigation modes. Table 1: REV vehicle specifications
text, autonomous robots and mobile manipulators form the forefront of recent developments. In the Master's Program 'Autonomous Systems' (MAS) students will learn the practical skills and intellectual abilities necessary for the design and develop-ment of such autonomous systems. This Master's Program takes the form of a "Master by Research". So .
Fraunhofer CML Autonomous Vehicles’ Impact on Port Infrastructure Requirements Hamburg Port Authority AöR 10 78 Management Summary 1 Management Summary Autonomous solutions are developed for road, rail and waterborne transport. Autonomous driving describes the independent locomotion of vehicles and is a
xt-generation autonomous systems - Main Characteristics . Next-generation autonomous systems emerge from the needs to further automate existing complex organizations by progressive and incremental replacement of human agents by autonomous agents. Such systems exhibit "broad intelligence" by using and producing knowledge in order to
Oracle has been steadily building out its autonomous software road map, having released Oracle Autonomous Database, Oracle Autonomous Linux and Autonomous Data Guard. Now the . and it provides simple document APIs for programming languages via REST or the command line interface. Contrary to widespread expectations regarding an Oracle
The overall design of the autonomous pesticide spraying robot is illustrated in Fig. 1. The design is done using Solidwork software and the development of the autonomous pesticide spraying system based on the design. The specification of the autonomous pesticide spraying robot is shown in Table I.
of autonomous military robots, or “autonomous weapon systems.” The treatment of such subjects in the ethics, robotics, and popular literature has generally assumed that autonomous systems either fit perfectly into existing legal regimes or threaten long-standing paradigms. This Article demonstrates that neither assumption is correct.
Introduction Model Learning on a Sony Aibo Model Learning on an Autonomous Car Conclusions Model Learning for Autonomous Robots Goal: To increase the effectiveness of autonomous mobile robots Plan: Enable mobile robots t
autonomous innovations that project staff may uncover while going about their regular tasks. Autonomous Innovation can be supported programmatically either through a 'mainstreaming' approach, where programme staff recognise contexts or individuals with a conducive enabling environment for Autonomous Innovation and work to encourage this, or
predict the steering wheel angle to navigate the autonomous vehicle safely. So that the autonomous vehicle is depended on the training dataset. If the CNN model is not trained on the roadway obstacle than navigation system of autonomous vehicle may generate incorrect information about the steer
Autonomous Mobile Robots Margarita Chli, Paul Furgale, Marco Hutter, Martin Rufli, Davide Scaramuzza, Roland Siegwart ASL Autonomous Systems Lab localization “Introduction to Autonomous Mobile Robots
autonomous driving solutions is a highly valuable skill in today's software engineering field. Robot Operating System (ROS) is a meta-operating system that simplifies the process of robotics programming. This master's thesis aims to demonstrate how ROS could be used to develop autonomous driving software by analysing autonomous driving
the autonomous navigation of these systems. The global positioning system (GPS) is used for external autonomous navigation . Because GPS signals are typically absent or weak indoors, autonomous navigation is difficult . There are various approaches for independent indoor navigation which have been proposed in recent years.
POINT METHOD OF JOB EVALUATION -- 2 6 3 Bergmann, T. J., and Scarpello, V. G. (2001). Point schedule to method of job evaluation. In Compensation decision '. This is one making. New York, NY: Harcourt. f dollar . ' POINT METHOD OF JOB EVALUATION In the point method (also called point factor) of job evaluation, the organizationFile Size: 575KBPage Count: 12Explore further4 Different Types of Job Evaluation Methods - Workologyworkology.comPoint Method Job Evaluation Example Work - Chron.comwork.chron.comSAMPLE APPLICATION SCORING MATRIXwww.talent.wisc.eduSix Steps to Conducting a Job Analysis - OPM.govwww.opm.govJob Evaluation: Point Method - HR-Guidewww.hr-guide.comRecommended to you b
4. 12 Meter (40') Drop Within Test 5. Fast Cook-Off Within Test 6. Slow Cook-Off Within Test 7. Bullet Impact Within Test 8. Fragment Impact Within Test 9. Sympathetic Detonation Within Test 10. Shaped Charge Jet Impact Within Test 11. Spall Impact Within Test 12. Specialty Within Test 13. Specialty Within Test 14. Specialty Within Test 15 .
duction test, version test, the Maddox rod test, and the red glass test . The red glass test is widely used in both emergency rooms and clinics because it can be performed quickly as an initial evaluation of diplopia . However, the red glass test is not quantitative, and the results can vary test by test or between examiners.