Virtual Integration And Incremental Assurance Of Critical .
Virtual Integration andIncremental Assurance ofCritical SystemsPeter H. FeilerLayered Assurance Workshop 2015Dec 8, 2015 2015 Carnegie Mellon UniversityDistribution Statement A: Approved for Public Release;Distribution is Unlimited
Copyright 2015 Carnegie Mellon UniversityThis material is based upon work funded and supported by the Department of Defense under ContractNo. FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software EngineeringInstitute, a federally funded research and development center.NO WARRANTY. THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERINGINSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITYMAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTERINCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE ORMERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL.CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITHRESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.[Distribution Statement A] This material has been approved for public release and unlimited distribution.Please see Copyright notice for non-US Government use and distribution.This material may be reproduced in its entirety, without modification, and freely distributed in written orelectronic form without requesting formal permission. Permission is required for any other use. Requestsfor permission should be directed to the Software Engineering Institute at permission@sei.cmu.edu.Carnegie Mellon is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.DM-0003120Distribution Statement A: Approved for Public Release;Distribution is UnlimitedVirtual Integration and IncrementalAssurance of Critical SystemsDec 8, 2015 2015 Carnegie Mellon University2
AgendaChallenges and Four Pillar Strategyfor Critical Software SystemsVirtual System IntegrationSoftware Hazards and VulnerabilitiesIncremental Lifecycle AssuranceDistribution Statement A: Approved for Public Release;Distribution is UnlimitedVirtual Integration and IncrementalAssurance of Critical SystemsDec 8, 2015 2015 Carnegie Mellon University3
We Rely on Software for Safe Aircraft OperationEmbedded softwaresystems introduce a newclass of problems notaddressed by traditionalsystem safety analysisDistribution Statement A: Approved for Public Release;Distribution is UnlimitedVirtual Integration and IncrementalAssurance of Critical SystemsDec 8, 2015 2015 Carnegie Mellon University4
Software Problems not just in AircraftHow do you upgrade washingmachine software?How do you prevent yourengine from cheating?Distribution Statement A: Approved for Public Release;Distribution is UnlimitedVirtual Integration and IncrementalAssurance of Critical SystemsDec 8, 2015 2015 Carnegie Mellon University5
High Fault Leakage Drives Major Increase in System CostAircraft industry has reached limits of affordabilitydue to exponential growth in SW size and complexity.RequirementsEngineering70% Requirements &system interaction errorsSystemDesign80% late errordiscovery at highreworkcostrepair costAcceptanceTest0%, 9% 80xSystemTest70%, 3.5% 1xSoftwareArchitecturalDesign20.5% 300-1000x10%, 50.5% 20xMajor cost savings through rework avoidanceby early discovery and correctionIntegrationTestA 10k architecture phase correction saves 3MComponentSoftwareDesignWhere faults are introducedWhere faults are foundThe estimated nominal cost for fault removal20%, 16%5xD. Galin, Software Quality Assurance: From Theory toImplementation, Pearson/Addison-Wesley (2004)B.W. Boehm, Software Engineering Economics, Prentice Hall (1981)Software as % of total system cost1997: 45% 2010: 66% 2024: 88%Post-unit test software rework cost50% of total system cost and growingSources:NIST Planning report 02-3, The Economic Impacts of InadequateInfrastructure for Software Testing, May 2002.UnitTestCodeDevelopmentDistribution Statement A: Approved for Public Release;Distribution is UnlimitedVirtual Integration and IncrementalAssurance of Critical SystemsDec 8, 2015 2015 Carnegie Mellon University6
System EngineerHazardsImpact ofsystem failuresSystemUnderControlOperator ErrorAutomation &human actionsPhysical PlantCharacteristicsLag, proximityData StreamCharacteristicsLatency jitter affectscontrol behaviorPotential event lossComputePlatformHardwareEngineerControl EngineerRuntimeArchitectureDistribution & RedundancyVirtualization, load balancing,mode confusionEmbedded SW SystemEmbedded software systemas major source of hazardsMeasurement Units, value rangeBoolean/Integer abstractionAir Canada, Ariane, 7500 Booleanvariable ation DeveloperSystem User/EnvironmentMismatched Assumptions in ITunes crashes on dual-coresWhy do system level failures still occur despite faulttolerance techniques being deployed in systems?Distribution Statement A: Approved for Public Release;Distribution is UnlimitedVirtual Integration and IncrementalAssurance of Critical SystemsDec 8, 2015 2015 Carnegie Mellon University7
Model-based Engineering PitfallsThe systemInconsistency betweenindependently developedanalytical modelsSystem modelsConfidence that modelreflects implementationSystem implementationThis aircraft industry experience has led to the SystemArchitecture Virtual Integration (SAVI) initiativeDistribution Statement A: Approved for Public Release;Distribution is UnlimitedVirtual Integration and IncrementalAssurance of Critical SystemsDec 8, 2015 2015 Carnegie Mellon University8
Awareness of Requirement QualityTextual requirement quality statistics Current requirement engineering practice relies onstakeholders traceability and document reviewsresulting in high rate of requirement changeNIST StudySystem to SW requirements gap [Boehm 2006]How do we trace low level SW requirements against system requirements?When StartUpComplete is TRUE in both FADECs andSlowStartupComplete is FALSE,the FADECStartupSW shall set SlowStartupInCompleteto TRUEDistribution Statement A: Approved for Public Release;Distribution is UnlimitedVirtual Integration and IncrementalAssurance of Critical SystemsDec 8, 2015 2015 Carnegie Mellon University9
Assurance & Qualification Improvement StrategyAssurance: Sufficient evidence that a systemimplementation meets system onArchitecture-centricVirtual SystemIntegrationStatic Analysis guration2010 SEI Study for AMRDECAviation Engineering DirectorateIncremental AssurancePlans & Casesthroughout Life CycleOperational& failuremodesResource,Timing lysisEarly Problem Discovery through Virtual System Integration & AnalysisImproved Assurance through Better Requirements & Automated VerificationDistribution Statement A: Approved for Public Release;Distribution is UnlimitedVirtual Integration and IncrementalAssurance of Critical SystemsDec 8, 2015 2015 Carnegie Mellon University10
Improved Cost, Time and Quality70% DefectIntroductionReduced Cost and Time through Early Discovery80% Post UnitTest DiscoveryImproved Quality through Better Requirements & EvidenceAssure theSystemDistribution Statement A: Approved for Public Release;Distribution is UnlimitedVirtual Integration and IncrementalAssurance of Critical SystemsDec 8, 2015 2015 Carnegie Mellon University11
AgendaChallenges and Four Pillar Strategyfor Critical Software SystemsVirtual System IntegrationSoftware Hazards and VulnerabilitiesIncremental Lifecycle AssuranceDistribution Statement A: Approved for Public Release;Distribution is UnlimitedVirtual Integration and IncrementalAssurance of Critical SystemsDec 8, 2015 2015 Carnegie Mellon University12
SAE Architecture Analysis & Design Language(AADL) to the RescueSW Design ArchitectureCommand &ControlPhysical systemTask & CommunicationArchitectureDeployed onPhysical interfaceDistributedComputer PlatformAADL Addresses Increasing InteractionComplexity and Mismatched AssumptionsDistribution Statement A: Approved for Public Release;Distribution is UnlimitedVirtual Integration and IncrementalAssurance of Critical SystemsDec 8, 2015 2015 Carnegie Mellon University13
SAE AADL Standard Suite (AS-5506 series)Core AADL language standard (V2.1-Sep 2012, V1-Nov 2004) Strongly typed language with well-defined semantics Textual and graphical notation Standardized XMI interchange formatStandardized AADL ExtensionsError Model language for safety, reliability, security analysisARINC653 extension for partitioned architecturesBehavior Specification Language for modes and interaction behaviorData Modeling extension for interfacing with data models (UML, ASN.1, )Guidance for runtime executive generationAADL Extensions in ProgressRequirements Definition and Assurance LanguageSynchronous System Specification LanguageHybrid System Specification LanguageSystem Constraint Specification LanguageDistribution Statement A: Approved for Public Release;Distribution is UnlimitedVirtual Integration and IncrementalAssurance of Critical SystemsDec 8, 2015 2015 Carnegie Mellon University14
AADL: The LanguagePrecise execution semantics for components Thread, process, data, subprogram, system, processor, memory, bus,device, virtual processor, virtual busContinuous control & event response processing Data and event flow, call/return, shared access End-to-End flow specificationsOperational modes & fault tolerant configurations Modes & mode transitionModeling of large-scale systems Component variants, layered system modeling, packaging, abstract,prototype, parameterized templates, arrays of components, connectionpatternsAccommodation of diverse analysis needs Extension mechanism, standardized extensionsDistribution Statement A: Approved for Public Release;Distribution is UnlimitedVirtual Integration and IncrementalAssurance of Critical SystemsDec 8, 2015 2015 Carnegie Mellon University15
System Level Problem AreasTime-sensitive Data Stream AssumptionsEnd-to-end latency analysisPort connection consistency Stream miss rates, Mismatched data representation, Latency jitter & agePartitions as Isolation Regions Competing demands by security and safetyProcess and virtual processor tomodel partitioned architectures Space and time partitioning for processors and networks Isolation not guaranteed due to undocumented resource sharingVirtualization of Resources Logical vs. physical redundancyVirtual processors & busesMultiple time domains Virtualization of timeInconsistent System States & Interactions Modal systems with modal components Concurrency & redundancy management Application level interaction protocolsResource guarantees Performance impedance mismatchess Unmanaged system resourcesOperational and failure modesInteraction behavior specificationDynamic reconfigurationFault detection, isolation, recoveryResource allocation &deployment configurationsResource budget analysis& scheduling analysisCodified in Virtual Upgrade Validation methodDistribution Statement A: Approved for Public Release;Distribution is UnlimitedVirtual Integration and IncrementalAssurance of Critical SystemsDec 8, 2015 2015 Carnegie Mellon University16
Well-defined Execution SemanticsOMG putationalModelAADL Execution &Communication ModelSAE AADLOMG MARTEFocus on implementation Timers to trigger task execution Send/receive operations Behavioral states and transitionsFocus on Architecture Abstraction Thread executionCommunication timingOperational modes & architecturereconfiguration Distribution Statement A: Approved for Public Release;Distribution is UnlimitedVirtual Integration and IncrementalAssurance of Critical SystemsDec 8, 2015 2015 Carnegie Mellon University17
Partitioned Run-Time icationSoftwareComponentAADL ModelApplication SW insertedinto runtime systemAADL Runtime SystemRuntime systemdispatches app codeAuto-generated task &communication codefrom AADL modelRuntime system performscommunication for appReal-Time Operating SystemEmbedded Hardware TargetStrong PartitioningVirtual processor andvirtual bus conceptsInteroperability/Portability Timing Protection Tailored Runtime Executive OS Call Restrictions Standard RTOS API Memory Protection Application ComponentsAADL ARINC653 Annex aligned with latest ARINC653 StandardDistribution Statement A: Approved for Public Release;Distribution is UnlimitedVirtual Integration and IncrementalAssurance of Critical SystemsDec 8, 2015 2015 Carnegie Mellon University18
Analysis of Virtually Integrated SoftwareSystemsSingle Annotated Architecture Model AddressesImpact Across Operational Quality AttributesSafety& ReliabilitySecurity Intrusion MTBF IntegrityArchitecture Model FMEA Confidentiality HazardanalysisPotential newhazardDataQuality Data precision/accuracy Temporalcorrectness ConfidenceAffects temporalcorrectnessAuto-generatedanalytical modelsChange of Encryptionfrom 128 bit to 256 bitResourceConsumptionReal-timePerformance Bandwidth Execution time/Deadline CPU time Powerconsumption Deadlock/starvation LatencyHigher CPU demandIncreased latencyDistribution Statement A: Approved for Public Release;Distribution is UnlimitedVirtual Integration and IncrementalAssurance of Critical SystemsDec 8, 2015 2015 Carnegie Mellon University19
Incremental Multi-Tier Assurance in SAVIAircraft: (Tier 0)Aircraft system: (Tier 1)Engine, Landing Gear, Cockpit, Weight, Electrical, Fuel, Hydraulics, LRU/IMA System: (Tier 2)Hardware platform, software partitionsPower, MIPS, RAM capacity & budgetsEnd-to-end flow latencySystem & SW Engineering:Mechatronics: Actuator & WingsSafety Analysis (FHA, FMEA)Reliability Analysis (MTTF)OEM & Subcontractor:Subsystem proposal validationFunctional integration consistencyData bus protocol mappingsSubcontracted software subsystem: (Tier 3)Tasks, periods, execution timeSoftware allocation, schedulabilityGenerated executablesRepeated Virtual Integration Analyses:Power/weightMIPS/RAM, SchedulingEnd-to-end latencyNetwork bandwidthProof of Concept Demonstration and Transition by Aerospace industry initiative Architecture-centric model-based software and system engineeringArchitecture-centric model-based acquisition and development processMulti notation, multi team model repository & standardized model interchange Multi-tier system & software architecture (in AADL) Incremental end-to-end verification of system propertiesDistribution Statement A: Approved for Public Release;Distribution is UnlimitedVirtual Integration and IncrementalAssurance of Critical SystemsDec 8, 2015 2015 Carnegie Mellon University20
Latency Sensitivity in Control SystemsSystem EngineerControl stemControlCommon latency data from systemengineering Processing latency Sampling latency Physical signal latencyImpact of Scheduler Choice on Controller StabilityA. Cervin, Lund U., CCACSD 2006Distribution Statement A: Approved for Public Release;Distribution is UnlimitedVirtual Integration and IncrementalAssurance of Critical SystemsDec 8, 2015 2015 Carnegie Mellon University21
Software-Based Latency ContributorsExecution time variation: algorithm, use of cacheProcessor speedResource contentionPreemptionLegacy & shared variable communicationRate group optimizationProtocol specific communication delayPartitioned architectureMigration of functionalityFault tolerance strategyDistribution Statement A: Approved for Public Release;Distribution is UnlimitedVirtual Integration and IncrementalAssurance of Critical SystemsDec 8, 2015 2015 Carnegie Mellon University22
Incremental Latency AnalysisLatency analysis throughout life cycle Functional & system architecture: latency budgets Task & communication architecture: processing, sampling, transfer Platform architecture: partitions, protocols, computer hardwareLatency contributors Systems: processing, sampling, queuing latency Connections: protocol overhead, physical transfer, sampling Partitions: sampling, window scheduleTrade studies Best-case & worst-case, latency jitter Mid-frame and frame-delayed communication Synchronous and asynchronous systems Partition end and major frame output policy Empty & full queue Data set processingTop-down & bottom-upUtilizes end-to-end flowsIncremental refinementInterprets deployment bindingsOperational mode specific analysis Latency budgets & rate, size, time based actualsDistribution Statement A: Approved for Public Release;Distribution is UnlimitedVirtual Integration and IncrementalAssurance of Critical SystemsDec 8, 2015 2015 Carnegie Mellon University23
Detailed Latency Analysis ReportsDistribution Statement A: Approved for Public Release;Distribution is UnlimitedVirtual Integration and IncrementalAssurance of Critical SystemsDec 8, 2015 2015 Carnegie Mellon University24
Finding Problems EarlyIssue: Contractor could notassess integration risk earlyenough.Action: 6 Week VirtualIntegration identified 20major issues.International CommercialAircraft Industry ConsortiumSystem Architecture Virtual Integration (SAVI) 2008Proof of concept with AADL led to ten year commitmentSAVI ROI Study (2009/10) 2B savings on 10B aircraft through 33% early detectionResult: Adjusted CDRSchedule to remediate. Prevented 12 monthdelay in a 2 year project.The current method wouldnot have identified theissues until 3 months beforedeliveryArchitecture-centric VirtualIntegration Practice (ACVIP)2014/15 Virtual Integration Shadow led to early discoveryof 85 potential integration issuesLed to acceleration of adoption by JMR contractors andinclusion in RFP for FY16/17 projectsVirtual Integration and Assurance: Impact on FVLJuly 14, 2015 2015 Carnegie Mellon University25
Towards an Architecture-Centric VirtualIntegration Practice (ACVIP)Army and other Government Shadow k IIIATAMJPLMission DataSystemFuture Vertical LiftArchitecture-centricAcquisitionJMR TD: ACVIP Shadow ProjectsCH47FHealthMonitorVirtual SystemIntegrationSystem AssuranceSystem Architecture Virtual Integration (SAVI) Software & Systems EngineeringAADLSoftware & suranceSAE AADL Standard & AADL Workbench: Research Transition PlatformDARPAMetaHACMEAADLError CMSSecurityUS & European Research nsAvionics NetworkStandardsSystem SafetyPractice StandardsRegulatory GuidanceNRC, FDA, ULOther Standards and Regulatory Guidance20042016Distribution Statement A: Approved for Public Release;Distribution is UnlimitedVirtual Integration and IncrementalAssurance of Critical SystemsDec 8, 2015 2015 Carnegie Mellon University26
AgendaChallenges and Four Pillar Strategyfor Critical Software SystemsVirtual System IntegrationSoftware Hazards and VulnerabilitiesIncremental Lifecycle AssuranceDistribution Statement A: Approved for Public Release;Distribution is UnlimitedVirtual Integration and IncrementalAssurance of Critical SystemsDec 8, 2015 2015 Carnegie Mellon University27
Safety Practice in Development Process ContextLeveson (MIT) Socio-technical Control Framework basedon Rasmussen (NASA) model of risk managementMultiple hazard contributors in development andoperational contextLabor-intensiveEarly in system engineeringRarely repeated due to costFocus on System Engineering LargelyIgnores Software as Hazard SourceSafety analysis automation through virtualsystem integrati
Introduction. Reduced Cost and Time through Early Discovery. Improved Quality through Better Requirements & Evidence. 12. . Core AADL language standard (V2.1-Sep 2012, V1-Nov 2004) . Concurrency & redundancy management .
Some other works on incremental learning and its applications include the incremental learning fuzzy neural (ILFN) network for fault detection and classification [5], incremental learning for multi-sensor data fusion [6], incremental genetic learning for data classification [7], incremental semi-supervised learn-ing [8], incremental learning .
44 Incremental Backups: John Snow; FOSDEM 2017 Life Cycle - First Incremental (The first step of our journey) Example 3: Create an incremental backup. Can be done via transaction or single QMP command. { "execute": "drive-backup", "arguments": {"device": "drive0", "bitmap": "bitmap0", "target": "inc.0.qcow2", "format": "qcow2", "sync": "incremental",
"Virtual PC Integration Components" software must be installed into each virtual machine. In a Windows host, the "Virtual PC Integration Components" software for a Windows virtual machine is located at C:\Program Files (x86)\Windows Virtual PC\Integration Components\ Multiple virtual machines can access the same target folder on the host.
Incremental learning for a mining algorithm, especially the classification mining algorithms, is a very important ability. Many studies of incremental learning ability were down with many classification methods like RBF neural network, Support vector and k-Nearest Neighbor [6-9]. And the applications of incremental classification focus
fundamentally different from the incremental algorithms studied in this paper.2 The paper is organized as follows. In Section 2, we formulate the problem of interest, and introduce the cyclic incremental and Markov randomized incremental method with stochastic errors. We also discuss some applications that motivate our interest in these methods.
30. Incremental Load Factor Cumulative Occurrences per 1000 Hours by Taxi and Roll 3 1. Incremental Load Factor Cumulative Occurrences per 1000 Hours Before and After Flight 32. Incremental Load Factor Cumulative Occurrences per 1000 Hours by Ground Phase 33. On-Ground Incremental Load Factor Cumulative Occurrences per 1000 Hours 34.
both in train and inference time. In domain incremental, the task identifier is provided only in train time, and the classi-fier does not need to infer the task identifier but rather just solve the task at hand. In class incremental, the learner also needs to infer the task identifier in inference time. We focus on the task incremental setting .
Incremental learning refers to learning from streaming data, . and applications which emerged in the last years. . ent possibilities to assess the performance of incremental learning algorithms: (1) Incremental-vs- non-incremental: In particular in the absence of concept
