A Proactive And Top-down Approach To Managing Risk At NASA

·,:iTtto -I A Proactive and Top-down Approach toManaging Risk at NASA2010 Trilateral Safety and Mission Assurance ConferenceOctober 26-28, 2010Cleveland, OhioHomayoon Dezfuli, Ph.D.NASA Technical Fellow (System Safety)NASA Headquarters

·.Outline Risk Management OverviewRisk-informed Decision Making ProcessContinuous Risk Management ProcessSummary2

Risk Management Policy NASA Policy Directive (NPD) 1000.5 (2009) states: "It is NASA policy toincorporate in the overall Agency risk management strategy a riskinformed acquisition process that includes the identification, analysis,and management of programmatic, infrastructure, technical,environmental, safety, cost, schedule, management, industry, andexternal policy risks that might jeopardize the success with which theAgency executes its acquisition strategies. " NPR 8000.4A (2009), Agency Risk Management Procedural Requirements,evolves NASA's risk management (RM) approach to entail twocomplementary processes:-Risk-informed Decision Making (RIDM) To risk-inform direction-setting decisions (e.g., space architecture decisions) -To risk-inform the development of credible performance requirements as part of the overallsystems engineering processContinuous Risk Management (CRM) To manage risk associated with the implementation of baseline performance requirements RM RIDM CRM'----/3

-.What Improvements Are We Looking for? To manage risk in a holistic and coherent manner across the Agency- To better match the stakeholder expectations and the "true" resourcesrequired to address the risks to achieve those expectations- Agency strategic goals explicitly drive RM activities at all levelsAll risk types and their interactions are considered collectively duringdecision-makingHaving an integrated perspective of risks when analyzing competingalternativesImplementation of RM in the context of complex institutional relationships(programs, projects, centers, contractors, . )Better comprehension of the risk that a decision-maker is accepting whenmaking commitments to stakeholdersTo better establish close ties between the selected alternative and therequirements derived from it-Derivation of achievable requirements through systematic characterizationof uncertainties4

-.The RM Process Begins with NASA StrategicGoals Within NASA's organizational hierarchy, high-level objectives (NASA Strategic Goals) flowdown in the form of progressively more detailed performance requirements, whosesatisfaction assures that objectives are metRIDM is designed to maintain focus on strategic goals as decisions are made throughoutthe hierarchyCRM is designed to manage "risks" in the context of requirementsRisk ManagementRIOM whhln an Organlzadonal Unitklentmc.tfon of Aftefnltfv Risk-infonned Aftemalive SeI.ctlonPerformance Requirements DevelopmentCRM within the Organizational Unit Talkedto Implement the S.lected AlternativeBevate o.daIon Ont L.neI Up'f Needed andReport Top RJIks One Level UpPRs: Performance RequirementsPMs: Performance Measures5

Definitions of Risk and RIDM per NPR 8000.4 Risk: The expression of the potential for performance shortfalls,which may be realized in the future, with respect to achievingexplicitly established and stated performance requirements- The performance shortfalls may be related to anyone or more ofthe following mission execution domains: Safety Technical performance Cost ScheduleRIDM: A risk-informed decision-making process that uses a diverseset of performance measures along with other considerations within adeliberative process to inform decision making- decisions are informed by an integrated risk perspective ratherthan being informed by a set of individual "risk" contributions- A decision-making process relying primarily on a narrow set ofmodel-based risk metrics would be considered "risk-based"6

RIDM Process StepsBased on NASAlSP-2010-576Risk-Informed Decision Making (RIDM)Part 1 - Identification of AlternativesStep 1 - Understand Stakeholder Expectations .and Derive Performance MeasuresStep 2 - Compile Feasible Alternatives Part 2 - Risk Anaillsis of AlternativesStep 3 - Set the Framework and Choose theAnalysis MethodologiesStep 4 - Conduct the Risk Analysis andDocument the Results Part 3 - Risk-Informed Alternative SelectionStep 5 - Develop Risk-Normalized PerformanceCommitmentsStep 6 - Deliberate, Select an Alternative, andDocument the Decision RationaleTo Requirements doctree/SP2010576.htm7

RIDM Process - Part 1Understand Stakeholder Expectations and Derive PerformanceMeasures An objectives hierarchy is constructed by subdividing the top-levelobjectives into more detailed objectives, thereby clarifying the intendedmeaning.At the first level of decomposition, the top-level objective is partitionedinto the mission execution domains of Safety, Technical, Cost, andSchedule.Within each domain, the objectives are further decomposed untilappropriate quantifiable performance objectives are coE:gII ObiectivesSafety 51MisExecsi nu lonDom ety [TechnicalObjective Objective1n,IIISafetyObj9ctlven1, 'I.IIIITechnicalObjectiveO2 ,1fITechnicalCostObjective Objective1n,:rechnicalObjective1,1I ObjectivesSchedUleslCost 51ObjectivesIIIiI e ve1,1ScheduleObjectiven.IIScheduleObjectiven 1Notional Objectives Hierarchy8

RIDM Process - Part 1Performance Measures and Performance Requirements A Performance Measure (PM) is a metric used to quantify theextent to which a Performance Objective is fulfilled- Safety (e.g., avoidance of injury, fatality, or destruction of key assets) Maintain Astronaut Safety Probability of Loss of Crew (P(LOC»- Technical (e.g., increase thrust or output, maximize amount ofobservational data acquired) Maximize Payload Capability Payload Capability (kg)- Cost (e.g., execution within minimum cost) Minimize Cost Cost ( )- Schedule (e.g., meeting milestones) Minimize completion time Schedule (months)The PM values imputed to the selected alternative arePerformance Requirements- They essentially define "success"- Significant shortfalls in performance are "failures"9

RIDM Process - Part 2Risk Analysis of Alternatives The goal is to develop a risk analysis framework that integrates domainspecific performance assessments and quantifies the performancemeasures- Risk Analysis - probabilistic modeling of performanceProbabilistically - DeterminedUncertain onmentLimitedDataTechnologyDevelopmentRisk Analysisof an AlternativeProduct ofRiskAnalysisPerformance Measure 1 Safety Risk Technical Risk Cost Risk Schedule RiskDesign, Test &ProductionProcessesEtc.* Performance measures depicted for a single alternative Establishing a transparent framework that:-Operates on a common set of performance parameters for each alternativeConsistently addresses uncertainties across mission execution domains and acrossalternativesPreserves correlations between performance measures10

Setting Risk Analysis Framework Detailed domain-specific analysis guidance is available in domainspecific guidance documents like the NASA Cost EstimatingHandbook, the NASA Systems Engineering Handbook, and the NASAProbabilistic Risk Assessment Procedures Guide } "I"m:Jne.MHsun 112iu' §/P' "":lne.M.;I u,",27I ,*,"":lnlMMNSU,. nP,"Drmanu PIlr.tm".r 1P,,*,rnulnc. P:tr.tm."r 211

RIOM Process Part 3Part 3 - Risk-Informed Alternative SelectionFromPart 2Risk-InformedSelection Report*- Technical Basis forDeliberation (TBfD)Step 5 -Develop Risk-NormalizedPerformance Commitments Establish risk tolerances onthe performance measures Establish performancemeasure ordering Determine performancecommitment valuesStep 6 - Deliberate, Select an Alternative,and Document the Decision Rationale Deliberate pros and cons of each alternative andassociated performance commitments Identify contending alternatives (downselection) Select an alternative Finalize the performance commitments Document decision rationaleAdditional Risk Analysis,Information Gathering,Performance CommitmentRevision, as Needed* To be defined in formal project management documentation.12

RIDM Process - Part 3Risk-informed Alternative Selection Performance measure probability density functions (pdfs)constitute the fundamental risk analysis results.However, there are complicating factors for performancemeasures that are expressed as pdfs:- The pdfs for different alternatives may overlap, preventing a definitiveassessment of which alternative has superior performancePDF(PM(Alt1)) &PDF(PM(Alt2)) gII!l z9W% II' - - -:: ; : :::-:-:l1 1 -----------,o2Good-Performance Measure34BadDifferent pdfs may exceed imposed constraints at different percentiles,thereby comingling issues of performance with issues of success13

Performance Commitment Mean values are used in many differentcontexts to compare alternatives, but thisapproach can:- A Performance Commitment is the level ofperformance whose probability of notbeing achieved matches the decisionmaker's risk tolerance- Produce values that are strongly influenced bythe tail ends of the pdfsIntroduce significant probabilities of falling shortof imposed constraints, even when the meanvalues meet imposed constraintsAnchors the commitment the decision maker (OM)is willing to make for that performance measurePerformanceCommitment ClC- - . . . ; ; . . . . - - -. Direction of GoodnessPerformance Measure XPerformance commitments support a risknormalized comparison of decisionalternatives, at a level of risk tolerancedetermined by the decision maker.14

Deliberation of the Merits of Each Alternative in the Contextof Performance Commitments (notional)Risk tolerances given by the shadedareas under the pdfs, on the "bad"side of the performance commitmentsPerformance commitments are set atperformance measure values thatcorrespond to given risk tolerancesAlternativeAPC A2IAlternativeB1/\ III EPC B3IAlternativeCIII1 /\IC C1Payload: CapabilityImposedConstraintIDirection of GoodnessL1IIPCB1PCA3PC C2PC C3ReliabilityCost & SchedulePerformance Measures·Notional Risk Tolerances: High Moderate Low These are arbitrary, notional choices15

.The Continuous Risk Management Process Is initiated by the results of the RIDM process:- The risk analysis for the selected alternative- An initial risk list Focuses on meeting performancerequirementsBy managing performance margins overtime so that associated performancerequirements are not violatedBy "burning down" (over time) the risk ofviolating performance requirementsBy means of mitigation actionsSteps in the CRM ProcessIdentifykientffy Risk Contributors (Shortfall in PerformanceRelative to Baseline Performance Requirements) -1IIIj-IAnalyzeEstimate Likelihood and Consequence Componentsof the Risk Through Analysis (Including UncertaintyEvaluation), Estimate Aggregate Risks if FeasibleI PlanDecide on Risk Disposition and Handling, Developand Execute Mitigation Plans. Decide VIAlat Will beTrackedtProgram/ProjectStartPerl.Req.R"k PerformanceMeasure XIntermediateMilestonePerl.Req.Mission Time Perl.""k1\ R"k JjPerformanceMeasure XTrackTrack Observables Relating to PerformanceMeasures (e.g. performance data, schedulevariances, etc.)Direction ofGoodness ControlControl Risk by Evaluating Tracking Data to VerifyEffectiveness of Mitigation Plans. Make Adjustmentto the Plans, and Execute Contr Measures-PerformanceMeasure X16

Summary Our ultimate goal is to manage risk in a holistic and coherentfashion across the Agency- The RIDM process is intended to risk-inform direction-settingdecisions- The CRM process is intended to manage risk associated with theimplementation of baseline performance requirements Currently we are working on- Enhancements to the CRM process- Better integration of the RIDM and CRM processes- Better integration of institutional risk considerations into RMframework17

o 2 3 4 Good Performance Measure Bad . Program/Project Intermediate Mission Time Start Milestone Perl. Perl. Perl. Req. R"k ""k1\ Req. R"k Jj Performance Performance Performance Measure X Measure X Measure X Dir