Integrated Diagnostics Of Root Cause Faults In Complex Network And It .
INTEGRATED DIAGNOSTICS OF ROOT CAUSE FAULTS IN COMPLEX NETWORK AND IT SYSTEMS. Dr. Yuri Rabover Co-founder, Director Product Strategy VMTurbo Yuri.Rabover@VMTurbo.com Member of the SMARTS founding team We are looking for summer interns! Please contact Yuri Å
WHAT IS IT ALL ABOUT? IN SEARCH OF ROOT CAUSE .
BACKGROUND Smarts: System Management ARTS Founded in 1993 as a network management R&D lab y Patented the leading correlation technology y Built a software platform and solutions to leverage it Thousands of customers throughout the world Networks Systems Applications Bought by EMC2 in 2005 for 275 mln. Yuri joined SMARTS in 1993.
PROBLEM SCENARIO PROBLEM Switch 0 L0 Card 0 Port 0 L1 Port 0 L0 L0 L1 L1 L0 L1 Port 1 Switch 2 Card 0 Card 1 L1 Port 1 Card 1 Port 0 Port 1 Port 1 L0 Port 0 Card 0 Port 1 Card 1 Port 0 L1 L0 L0 L1 L1 L0 L0 Port 1 Switch 3 Port 0 L1 L1 L0 L0 L1 L1 Card 0 Port 1 Card 1 L0 Port 0 Switch 1 L0 L0 Port 0 L1 L1 L0 L0 L1 L1 Port 1 -4
PROBLEM SCENARIO SYMPTOMS Switch 0 L0 Card 0 Port 0 L1 L1 L0 L0 L1 L1 Port 1 Card 1 Port 0 L0 L1 Port 1 Switch 2 Card 0 Card 1 PROBLEM Port 0 Port 1 Port 1 L0 Port 0 Card 0 Port 1 Card 1 Port 0 L1 L0 L0 L1 L1 L0 L0 Port 1 Switch 3 Port 0 L1 L1 L0 L0 L1 L1 Card 0 Port 1 Card 1 L0 Port 0 Switch 1 L0 L0 Port 0 L1 L1 L0 L0 L1 L1 Port 1 -5
PROBLEM SCENARIO SYMPTOMS Switch 0 L0 Card 0 Port 0 L1 L1 L0 L0 L1 L1 Port 1 Card 1 Port 0 L0 L1 Port 1 Switch 2 Card 0 Card 1 PROBLEM Port 0 Port 1 Port 1 L0 Port 0 Card 0 Port 1 Card 1 Port 0 L1 L0 L0 L1 L1 L0 L0 Port 1 Switch 3 Port 0 L1 L1 L0 L0 L1 L1 Card 0 Port 1 Card 1 L0 Port 0 Switch 1 L0 L0 Port 0 L1 L1 L0 L0 L1 L1 Port 1 -6
PROBLEM SCENARIO SYMPTOMS Switch 0 L0 Card 0 Port 0 L1 L1 L0 L0 L1 L1 Port 1 Card 1 Port 0 L0 L1 Port 1 L0 Card 0 Port 1 Card 1 Port 0 L1 L0 L0 L1 L1 L0 Card 0 Port 0 Port 1 Port 1 Port 1 Switch 3 L0 Port 0 L1 L1 L0 L0 L1 L1 Card 0 Port 1 Card 1 L0 Port 0 Switch 1 Port 0 UNIQUE SIGNATURE Switch 2 Card 1 PROBLEM L0 L0 Port 0 L1 L1 L0 L0 L1 L1 Port 1 -7
RCA CHALLENGES Problems can start in any logical or physical object in the network, attached systems, or applications. A single problem often causes many symptoms in many related objects. The absence of particular symptoms is as meaningful as the presence of particular symptoms. Different problems can cause many overlapping symptoms. For example, the operationally down status of logical port 0 over physical port 1 in Card 0 of Switch 1 could be caused by a failure in any one of the following components: y Switch 1 y Switch 2 y Switch 1 Card 0 Switch 1 Card 0 Port 1 Switch 1 Card 0 Port 1 Logical Port 1 Switch 2 Card 0 Switch 2 Card 0 Port 1 Switch 2 Card 0 Port 1 Logical Port 1 The trunk connecting Port 1 in Card 0 in Switch 1 with Port 1 in Card 0 in Switch 2.
RCA CHALLENGES – CONT. Symptoms can propagate across related components. In our example, a symptom propagated from a card to its ports and from a port to a peer port. This makes it necessary to examine all the symptoms across related elements in order to identify the root cause. Problems are not always observable in the object where they originate. For example, if the switch does not generate card failure traps, the card failure would have no direct (local) symptoms. Problems can occur in objects that do not generate any observable events. For example, there is no event and/or trap associated with a trunk.
CONVENTIONAL APPROACH WRITE RULES
CONVENTIONAL APPROACH DOWNSTREAM SUPPRESSION PSTN
CONVENTIONAL APPROACHES difficult to separate symptoms from root-cause problems manual problem diagnosis rules-based “models” change outpaces rules writing significant expertise and time silo management—no correlation across technology silos multiple groups chasing the same problem need the whole view to isolate faults “Sea of red” Rules-Based Management puts the burden of analysis on IT Operations BUDGETS
WHAT’S MISSING? Display Modeling Analysis No built-in analysis — ongoing rules-writing required Data & Event Collection Databases IDS Firewalls Servers Applications Switches Telephony Video Optical Routers Storage
MODELING VS ROOT CAUSE ANALYSIS Modeling uses a top-down approach: y Starting from the problems to be diagnosed y Modeling Moving to the symptoms caused by the problem Problem: the battery is dead Symptom: the car does not start Root cause analysis uses abduction y Starting from the symptoms observed y Analysis Symptom: the car does not start Infer the root cause Problem: the battery is dead «symptom» Car.DoesNotStart «problem» Battery.Dead «causes» 14
ROOT CAUSE ANALYSIS – PROBLEM DEFINITION Wikipedia: y Root cause analysis (RCA) is a class of problem solving methods aimed at identifying the root causes of problems or events. Smarts: y Given a set of symptoms, find the problem, or the set of problems, that best explains those symptoms. Smarts: Given a set of symptoms, find the set of problems that best explains those symptoms. 15
Causal Network Model of a Technology Object Causation C – relation from P to S associating individual problems and symptoms. P S d1 d2 d3 d4 d5 m1 m2 m3 m4 m5
BOTTOM-UP APPROACH TO ROOTCAUSE ANALYSIS Q. What are the symptoms available to me? Q. What can I make out of these symptoms? 1 K4 N1 om tom pt mp ym Sy S 33 mM S pto Syymmp t o m 6 mX pto Sym Z3 om t p m Sy 75 W ? Sympt om D1 T9 S Symptom ym oAm1 Sy Sym pto t 2 ptom mp C4m m p m K1 y 4 o to Vp6t A S 9 m y m 2 S 5 m S tom SQyH o t 3m17pto p p m S y m A7 m y S m tompto p y m mF S Sy 12 17
TOP-DOWN APPROACH TO ROOT-CAUSE ANALYSIS Q. What problems do I want to be able to diagnose? Q. Which symptoms are manifested by those problems And what is the minimum set of symptoms needed to uniquely diagnose each problem? Problem D2 Problem D3 1 K4 N1 om tom pt mp m Sy Sy 33 mM S pto Syymm p tom Problem D1 Z3 om t p m y S 75 W Sympt om D1 T9 S Symptom ym oAm1 Sy Sym pto t 2 ptom mp C4m m pt m K1 y 4 o t V p o S 6 A 9 m m 5S2y m S SQyH to tom 3m17pto p p m S y m A7 S m tyommpto p y m mF S Sy 12 6 mX pto Sym y 18
TOP-DOWN APPROACH TO ROOTCAUSE ANALYSIS Q. What problems do I want to be able to diagnose? Q. Which symptoms are manifested by those problems And what is the minimum set of symptoms needed to uniquely diagnose each problem? Problem D2 Problem D3 Problem D1 Sy m pt 4 om A 2 5 m S tom SQyH o t 3m17pto p p m y m A7 m S m to p y m S Sy Sy Sympt om D1 y mp t o m Z3 19
REMINDER Entia non sunt multiplicanda praeter necessitatem. The simplest explanation is usually the best. The simplest explanation Considers the minimum set of symptoms to support each possible explanation. The law of parsimony plays a major role in resource management. Don’t collect all available symptoms, only those relevant to the problems diagnosed. 20
EMC SMARTS TECHNOLOGIES: BEHAVIOR MODELS Service subscriber PROBLEMS Codebook Codebook s0 Application Switch Router UNIQUE SIGNATURE s2 s3 s1 s2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 s0 c0 s3 s0 c0 s0 c0 s0 c0 s0 c0 s0 s0 s0 s0 s0 c0 c0 c0 c0 c0 s0 s0 s0 s0 s0 c0 c0 c0 c0 c0 s0 s0 s0 c0 c0 c0 s0 s0 s0 c0 c0 c0 s0 c0 s0 c0 s0 c0 s0 c0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 s0 c0 s0 c0 1 1 1 1 1 1 1 1 s0 c0 s0 c0 s0 c0 s0 c0 1 1 1 1 s0 c0 s0 c0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 s0 c0 s0 c0 1 1 1 1 1 1 1 1 1 1 1 s0 c0 s0 c0 1 1 1 1 1 1 11 1 1 1 1 1 s0 c0 s0 c0 1 1 1 1 1 s0 c0 s0 c0 s0 c0 s0 c0 s0 c0 s0 c0 s0 c0 s0 c0 s0 c0 1 1 1 1 1 1 1 1 1 1 1 SYMPTOMS Host s1 s0 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 1 s0c0p010 s0c0p010 1 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 s0c0p010 Service Offering Library of Generic Objects Problem Signatures - 21
EMC SMARTS TECHNOLOGIES: BEHAVIOR MODELS Logical Port y Problem Down Causes OperationallyDown,ConnectedPortDown Local symptom OperationallyDown Propagated symptom ConnectedPortDown to Connected Logical Ports Down y Physical Port – Problem Down Causes PortDown Propagated symptom PortDown To Logical Ports Layered over the Port are Down Propagated symptom ConnectedPortDown to Layered over Logical Ports y Card – Problem Down Causes PortDown Propagated symptom PortDown To Physical Ports in the Card are Down Propagated symptom ConnectedPortDown to Physical Ports in the Card y Switch – Problem Down Causes ConnectedPortDown Propagated symptom ConnectedPortDown to Cards in the Switch - 22
EMC SMARTS TECHNOLOGIES: SWITCH 0 CARD 0 SYMPTOMS SSwitch 00 PROBLEM Switch 1 L0 C 00 Card Port 0 L1 L0 L1 L0 L0 L1 L1 Port 1 C1 1 Card Port 0 pr o L1 bl sy em m s pt om s s0c0p0l0 s0c0p0l1 s0c0p1l0 s0c0p1l1 s0c1p0l0 s0c1p0l1 s0c1p1l0 s0c1p1l1 s1c0p0l0 s1c0p1l0 s1c1p0l0 s1c1p1l0 s3c0p0l1 s3c0p1l1 s3c1p0l1 s3c1p1l1 L0 Port 1 1 1 1 1 Port 0 L0 L1 L1 L0 L0 Port 0 1 1 1 1 Port 1 Port 1 Card 1 Port 0 Port 1 Switch 3 Port 0 L1 L0 1 1 Card 0 L1 L0 1 L0 1 1 L1 Port 1 1 SIGNATURE 1 1 L0 Port 0 L1 Card 0 L0 L1 Port 1 Card 1 L0 Port 0 L1 L1 L0 L0 L1 L1 - 23 Port 1
S Switch 00 PROBLEM Switch 1 L0 C 00 Card Port 0 L1 L0 L1 L0 L0 L1 L1 Port 1 C1 1 Card pr o Port 0 bl sy em m s pt om s S0 C0 s0c0p0l0 s0c0p0l1 s0c0p1l0 s0c0p1l1 s0c1p0l0 s0c1p0l1 s0c1p1l0 s0c1p1l1 s2c0p0l0 s2c0p1l0 s2c1p0l0 s2c1p1l0 s3c0p0l1 s3c0p1l1 s3c1p0l1 s3c1p1l1 1 1 1 1 L1 S0 C1 Port 1 Port 0 1 1 1 1 L0 Port 1 1 1 Port 0 1 1 1 1 Port 1 Card 0 Port 1 Card 1 Port 0 L1 L0 L0 L1 L1 L0 L0 Port 1 Switch 3 Port 0 L1 L1 L0 L0 L1 L1 Card 0 Port 1 Card 1 L0 1 1 L0 Port 0 Module 1 - Introduction to Smarts Technology EMC SMARTS TECHNOLOGIES: SWITCH 0 CARD 0 SYMPTOMS L0 Port 0 L1 L1 L0 L0 L1 L1 - 24 Port 1
EMC SMARTS TECHNOLOGIES: CODEBOOK & PROBLEM SIGNATURES SYMPTOMS PROBLEMS s0 s0c0p0l0 s0c0p0l1 s0c0p1l0 s0c0p1l1 s0c1p0l0 s0c1p0l1 s0c1p1l0 s0c1p1l1 s1c0p0l0 s1c0p0l1 s1c0p1l0 s1c0p1l1 s1c1p0l0 s1c1p0l1 s1c1p1l0 s1c1p1l1 s2c0p0l0 s2c0p0l1 s2c0p1l0 s2c0p1l1 s2c1p0l0 s2c1p0l1 s2c1p1l0 s2c1p1l1 s3c0p0l0 s3c0p0l1 s3c0p1l0 s3c0p1l1 s3c1p0l0 s3c1p0l1 s3c1p1l0 s3c1p1l1 Service Subscriber:: John s1 s2 s3 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 s0 c0 s0 c1 Symptoms Observed in Environment s1 s1 s2 s2 s3 s3 c0 c1 c0 c1 c0 c1 s0 s0 s0 s0 s1 s1 s1 s1 s2 s2 s2 s2 s3 s3 s3 s3 c0 c0 c1 c1 c0 c0 c1 c1 c0 c0 c1 c1 c0 c0 c1 c1 p0 p1 p0 p1 p0 p1 p0 p1 p0 p1 p0 p1 p0 p1 p0 p1 Subscribes 1 1 1 1 1 1 1 1 Service Offering::1 1 1 1 Secure Web 1 1 1 1 1 1 1 1 Service Offering:: 1 1 1 1 DNS 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 ServedBy 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Hosted App::URL1 1 1Hosted App::URL3 1 1 1 1 1 1 1 1 1 HostedBy 1 HostedBy HostedBy 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Host::WebServer 1 1 Host::WebServer 2 1 1 1 Host 1 1 1 1 1 1 1 1 1 Current Real Time Topology 1 1 1 1 1 1 1 1 1 - 25
Estimation Algorithm Diagnosis problem: Given binary vector of symptoms Y Find root cause (codebook column) k Solution: k arg min Y B k Bk is column k of the codebook y norm denotes Hamming distance y
UNFORTUNATELY, NOT THAT SIMPLE In a very simple case we can get away with binary symptoms In reality symptoms may manifest themselves with some certainty What’s worse, your data collection is not perfect Symptoms could be lost y Symptoms could be misdiagnosed y The actual analysis algorithm is using a combination of the codebook analysis and abductive inference
ANALYSIS METHODS Given: y y y y After numerous examples (evidences) of α and therefore β Induce R Deduction: Inference of β y y (α therefore β). Induction: Inference of R y Preconditions α Post conditions β and Rule R : α β Using R, and α Make a conclusion about β (α R β) Abduction: Inference of α y y Using the post condition β and the rule R Infer that the precondition α could explain β (β R α) 28
ABDUCTION VS. INDUCTION Induction Abduction Hypothesize general rule Hypothesize specific rule Drawn from many observations Drawn from a single observation 29
ABDUCTION VS. DEDUCTION Deduction Abduction The result is produced for a specific case The result is produced for a specific case If both rule and case are true then the result is true Even if both rule and result are true, the inferred specific case is only a possibility 30
ROOT CAUSE ANALYSIS IS ACTUALLY ABDUCTION R : α β (α therefore β) β are the symptoms α are the problems R is the causality model: which problems cause the manifestation of which symptoms 31
SIMPLE EXAMPLE: EVERY PROBLEM CAUSES ONE UNIQUE SYMPTOM When symptom βi is observed, it is clear that problem αi has occurred There is a one-one mapping between symptoms and problems Abduction Modeling α1 β1 α1 β1 α2 β2 α2 β2 α3 β3 α3 β3 32
ANOTHER SIMPLE EXAMPLE Every problem causes a unique set of symptoms α1 α2 α3 β1a β1b α1 β2a β2b β3a α2 α3 β1a β1b β2a β2b β3a Still, given a set of symptoms, it is clear which problem has occurred 33
LET’S COMPLICATE THINGS Some symptoms caused by more than one problem ? α 1 α1 α2 α3 βX ? α2 α3 βY βX βY α1 ? α1 ? ? α2 α3 βX βY α2 α3 βX βY 34
A SPECIFIC EXAMPLE – DIAGNOSING CHICKENPOX Chickenpox causes positive result of some blood test and also causes fever A Cold causes fever Mild chickenpox causes positive blood test but no fever Modeling MCP CP C BT F 35
DIAGNOSING CHICKENPOX IN A FLAWLESS ENVIRONMENT Case 1: Only fever Cold Case 2: Fever and blood test is positive Chickenpox Case 3: Only positive blood test Mild Chickpox MCP Case 4: No symptoms Healthy CP C BT F 36
ANOTHER COMPLICATION (OF THE MODEL) The blood test is not 100% reliable When you have chickenpox or mild chickenpox, there is a 50% chance that your blood test would turn out positive y When you don’t have chickenpox or mild chickenpox, there is a 2% chance that your blood test would turn out positive H y Remark BT and F are conditionally independent given CP Problem causes symptom with probability 2% MCP CP C BT F Problem causes symptom with probability 50% Problem causes symptom with probability 100%37
MORE INFORMATION Every disease has an a-priori probability At any given time, you have a chance of: 0.5% of having chickenpox y 1.5% of having mild chickenpox y 7% of having a cold y 91% of being healthy y Now apply the principle of parsimony in the abductive reasoning process Use Bayes law to calculate the likelihood of each explanation (probability of a disease given symptoms) 38
BAYES LAW APPLIED TO ABDUCTIVE REASONING Given a set of observed symptoms S, what is the probability that a given problem di has occurred ? Bayes law: P(di S) P(S di) x P(di) / P(S) The probability that a set of symptoms occurred knowing that the problem di occurred Probability of set of symptoms S. A priori probability of problem di 39
CHICKENPOX EXAMPLE P(d P(S di))xxP(d P(di S) P(di)i)/ /P(S) P(S) i S) P(S d i Case 2: Fever and blood test is positive P(CP F BT) P(F BT CP) x P(CP) / P(F BT) P(F CP) x P(BT CP) x P(CP) / P(F BT) 100% x 50% x 0.5% / P(F BT) H 0.25% / P(F BT) Similarly MCP P(MCP F BT) 0% / P(F BT) CP P(C F BT) 0.14% / P(F BT) P(H F BT) 0% / P F BT) C BT F 40
CHICKENPOX EXAMPLE (CONT.) Sum of probabilities is 1 : P(CP F BT) P(MCP F BT) P(C F BT) P(H F BT) 1 0.25% / P(F BT) 0.14% / P(F BT) 1 P(F BT) 0.39% P(C F BT) 0.14/0.39 36% P(CP F BT) 0.25/0.39 64% 41
DIAGNOSING CHICKENPOX AND COLD Case 1: fever, blood test negative y y Case 2: fever, blood test positive y y With probability 64% you have chickenpox With probability 36% you have a cold Case 3: no fever, blood test positive y y With probability 96.5% you have a cold With probability 3.5% you have chickenpox With probability 71% you are healthy With probability 29% you have mild chickenpox Case 4: no fever, blood test negative y y With probability 99% you are healthy With probability 1% you have mild chickenpox H MCP CP BT 42 C F
EMC SMARTS APPROACH Automated Incident Management—Start to Finish 4 Codebook Correlation 6 Root Cause Analysis Business Impact 1 ICIM Library 3 ICIM Repository Context 2 Discovery Collection 5 Polling/Pinging
ICIM: THE INCHARGE COMMON INFORMATION MODEL Based on DMTF CIM, extended with rich semantics for integrating and automating management applications Comprehensive y y y Within entities, across entities Models cross-domain relationships y y Service Subscriber ComposedOf Service Offering Models the complex web of relationships in the real world: y Models network, systems, applications, services, business entities 100 classes, 50 relationships Infinitely extensible via inheritance Subscribes Key to service management, end-to-end view Pieces together information from heterogeneous sources Abstract y y To scale to networks of unlimited size and complexity through multiple levels of abstraction To decouple management application logic from the specifics of an ever-increasing stream of vendor products Efficient Open Hosted Application Neighboring Systems HostedBy Neighboring Systems Host Router Switch
ICIM: THE INCHARGE COMMON INFORMATION MODEL Attributes y Stored or instrumented – this is transparent to applications Relationships y 1-1, 1-many, many-1, many-many Behaviors y Specific extensions for each type of FCAPS application, e.g., problems for fault, constraints for configuration Constraints y Assure consistency of assigned values, e.g., speed match at both end of a circuit MODEL: Managed Object DEfinition Language y Based on CORBA IDL syntax Relationship Types ConnectedVia/ConnectedTo ParentOf/ChildOf LayeredOver/Underlying SwappedFrom/SwappedTo NextHop/PreviousHop ImportedBy/Import IP Technology Objects Switches Host ATM Technology Objects MAC Addresses ATM Switches IP Networks/Addresses Edge Routers Application/Business/Service Cards CardsObjects Chassis Chassis Hosts Protocol Technology Objects Interfaces Interfaces Applications MPLS Ports Physical Ports Application Clusters P Routers Servers Logical Ports Sessions PE Routers Cables Logical Trunks Transactions CE Routers Trunks Cables Customers LSP Fans Physical Trunks Business LSPUnits Hops Power Supplies Logical Trunks Services VPNs PVCs VRFs Many Others Routing BGP Services/Sessions Route Reflectors (RR) Route Reflector Clusters OSPF Services/Sessions OSPF Areas OSPF members
INCHARGE COMMON INFORMATION MODEL Multi-Layered Relationships – Intra-Domain Relationships – Inter-Domain Relationships – Business Relationships Subnet Y Subnet Y Subnet X Subnet X TCP Connections, Sessions, Transactions Business Relationships Subnet Z Subnet Z L1/L2 - Physical Connectivity Subnet A Subnet A L2 - Logical Connectivity (e.g. VLANS) L3 - Logical IP-Subnet Connectivity - MPLS LSP/LSP Hop/VFR/VPN - OSPF/BGP Areas/Adjacencies
THANK YOU! Questions, Comments, Suggestions? And we always in search of good people! Yuri.Rabover@VMTurbo.com
ROOT CAUSE ANALYSIS - PROBLEM DEFINITION Wikipedia: yRoot cause analysis (RCA) is a class of problem solving methods aimed at identifying the root causes of problems or events. Smarts: yGiven a set of symptoms, find the problem, or the set of problems, that best explains those symptoms. Smarts: Given a set of symptoms, find the set
USING SAP ROOT CAUSE ANALYSIS & SYSTEM MONITORING FOR SYBASE UNWIRED PLATFORM 6 2. ROOT CAUSE ANALYSIS FOR SUP IN SOLUTION MANAGER After SMD Managed System Setup and Configuration, the Root Cause Analysis features of SAP Solution Manager Diagnostics are available in the Root Cause Analysis work center of SAP Solution Manager. Find further information about End-to-End Root Cause Analysis on SAP .
"Fishbone" Diagram: Measures Top Primary Root-Cause Primary Root-Cause Second level Root-Cause Third level Root-Cause Fourth level Root-Cause Measures Education & Training To Recognize Fatigue Failure Of IRS Fatigue Management Systems Political Will Regulation & Policy Under-Reporting Hours Of Service (HOS) Recording Device
WHAT IS ROOT CAUSE ANALYSIS? 2 Root cause analysis (RCA), is a structural step by step technique that focuses on finding the real cause of a problem and deals with it. Root Cause Analysis is a procedure for ascertaining and analyzing the cause of problems, to determine how these problems can be solved or be prevented from occurring. 8.6.2014
IL PB BK DI8 DO4/EF-PAC 2-2 PHOENIX CONTACT 7725_en_01 2.1.1 Diagnostics in the IL PB BK DI8 DO4 format This diagnostic format consists of the following blocks: 1 PROFIBUS standard diagnostics 2 ID-specific diagnostics 3 Status diagnostics (terminal status) 4 Channel-specific diagnostics 5 Revision diagnostics (manufacturer-specific)
7 [ROOT likes] [.] A [ OBJ(likes,books) Right Arc(OBJ) 8 [ROOT likes .] ; Shift 9 [ROOT likes] ; A [ P(likes,.) Right Arc(P) 10 [ROOT] ; A [ ROOT(ROOT,likes) Right Arc(ROOT) Figure 2: An example of arc-standard transition dependency parsing. s3 s2 s1 b1 b2 b3 s2.lc1 s2.lc2 s2.rc2 s2.rc1 s2.rc2.lc1 2 rc2
7.5 Graphing Square Root and Cube Root Functions 431 Graph square root and cube root functions. Use square root and cube root functions to find real-life quantities, such as the power of a race car in Ex. 48.
under VMX non Root Mode, CPU stops execution of VMX non Root Mode, exit to VMX Root Mode. Then it trapped by hypervisor, hypervisor emulates the instruction which guest tried to execute. Mode change from VMX Root Mode to VMX non-root Mode called VMEntry, from VMX non-root Mode to VMX Root Mode called VMExit(Figure 2). User (Ring 3) Kernel (Ring .
Auditing and Assurance Services, 14e (Arens) Chapter 10 Section 404 Audits of Internal Control and Control Risk Learning Objective 10-1 1) Which of the following is not one of the three primary objectives of effective internal control? A) reliability of financial reporting B) efficiency and effectiveness of operations C) compliance with laws and regulations D) assurance of elimination of .
