How To Measure Anything In Cybersecurity Risk
How to Measure Anything inCybersecurity RiskPresented by:Douglas HubbardHubbard Decision Research
My Co-Author and IRichard SeiersenCurrently the General Manager of Cybersecurity and Privacy at GE HealthCare. Data driven executive with 20 years experience spanning subjectmatters in Cyber Security, Quantitative Risk Management, PredictiveAnalytics, Big Data and Data Science, Enterprise Integrations andGovernance Risk and Compliance (GRC). Led large enterprise teams,provided leadership in multinational organizations and tier one venturecapital backed start-ups.Douglas HubbardMr. Hubbard is the inventor of the powerful Applied Information Economics(AIE) method. He is the author of the #1 bestseller in Amazon’s math forbusiness category for his book titled How to Measure Anything: Findingthe Value of Intangibles in Business (Wiley, 2007; 3rd edition 2014). Hisother two books are titled The Failure of Risk Management: Why It’sBroken and How to Fix It (Wiley, 2009) and Pulse: The New Science ofHarnessing Internet Buzz to Track Threats and Opportunities (Wiley, 2011).2
The Biggest Cybersecurity RiskQuestion: What is Your Single Biggest Risk in Cybersecurity?Answer: How You Measure Cybersecurity Risk3
Current Solution Here are some risks plotted on a“typical heat map”. Suppose mitigation costs were:o Risk 1: 725K – Higho Risk 2: 95K – Lowo Risk 3: 2.5M – Criticalo Risk 5: 375K – Moderate What mitigations should befunded and what is the priorityamong these?4
Current SolutionsMost standards and certification tests promote risk analysis as a type of ordinal scoringmethodThe “Risk Rating Methodology” on OWASP.org states: “Once the tester has identified a potential risk and wants to figure out how serious it is, thefirst step is to estimate the "likelihood". At the highest level, this is a rough measure ofhow likely this particular vulnerability is to be uncovered and exploited by an attacker. It isnot necessary to be over-precise in this estimate. Generally, identifying whether thelikelihood is low, medium, or high is sufficient .”5
Can Analysis or Expertise be a“Placebo”?“The first principle is that you must not fool yourself, and youare the easiest person to fool.” — Richard P. Feynman Collecting more than a few data points on horses makes experts worse at estimatingoutcomes. (Tsai, Klayman, Hastie) Interaction with others only improves estimates up to a point, then they get worse. (Heath,Gonzalez) Collecting more data about investments makes people worse at investing. Collecting moredata about students makes counselors worse at predicting student performance.(Andreassen) An experiment with a structured decision analysis method shows confidence increasedwhether decisions are improved or degraded. (Williams, Dennis, Stam, Aronson)In short, we should assume increased confidence from analysis is a “placebo.” Realbenefits have to be measured.6
What the Research Says There is mounting evidence against (and none for) the effectiveness of“risk scores” and “risk matrices.” Fundamental misconceptions about statistical inference may keep somefrom adopting quantitative methods. Experts using even naïve statistical models outperform human expertswho do not.Note: Every improvement we are about to has already been adopted inseveral cybersecurity environments.7
Summarizing Research on OrdinalScales Bickel et al. “The Risk of Using Risk Matrices”, Society ofPetroleum Engineers, 2014 They performed an extensive literature review to-date as well as astatistical analysis of RM used in Petroleum Engineering Risk(which are nearly identical to RM’s in Cyber) – including computinga “Lie Factor” of the degree of distortion of data.“How can it be argued that a method that distorts the information underlying anengineering decision in nonuniform and uncontrolled ways is an industry bestpractice? The burden of proof is squarely on the shoulders of those who wouldrecommend the use of such methods to prove that these obvious inconsistencies donot impair decision making, much less improve it, as is often claimed.”8
What if We Could Actually MeasureRisk in Cybersecurity?What if we could measure risk morelike an actuary – “The probability oflosing more than 10 million due tosecurity incidents in 2016 is 16%”What if we could prioritize securityinvestments based on a “Return onMitigation”?DB AccessPhysical AccessData in TransitNetwork Access ControlFile AccessWeb VulnerabilitiesSystem ConfigurationExpectedLoss/Yr 24.7M 2.5M 2.3M 2.3M 969K 409K 113KCost ofControl 800K 300K 600K 400K 600K 800K 500KControlReturn teMitigateMonitorTrackTrackThis means there is about a 40% chance oflosing more than 10M in a year and abouta 10% chance of losing more than 200M.9
Why Not Better Methods? Cybersecurity is too complex or lacks sufficient data for quantitativeanalysis yet can be analyzed with unaided expert intuition or soft scales. Probabilities can’t be used explicitly because . yet we can imply probabilities with ambiguous labels.Remember, softer methods never alleviate a lack of data, complexity, rapidlychanging environments or unpredictable human actors they can only obscure it.10
A Major Fallacy RegardingComparing Methods Don’t make the classic “Beat theBear” fallacy.Exsupero Ursus If you doubt the effectiveness of quantitative methods, remember, all youhave to do is outperform the alternative: unaided expertise or soft scoring methods.11
Your Intuition About SampleInformation is Wrong Cybersecurity experts are not immune to widely held misconceptions about probabilitiesand statistics – especially if they vaguely remember some college stats. These misconceptions lead many experts to believe they lack data for assessinguncertainties or they need some ideal amount before anything can be inferred.“Our thesis is that people have strongintuitions about randomsampling these intuitions are wrong infundamental respects.[and] are sharedby naive subjects and by trainedscientists”Amos Tversky and Daniel Kahneman,Psychological Bulletin, 197112
You Need Less Data Than You Think A beta distribution computes the probability of a frequency being below a given amount (e.g. chancethat rate of occurrence is 2/100) In Excel it can be written as “ Betadist(frequency,alpha,beta)” A uniform prior can be made with alpha 1 and beta 1. This can be used as a starting point formaximum uncertainty. “Hits” and “Misses” can be simply added to the priors ( Betadist(frequency,hits 1,misses 1))2 events in 7 companies over 5 years (35 company –years) of data: 2 hits, 33 misses00.050.10.150.20.250.30.350.40.450.513
Survey Results: The “StatsConcepts” Quiz We conducted a survey of 171 Cybersecurity professionals One Finding: Strong opinions against “quant” are associated with poor statsunderstanding.“It’s not what youdon’t know thatwill hurt you, it’swhat you knowthat ain’t so.”MarkTwain14
Historical Models – Still BetterThan ExpertsWhen experts assess probabilities, many events “. . .are perceived as so unique that past history does notseem relevant to the evaluation of their likelihood.” Tversky, Kahneman, Cognitive Psychology (1973)Yet, Historical models routinely outperform experts in a variety of fields (even considering “Black Swans”)Paul Meehl assessed 150 studies comparingexperts to statistical models in many fields (sports,prognosis of liver disease, etc.).“There is no controversy in social sciencewhich shows such a large body of qualitativelydiverse studies coming out so uniformly in thesame direction as this one.”Philip Tetlock tracked a total of over 82,000forecasts from 284 political experts in a 20 yearstudy covering elections, policy effects, wars, theeconomy and more.“It is impossible to find any domain in whichhumans clearly outperformed crudeextrapolation algorithms, less stillsophisticated statistical ones.”15
Monte Carlo: How to ModelUncertainty in DecisionsOutageFrequencyand DurationFrequencyandmagnitudeof breaches 20 25 30 35 404%5% 6% 7% 8%Cost PerOutage hourLegalLiabilities( MM)10% 15% 20% 15% 30%Losses 30 40 50 60 70? Simple decomposition greatly reduces estimation errorfor estimating the most uncertain variables (MacGregor,Armstrong, 1994) As Kahneman, Tversky and others have shown, wehave a hard time doing probability math in our heads In the oil industry there is a correlation between the useof quantitative risk analysis methods and financialperformance – and the improvement started after usingthe quantitative methods. (F. Macmillan, 2000) Data at NASA from over 100 space missions showedthat Monte Carlo simulations beat other methods forestimating cost, schedule and risks (I published this inThe Failure of Risk Management and OR/MS Today). 1M 2M 3M 4M 5M16
A Simple “One-For-One Substitution”Each “Dot” on a riskmatrix can be betterrepresented as a rowon a table like thisThe output can thenbe represented as aLoss ExceedanceCurve.17
Loss Exceedance Curves: Beforeand After How do we show the risk exposure after applying available mitigations?Chance of Loss or GreaterRisk ToleranceStochasticDominanceInherent RiskResidual RiskGiven Loss or Greater (Millions)18
Overconfidence “Overconfident professionals sincerelybelieve they have expertise, act as expertsand look like experts. You will have tostruggle to remind yourself that they may bein the grip of an illusion.” Daniel Kahneman, Psychologist, EconomicsNobel Decades of studies show that most managers are statistically “overconfident” when assessing theirown uncertainty. Studies also show that measuring your own uncertainty about a quantity is a general skill that can betaught with a measurable improvement Training can “calibrate” people so that of all the times they say they are 90% confident, they will beright 90% of the time.19
Inconsistency vs. Discrimination Discrimination is how much your estimates vary whengiven different information. Inconsistency is the amount of your discrimination thatis due to random differences in estimates - this maybe in addition to differences in interpreting verbalscales, so let’s assume we are using explicitprobabilities. Experts are routinely influenced by irrelevant, externalfactors - anchoring, for example, is the tendency for anestimator to be influenced by recent exposure to ananother unrelated number (Kahneman).20
Inconsistency Measurement Results Total: Over 30,000 individual estimates ofprobabilities These estimates included over 2,000 duplicatescenarios pairs.Judgment 2 We have gathered estimates of probabilities ofvarious security events from:o 48 experts from 4 different industries.o Each expert was given descriptive data forover 100 systems.o For each system each expert estimatedprobabilities of six or more different types ofsecurity events.Comparison of 1st to 2nd Estimates of Cyberrisk judgements by same SME1.00.90.80.70.60.50.40.30.20.10.00.0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0Judgment 121% of variation in expert responses areexplained by inconsistency.(79% are explained by the actualinformation they were given)21
Modeling Group Estimates of ITSecurity Event Likelihood Examples of Models vs. Group Averages: Probabilities of different security events happening in the next12 months for various systems prior to applying particular controls.Internal Unauthorized AccessResulting In Productivity LossConfidentiality BreachResulting In 0.500.1Model Estimate0.20.30.40.5Model Estimate The models created produce results which closely match the group’s average. A large portion of the model error is due to judge inconsistency. This nearly eliminates the inconsistency error.22
Effects of Removing Inconsistency Alone A method of improving expertestimates of various quantitieswas developed in the 1950’s byEgon Brunswik. He called it the “Lens Method” It has been applied to severaltypes of problems, includingexpert systems, withconsistently beneficial results.23
Measurement Challenge:Reputation Damage One of the perceived most difficultmeasurements in cybersecurity is damage toreputation.eBay Trick: There is no such thing as a “secret”damage to reputation! How about comparing stock prices afterincidents? (That’s all public!)Home Depot So what is the REAL damage?o Legal liabilities,o Customer outreacho “Penance” projects (security overkill) The upshot, damage to reputation actuallyhas available information and easilyobservable measured costs incurred to avoidthe bigger damages!Target201120122013201424
Supporting Decisions If risks and mitigation strategies were quantified in a meaningful way, decisions could besupported. In order to compute an ROI on mitigation decisions, we need to quantify likelihood, monetaryimpact, cost, and effectivenessLikelihood / YrImpact / YrMitigationEffectivenessRisk 137% 2M to 40M95% 725K725%MitigateRisk 211% 50K to 400K100% 95K-80%TrackRisk 334% 5M to 80M90% 2.5M329%MonitorRisk 429% 500K to 20M98% 375K437%MitigateRiskMitigation Cost/ YrMitigation ROIAction The optimal solution would be to mitigate Risks1 & 4 first. If you have the resources, then mitigate Risk 3. Risk 2 is not worth fixing.25
Call to Action for Cybersecurity Organizations should stop using risk scores and risk matrixes andstandards organizations should stop promoting them Adopt simple probabilistic methods now: They demonstrate a measurableimprovement over unaided intuition and they have already been used. Sothere is no reason not to adopt them. Build on simple methods when you are ready – always based on whatshows a measurable improvement.26
Supplementary Material27
Parameters Cybersecurity Models Experts are given values on avariety of parameters as a basis fortheir estimates. For each scenario they may beasked to estimate a probability of abreach, outage, legal liability, etc. Some companies estimated risks ofincidence for particular systems,others estimated threats oradditional detail for types of losses,but there were some commonthemes (see table).28
“Opinion Toward QuantitativeMethods” (18 Questions)18 questions on opinions of the use of quantitative methods in cybersecurity were asked.Here are some examples:(Responses: Agree, Disagree, No Opinion/Don’t Know)Information security is too complex to model with probabilistic methods.Management and users won't understand the quantitative methods’ output.An expert using quantitative probabilistic methods will do better risk assessments then an expertusing intuition alone.RESULTS: 80% of respondents had more “pro” than “anti” quantitative responses. Only 22%were consistently “pro” on quantitative and “anti” on softer scoring methods.29
The Stats Concepts Quiz (10 Questions)EXAMPLE: Assume that you have a portfolio of systems for which you haveobserved no security events in the past year that resulted in a monetary orproductivity loss, which of the following statements is true?Answer Optionsra!rrrResponsePercentIf no events were observed, then we have no data about the likelihood of theseevents.2.2%The fact that no events were observed tells us something about the likelihood ofthese events.37.0%One year is not long enough time to gather enough observations to make aninference.4.4%Since some events may not have been observed, the lack of observed losses tells usnothing.There is insufficient information to answer the question.I don't know31.9%17.8%6.7%30
Bayesian Methods: NodeProbability Tables Conditional probabilitieswith combinations ofconditions are recordedwith an NPT With more than a fewconditions and conditionsthat are more than binary,it will become unwieldly (Recent models we createdwould have had thousandsof rows)31
Rasch (Logodds) Model A Rasch Model is a relatively simple approximation to “add up” a number of parameters that modify aprobability when NPTs would be large. Logodds of X LO(X) ln(P(X)/(1-P(X)) Adjustment due to condition Y A(Y) LO(P(X Y))– LO(P(X)) P(X A,B,.) A(Sum of (LO(A),LO(B), ) LO(P(X))) The more independent the parameter are, the better the Rasch approximation.Initial Prob: P(E)Baseline Logodds10%-2.197ConditionsP(E X)P(E X)P(X)Test P( E )Logodds change XLogodds change D12.0%8.0%50.0%10.0%0.2048-2.442332
Beta Distribution and the LEC Consider a portfolio of systems, each with chance of monetary loss event and a range ofloss amount if it occurs. If we consider the possibility of systemic under/overestimation of P(Event), the LEC isrotated so that expected loss is constant but extreme loss.es are more likely33
Effects of Ordinal Scales andMatrices of Ordinal Scales Bob Clemen and Craig Fox paper on ordinal scales for general decision analysis, ManagementScience, 2004 “Analysts typically assume that the particular choice of intervals does not unduly influenceassessed probabilities. Unfortunately, our experimental results demonstrate that thisassumption is unfounded: assessed probabilities can vary substantially with the particularpartition that the analyst chooses.” Tony Cox “What’s wrong with Risk Matrices” investigates various mathematical consequencesof ordinal scales on a matrix. “Risk matrices can mistakenly assign higher qualitative ratings to quantitatively smallerrisks. For risks with negatively correlated frequencies and severities, they can be “worsethan useless,” leading to worse-than-random decisions.”34
The “Illusion of Communication” Budescu et. al. Psychological Science, 2009 on the use of verbal, qualitative scales for likelihoods:“[Verbal terms] induce an illusion of communication People assume that everyone interprets theterms consistently and similarly, and fail to appreciate the variance in the interpretations of thesewords. The high level of potential miscommunication has been widely documented in manycontexts.”35
The State of CybersecurityJames B. Comey, Director FBI, made the following statementbefore the Senate Committee on Homeland Security andGovernment Affairs on Nov 14, 2013:“The diverse threats we face are increasingly cyber-based. Much ofAmerica’s most sensitive data is stored on computers. We are losingdata, money, and ideas through cyber intrusions. This threatensinnovation and, as citizens, we are also increasingly vulnerable to losingour personal information. That is why we anticipate that in the future,resources devoted to cyber-based threats will equal or even eclipse theresources devoted to non-cyber based terrorist threats.”.[FBI]36
Methods of Measurement Most real-world scientific measurements are based on random samples of somekind. There are a variety of methods for many situations but they all come down to onesimple idea: What is being measured has some effect on the likelihood of aparticular observation. This means that you don’t have to: Count everything Eliminate or even know all sources of error “Exception anxiety” is where you think of a possible source of error and assumethis means the measurement tells you nothing. This is itself a hypothesis thatrequires evidence. It assumes a measured quantity of error smothers the“signal.”37
Practical Assumptions Its been measured beforeYou have more data than you thinkYou need less data than you think“It’s amazing what you can see when you look”Yogi Berra38
Common MisconceptionsCorrected by Bayes “A positive result on a test can tell us something but a negative result tellsus nothing” “If nothing occurred, I have no data about the rate of occurrence” OR“Absence of evidence is not evidence of absence” “A few data points tell us nothing” OR “We need more data to bestatistically significant”For each of 10 systems, you estimate an 8% chance per year of a loss dueto integrity breaches. T
matters in Cyber Security, Quantitative Risk Management, Predictive Analytics, Big Data and Data Science, Enterprise Integrations and Governance Risk and Compliance (GRC). Led large enterprise teams, provided leadership in multinational organizations and tier one venture capital backed start-ups. Douglas Hubbard
without a remainder and is not attached to anything. “One who is not attached to anything does not cling to anything in the whole world. One who does not cling to anything does not seek for anything. One who does not seek for anything personally realizes Nirvāṇa, [knowing]: ‘Birth f
work/products (Beading, Candles, Carving, Food Products, Soap, Weaving, etc.) ⃝I understand that if my work contains Indigenous visual representation that it is a reflection of the Indigenous culture of my native region. ⃝To the best of my knowledge, my work/products fall within Craft Council standards and expectations with respect to
List of core vocabulary Cambridge Pre-U Mandarin Chinese 9778: List of core vocabulary. 5 3 Measure words 把 bǎ measure word, hold 杯 bēi a cup of / cup 本 běn measure word 遍 biàn number of times 层 céng layer; storey 次 cì number of times 段 duàn paragraph, section 队 duì team 封 fēng measure word 个 gè measure word 壶 hú measure word 件 jiàn measure word
JOSEPH OF ARIMATHEA "Do not believe in anything simply because you have heard it. Do not believe in anything simply because it is spoken and rumored by many. Do not believe in anything simply because it is found written in your religious books. Do not believe in anything merely on the authority of your teachers and elders.
MEASURE WIDTH MEASURE HEIGHT 3 2 1 6 5 4 L-FRAME MOLDING How to Measure 1. Use a steel tape measure only. 2. Do not make any deductions, allowances, or additions to your measurements. 3. Measure to the nearest 1 16". 4. Measure window width in three places from the outside edge of the mol
Stravinsky, Rite of Spring o Part 1: Beginning to rehearsal 4 o Part 1: First 4 measures of rehearsal 12 o Part 1: Rehearsal 49 to 4th measure after rehearsal 53 Mozart, Requiem o I. Beginning to end of measure 7 o I. Measure 20-21 o I. measure 32 to downbeat 40 o II. Opening to measure 10 o III. Pickup to measure 53 to end
Inscribed Angle Theorem The measure of an inscribed angle is _ _ the measure of its intercepted arc. In other words, the intercepted arc is _ the measure of the inscribed angle. Guided Examples: Find the measure of the missing variable. 1. 2. 3. Find the measure of arc XZ 4. Find the measure of angle LPN. Theorem
This measure may be reported by clinicians who perform the quality actions described in the measure based on the services provided and the measure-specific denominator coding. Measure Reporting via Claims: CPT or HCPCS codes and patient demographics are used to identify patients who are included in the measure's denominator.
