Practical SOC Metrics - FireEye
Practical SOC MetricsPresented by Carson Zimmerman
About Carson§ Worked in Security Operationsfor 15 years§ SOC Engineering Team Lead @Microsoft§ Previously SOC engineer, analyst& consultant @ MITRE§ Check out my book if youhaven’t nscenter2 2019 2019 CarsonCarson ZimmermanZimmerman
About Chris§ Independent Consultant(Montance.com)§ SANS Institute– Senior Instructor & Course Author– SOC Survey Author (2017, 2018, 2019)– Security Operations Summit Chair§ SOC-class.com – Security OperationsClass on building & running a SOC§ Engagements with Defense, Education,Energy, Financial, IT, Manufacturing,Science, Software Development, 3 2019 2019 CarsonCarson ZimmermanZimmerman
Pick Something You Love 4http://disney.wikia.com/wiki/File:TS2 Jessie hugs Woody.jpg 2019 2019 CarsonCarson ZimmermanZimmerman
And Measure It5https://en.wikipedia.org/wiki/Tape measure#/media/File:Measuring-tape.jpg 2019 2019 CarsonCarson ZimmermanZimmerman
Measuring Things Usually Drives ChangeEven if you’re not at CMM level 3, you can still get started!OptimizingMeasuredDefinedManagedInitial6 2019 2019 CarsonCarson ZimmermanZimmerman
Metrics are Like ghtsaber-Green-Science-Fiction-Space-16752117 2019 2019 CarsonCarson ZimmermanZimmerman
They Can Be Used for Good enes-we-want-to-see-on-blu-ray/8 2019 2019 CarsonCarson ZimmermanZimmerman
And for edClan-RotS.jpg9 2019 2019 CarsonCarson ZimmermanZimmerman
Top Tips§ Metric data should be free and easy to calculate– ½ of all SOCs collect metrics according to SANS SOC survey 2017 & 2018§ There should be a quality measure that compensates for perversion– Especially when there’s a time based metric!§ Metrics aren’t (necessarily) Service Level Objectives (SLOs)– The metric is there to help screen, diagnose, and assess performance– Don’t fall into a trap of working to some perceived metric objective– Any metric should have an intended effect, and realize the measurement andcalculation isn’t always entirely valid§ Expectations, messaging, objectives- all distinct!11 2019 2019 CarsonCarson ZimmermanZimmerman
Data Sources§ SOC Ticketing/case managementsystem§ SIEM / analytic platform / EDRanywhere analysts create detections,investigate alerts§ SOC code repository§ SOC budget– CAPEX including hardware & software– OPEX including people & cloud§ Enterprise asset management systems§ Vulnerability ges-159825349.jpeg12 2019 2019 CarsonCarson ZimmermanZimmerman
Existing nual-threatreport/mtrends.html§ SOC CMM: measure your SOCtop to bottom§ VERIS Framework: track yourincidents well§ SANS SOC Survey: recent pollsfrom your rts/dbir/13 2019 2019 CarsonCarson ZimmermanZimmerman
Example Metrics
Metric Focus 1: Data Feed Health§ Is it “green”§ What is green anyway?§ Just because it’s up doesn’tmean all is well– Delays in receipt– Drops§ Temporary§ Permanent– ia/File:Watermelon cross BNC.jpg15 2019 2019 CarsonCarson ZimmermanZimmerman
5 Minutes’ of Work: Which Sensors are Down17 2019 2019 CarsonCarson ZimmermanZimmerman
15 Minutes’ More Work: AutomatedDetection of Downed FeedsCollector ACollector BCollector CCollector DCollector EOLD COUNT2230120332031120342NEW COUNT OLD DEVICES NEW 502496IS BROKENNoYesNoYesYes§ Automate detection of dead, slow or lagging collectors§ Query for old data (1-7 days ago) vs recent data (last 24 hours)§ Look for major dips or drops: done through query logic§ Consider human eyes on: daily or weekly19 2019 2019 CarsonCarson ZimmermanZimmerman
Metric Focus 2: CoverageDimensions:1. Absolute number andpercentage of coverage percomputeenvironment/enclave/domain2. Kill chain or ATT&CK cell3. Layer of the compute stack(network, OS, application,etc.)4. Device covered (Linux,Windows, IoT, network device)21 2019 2019 CarsonCarson ZimmermanZimmermanTips:1. Never drive coverage to 100%§You don’t know what you don’tknow§Always a moving target2. There is always anotherenvironment to cover,customer to serve3. There will always be morestones to turn over; don’tignore any of these dimensions
Managed vs Wilderness§ Percentage of systems “managed”:– Inventoried?– Tied to an asset/business owner?– Tied to a known business/mission function?– Subject to configuration management?– Assigned to a responsible security team/POC?– Risk assessed?§ If all are yes: it’s managed§ If not: it’s “wilderness”§ SOC observed device counts help identify“unknown unknowns” in the wilderness22 2019 2019 CarsonCarson ZimmermanZimmerman
Monitoring SLAs/SLOs§ SLA: Agreement monetary (orother penalty) for failing to meet§ SLO: Objective no specificpenalty agreed to for failing tomeet§ Institution & missions specificwhere these need to be set inplace§ Don’t monitor everything thesame way!– Instrumentation, customdetections, response times,retention24 2019 2019 CarsonCarson ZimmermanZimmermanBasic Service§ Host EDR§ Network logs§ Standard mix of detections§ Yearly engagementAdvanced Service§ Basic, plus:§ 3 application logs§ 1 focused detection/quarter§ Quarterly engagement
Metric Focus 3: Scanning and SweepingBasic§ # % of known on prem & cloudAdvanced§ Time to sweep and compileassets scanned for vulnsresults for a given vuln or IOC:§ Amount of time it took to§ A given domain/forest identitycompile vulnerability/risk statuson covered assets during lasthigh CVSS score “fire drill”§ Number of people needed tomassage & compile thesenumbers monthly25 2019 2019 CarsonCarson ZimmermanZimmermanplane§ Everything Internet-facing§ All user desktop/laptops§ Everything§ # % of assets you can’t/don’tcover (IoT, network devices,etc.)
Metric Focus 4: Your AnalyticsBasics:Advanced:NameDescriptionKill chain mappingATT&CK cell mappingDepends on which datatype(s) (OS logs, Netflow, etc.)6. Covers whichenvironments/enclave7. Created- who, when8. Runs in what framework1.2.3.4.5.26 2019 2019 CarsonCarson ZimmermanZimmerman(Streaming, batched query,etc.)9. Last modified- who, when10. Last reviewed- who, when11. Status- dev, preprod, prod,decom12. Output routes to (analysttriage, automated notification,etc.)
Measure Analyst ProductivityIs this goodor evil?Can this begamed?Analytics Status for Last Month6050403020100AliceDev27 2019 2019 CarsonCarson yDecom
How Fruitful are Each Author’s Detections?Alert Final Disposition by Detection Author§ # of times adetection oranalytic fired,attributed to thedetectionauthor§ Is this evil?§ How can this begamed?6050403020100AliceBobQuick F by Tier 1True 28 2019 2019 CarsonCarson ZimmermanZimmermanCharlieTrudyQuick F by Tier 2MalloryGarnered Further work
How are You Supporting Your Customers?Analytic 9 2019 2019 CarsonCarson stallMarketingC&CVIPGeneralActions
Map Your Analytics to ATT&CKProps to MITREfor the greatexampleMany places todo this consider anystructuredcode repo orwiki30 2019 2019 CarsonCarson ZimmermanZimmermanhttps://car.mitre.org
Metric Focus 5: Analyst Performance1.2.3.4.5.6.7.8.NameJoin dateCurrent role & time in roleNumber of alerts triaged in last30 days% true positive rate forescalations% response rate for customerescalationsNumber of escalated caseshandled in last 30 daysMean time to close a case31 2019 2019 CarsonCarson ZimmermanZimmerman1. Number of2.3.4.5.6.analytics/detections createdthat are currently in productionNumber of detectionsmodified that are currently inproductionTotal lines committed to SOCcode repo in last 90 daysSuccess/fail rate of queriesexecuted in last 30 daysMedian run time per queryMean lexical/structuralsimilarity in queries run
Tier 1 InputsDaily Review Dashboard2015Top firing detections10518-319: Hacking tool used by18-317: AV hit on carsonzcrowley 2019 Carson Zimmermanwork host 2019 Carson Zimmerman18-367: RDCsession from salesto DC 1Everything else18-386: interactivelogin in DC host 218-410: IoC inmarketing18-326:suspicioussession tohost18329TipsfromhuntTipsfromIntelnter DataCenter CeatagrketinSalesDOMarati onsineer inEng18-384: IoC hit inegineering10s ofalerts80706050403020100gTop time spent per caseEmailAlert DispositionpeDetection 56: lowentropy on 443WebsitenceDetection 23:downrev AVDetection34:VPNtimetravelPhonecallsaDetection 22:AVdeactivatedDetection 87:highentropyon 800FinDetection 21: IoC file hash matchDetection 76: Elephantflow on weird portDetection 33:downrev useragent stringDetection 64:SQLinjectionDetection 34:SSLbadciphersuiteQuic k F by T1Quic k F by T2True Garnered Further WorkAuto RemediatedAuto notified
Metric Focus 6: Incident Handling§ Mean/median adversary dwelltime§ Mean and median time to – Triage & Escalate– Identify– Contain– Eradicate & recover§ Divergence from SLA/SLO?§ Insufficient eradication?§ Threat attributed?34 2019 2019 CarsonCarson ZimmermanZimmerman§ Top sources of confirmedincidents§ Proactive? Reactive?§ User reports? SOC monitoring?Data & ”anecdata”: unforcederrors and impediments§ Time waiting on other teams todo things§ No data/bad data/ data lost§ Incorrect/ambiguousconclusions§ Time spent arguing
Typical Incident Metrics§ More ideas:§ Mean/median time to respond§ Cases left open time threshold§ Cases left open by initialIncidents: Last 6 Months250200reporting/detection type§ Stacked bar chart by case type150100500122369January February MarchOpen Cases35 2019 2019 CarsonCarson ZimmermanZimmermanClosed Cases8April7MayJuneEscalated to 3rd party
Incident Avoidability§ Most incidents are avoidable everyone realizes this– Collect metrics on how avoidable, what could have been done to prevent§ Crowley’s Incident Avoidability metric– A measure, already available in the environment, is applied to othersystems/networks, but wasn’t applied - resulting in the incident– A measure is available (generally) and something (economic, political) preventsimplementing it within the organization– Nothing is available to prevent that method of attack§ Attribution for measure/mechanism in 1 & 2 is critical38 2019 2019 CarsonCarson ZimmermanZimmerman
Metric Focus 7: Top Risk Areas & Hygiene§ Make vulnerability managementdata available to customers– Self service model– Scan results down to asset & itemscanned§ But don’t beat them over thehead with every measure!– Pick classic ones they will always bemeasured on– Scanning, monitoring, patching41 2019 2019 CarsonCarson ZimmermanZimmerman§ Pick top risk items from ownincident avoidability metrics andpublic intel reporting to focus oneach year, semester, or quarter– Internet-exposed devices– Code signing enforcement– EDR deployment– Single factor auth– Non-managed devices & cloudresources
Conclusion
Closing§ Whatever you do, measuresomething– Include both internal and externalmeasures– Behaviors and outcomes!§ You can do it, regardless of howmature, old, or big your SOC is§ Pick your investments carefully§ Iterate an-do-it-1813446 2019 2019 CarsonCarson ZimmermanZimmerman
Questions
§ SOC Ticketing/case management system § SIEM / analytic platform / EDR-anywhere analysts create detections, investigate alerts § SOC code repository § SOC budget – CAPEX including hardware & software – OPEX including people & cloud § Enterprise asset management systems
Figure 5: FireEye NX 4420 Figure 6: FireEye NX 7400 Figure 7: FireEye NX 7420 . FIPS 140-2 Security Policy v0.2 8 Figure 8: FireEye NX 7500 Figure 9: FireEye NX 9450 Figure 10: FireEye NX 10000 . FIPS 140-2 Security Policy v0.2 9 Figure 11: FireEye NX 10450 .
FireEye Email Security—Server Edition Administration and Diagnostics x x x x x FireEye Endpoint Security Administration and Diagnostics x x x x x FireEye Helix x x x x x Fundamentals of Network Traffic Analysis using FireEye Network Forensics x x x x x Helix Threat Analytics x x x x x Investigations with FireEye Endpoint Security x x x x x
GigaVUE-HC2 and FireEye NX 2400, a inline tool group solution through the FireEye GUI and Gigamon-OS H-VUE. The procedures are organized as follows: FireEye NX 2400 Configuration: Inline Tools Gigamon GigaVUE-HC2 Configuration: Inline Network and Inline Tool Groups. The FireEye GUI procedures focus on FireEye inline block operational mode.File Size: 1MBPage Count: 30
SOC/G&WS 200 Intro to LGBTQ Studies SOC 210 Survey of Sociology SOC/C&E SOC 211 The Sociological Enterprise SOC/C&E SOC/G&WS 215 Gender & Work in Rural Am SOC/ASIAN AM 220 Ethnic Movements in the US SOC/C&E SOC 222 Food, Culture, and Society x Any SOC course with a Social Sciences breadth will satisfy this prerequisite.
The FireEye CM series is a group of management platforms that consolidates the administration, reporting, and data sharing of the FireEye NX, EX, and FX series in one easy-to-deploy, network-based platform. Within the FireEye deployment, the FireEye CM enables real-time sharing of the auto-
The FireEye CM Series: CM-4400, CM-7400, CM-9400 (the module) is a multi-chip standalone . administration, reporting, and data sharing of the FireEye NX, EX, FX and AX series in one easy-to-deploy, network-based platform. Within the FireEye deployment, the FireEye CM enables
Configuring FireEye NX 2400 for Inline Block Operation Mode The FireEye GUI procedures focus on FireEye inline block operational mode. The configuration procedures in the later section will configure the GigaVUE -HC2 to send live traffic to the FireEye inline tool group, which will allow the use of FireEye's on-system deployment testing tools.
FireEye Network Security is an effective cyber threat protection solution that . 2550, NX 3500, NX 5500, NX 10550. Flexible Deployment Options FireEye Network Security offers various deployment options to match an organization’s needs and budget: . FireEye Network Security datasheet .
