2019 SPLUNK INC. Lessons Learned AWS .conf19 SPEAKERS: Please Use .
2019 SPLUNK INC.Migrating Splunk toAWSLessons LearnedShane NewmanIgor AlekseevPrinciple Splunk ArchitectRed Hat, Inc.Partner Solution ArchitectAWS
2019 SPLUNK INC.Migrating Splunk to AWSLessons LearnedShane NewmanIgor AlekseevPrinciple Splunk ArchitectRed Hat, Inc.Partner Solution ArchitectAWS
2019 SPLUNK INC.ForwardLookingStatementsDuring the course of this presentation, we may make forward‐looking statementsregarding future events or plans of the company. We caution you that such statementsreflect our current expectations and estimates based on factors currently known to usand that actual events or results may differ materially. The forward-looking statementsmade in the this presentation are being made as of the time and date of its livepresentation. If reviewed after its live presentation, it may not contain current oraccurate information. We do not assume any obligation to updateany forward‐looking statements made herein.In addition, any information about our roadmap outlines our general product directionand is subject to change at any time without notice. It is for informational purposes only,and shall not be incorporated into any contract or other commitment. Splunk undertakesno obligation either to develop the features or functionalities described or to include anysuch feature or functionality in a future release.Splunk, Splunk , Turn Data Into Doing, The Engine for Machine Data, Splunk Cloud,Splunk Light and SPL are trademarks and registered trademarks of Splunk Inc. in theUnited States and other countries. All other brand names, product names, ortrademarks belong to their respective owners. 2019 Splunk Inc. All rights reserved.
2019 SPLUNK INC.Splunk on AWSWe are better together
2019 SPLUNK INC.“Our partnership with Splunk isincredibly important for ourcustomers. Customers love AWSagility with Splunk visibility.”Andy Jassy, CEO, Amazon Web Services
2019 SPLUNK INC.Splunk’s AWS Credentials AWS Advanced Technology Partner AWS Big Data Competency AWS Security Competency AWS DevOps Competency AWS Government Competency AWS Education Competency AWS IoT Competency AWS MSP Technology Provider AWS Marketplace Partner AWS Security by Design Program Partner
2019 SPLUNK INC.Comprehensive AWS VisibilityAWS Data SourcesAWS EC2Amazon S3Amazon RDSExploreAmazon EMRDashboardAlertAmazon SNSAmazon ELBAmazon RedshiftAWS CFAmazonKinesisAmazonAPI k App for AWSAct
2019 SPLUNK INC.Why is Splunk Important for AWSCustomers?“You can’tprotect whatyou can’tsee.”Best Practices for SecuringWorkloads in Amazon Web ServicesGartner, Neil MacDonald, Greg Young“Securitymonitoring willmake or break atechnology riskmanagementprogram.”Assessing the Risk: Yes, the Cloud CanBe More Secure Than Your On-PremisesEnvironmentIDC, Pete Lindstrom“Securityrequiresvisibility.”Amazon Web Services“Intro to AWS Security”AWS Summit Series
2019 SPLUNK INC.Red Hat JourneyAWS Services Red Hat Main Objective Migrate to AWS Networking Amazon ELB, Amazon EBS, Amazon EC2, Amazon S3 Support from AWS Choose migration Path Instance Types Migration of Historical Data Snowball family
2019 SPLUNK INC.AWS Global InfrastructureAWS Global Infrastructure 22 Regions with 69 Availability Zones 3 Regions coming soon:Cape Town, Milan and Jakarta 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
2019 SPLUNK INC.AWS Global NetworkAWS Global Network Redundant 100 GbE network Private network capacity betweenall AWS Region, except China 2019, Amazon Web Services, Inc. or its Affiliates. All rights reserved. Amazon Confidential
2019 SPLUNK INC.World-class network performanceNEW!C4C3C1 1 GbpsCC1 EnhancedNetworking 10 Gbps 20x PPS 100 µslatencyC5C5n ENA EBSoptimizedby default 25 Gbps EFA 50 µslatency 100 Gbps 3x PPS
2019 SPLUNK INC.EC2 lprivateserversT3M5 M5dD2H1Generalpurpose DensestorageBurstableBig dataoptimizedR5 R5d R5mMemoryoptimizedX1 X1eMemoryintensiveIn-memoryHighI3 I3mI/OHighI/OC5 C5dCompute-optimizedG3P3F1z1dz1dmGraphicsFPGAs Compute andintensivememoryGeneralintensivepurposeGPU
2019 SPLUNK INC.Complete set of data building blocksData movementAWS Storage Gateway FamilyData securityand managementBlockFileAmazon MacieAmazon KinesisVideo StreamsAWS QuickSightAmazon KinesisData StreamsAWS LambdaAmazon EBSAmazon EFSAmazon EFS File SyncAWS CloudFormationAmazon S3Transfer AccelerationAWS CloudTrailAWS Direct ConnectAWS CloudWatchStorage PartnersAWS Snow familyAmazonS3Amazon GlacierObjectAWS IAMAWS KMS
2019 SPLUNK INC.AWS Snow FamilyAWS Snowball EdgeAWS SnowballData Transfer OnlyAWS SnowmobileData Transfer & Edge Compute 50 or 80TB storage capacity 42/100TB storage capacity (s3) 10GE networking 10/25/40GE networking Data encryption end-to-end Data encryption end-to-end Rugged 8.5 G impact case Rugged 8.5 G impact case Rain and dust resistant Rain and dust resistant Clustering / AWS Greengrass EC2/AMI support / GPU options20 PB Data Transfer Exabyte-scale storage in a 45ftcontainer (90PB s3/Glacier/EBS) Data encryption end-to-end Dedicated security personnel GPS tracking, alarm monitoring, 24/7surveillance, and optional additionalsecurity
2019 SPLUNK INC.Red HatA brief history of IT Data Analytics
2019 SPLUNK INC.Why Red Hat uses Splunk.Why do we (continue to) invest in Splunk over other products in this space? Splunk is a strong company and continues to grow their product offerings Vibrant app marketplace (splunkbase), providing 1,000 apps and add-ons Splunk continues to add capability through development or acquisition Willing to engage in partner-level activity Welcomes participation in feature development, through alpha/beta, advisory council, UX testing, directproduct development Active co-development of join go-to-market opportunities Specific product features (transformation on-the-fly, simple and advanced statistical analysis, rich andwell-documented methods for getting data in and out)Frankly, there is no product that provides a comparable, broad complement of capabilities and platformsupport
2019 SPLUNK INC.Where Splunk Fits Splunk Enterprise is the primary product supporting our DataAnalytics offering Splunk Enterprise Security is Red Hat’s SIEM solution, layeredon top of Splunk Enterprise Splunk offers capabilities which align with our technicalroadmap:– product/data integration– scalability– availability Splunk can be a data consumer and/or data provider Provides data normalization of disparate sources (through theSplunk Common Information Model) Working towards native support for our container platform(OpenShift) Native metrics capabilities
2019 SPLUNK INC.Red Hat’s Legacy Deployment
2019 SPLUNK INC.Overview of Red Hat’s Deployment Who’s using Splunk? 1000 active users, mostly within IT, but increasingly outside On a typical day, we serve about 180 unique users 300 average concurrent searches during peak usage, bursts of over 900 On average, Splunk processes:– 300 searches/minute, 18k/hour, 432k/day Some data points on volume and scale– Daily data ingested: 4TB / 10bn events / 18k distinct data sources– Storage 570TB NVMe over 1PB S3 Historic growth (platform, users, volume)––––2013 - 8 nodes, 100 users, 1,500 forwarders, 400GB/daily2015 - 30 nodes, 250 users, 2,000 forwarders, 800GB/daily2018 - 45 nodes, 800 users, 3,000 forwarders, 1.2TB/daily2019 - 112 nodes, 1000 users, 10,000 forwarders, 4TB/daily
2019 SPLUNK INC.Overview of RedHat’sDeployment
2019 SPLUNK INC.Changes to Criticality StatusBecoming a Mission Critical applicationSplunk promoted from C3 application to C1 (Mission Critical) Availability: 95% 99.99% Maximum Tolerable Downtime: 48 hours 24 hours Recovery Time Objective: 24 hours 4 hours Acceptable Recovery Time: 24 hours 4 hours Recovery Point Objective: 24 hours 1 hour Acceptable Data Loss: 24 hours 1 hour Active discussions to promote to C0 (above Mission Critical)
2019 SPLUNK INC.Growing PainsVictim of Success
2019 SPLUNK INC.Deployment ChallengesBudget constraintsContinuing to onboard data without scaling the platform Limited funding– License OR hardware License now, hardware later Exceeded 1k active users More users more searches More searches higher hardware utilization Platform instability Index tier overloaded– CPU, Memory, IOWait all at/near 100% at least 20 hours/day– Skipped searches at 20%– Rolling restarts for 12 indexers taking 10-15 hours The platform was a ticking time bomb
2019 SPLUNK INC.Tipping PointFailure to meet SLAsNetwork change resulted in packet loss between DC’s Removed support for jumbo frames between DC’s Index replication could not complete between sites Unable to search any data Splunk is only application impacted– 1st and ONLY major outage for the platform Data still flowing in, unable to search 0 data loss Inability to search index tier– Knowledge bundles unable to transfer– Indexers unable to complete batch-add process to cluster master 3 Days to recover All data searchable took over 30 hours This was unacceptable to stakeholders
2019 SPLUNK INC.The Path Forward
2019 SPLUNK INC.Due Diligence for IT Data AnalyticsSplunk vs. Open SourceOpen dialog started between stakeholders and Data Analytics 2 years since last application analysis– Functional/Non-Functional requirements defined– Thorough feature analysis of similar competing products– Cost analysis Results Closest competitor did not meet all Functional/Non-Functional requirements defined by stakeholders Higher cost of ownership by 20% Very involved and complex migration path Required staffing an entire team to support Outcome Decision made to make further investment into the Splunk platform
2019 SPLUNK INC.Due Diligence for IT Data AnalyticsOn-Prem vs. AWSMany factors to consider: Scalability Performance Data durability Physical footprint Automation potential Cost
2019 SPLUNK INC.OutcomesOn-Prem vs. AWS vs. HybridOn-PremAWS Scalability concerns Limitless scalability– Due to DC footprint Great performance– 1.5M IOPS (40K/host) 99.99% data durability Large DC footprint– 2 storage frames (20U each)– 50U compute (minimum) 2X cost of AWS Mind-boggling performance– 49M IOPS (1.3M/host) 99.999999999% data durability No dedicated DC footprint– All HF’s are virtual 50% cost of On-Prem solution
2019 SPLUNK INC.Workload Migration CreditReimbursement for portion of the costs to be used on future projects
2019 SPLUNK INC.The Largest WholesaleMigration of an On-PremSplunk deployment toAWS . EVER
2019 SPLUNK INC.Migration Planning Flow
2019 SPLUNK INC.Planning the Largest AWS SplunkMigrationQuestions Do we age out?– Support dying infrastructure that is almost out of support for an additional 18 months? Should we create a massive index cluster and replicate?– How long will replication take?– Latency concerns?– Bandwidth constraints? Do we move wholesale?– How long will it take to move 600TB?– Do we have to use EBS storage?– Can we just load data into S3 and use that? Can we run in parallel for a while?– Net-new data in AWS while moving historical data?– Can we have a hybrid deployment? How much downtime will end users experience?
2019 SPLUNK INC.Planning the Largest AWS SplunkMigrationGuidance Do we age out?– Support dying infrastructure that is almost out of support for an additional 18 months? Absolutely not Should we create a massive index cluster and replicate?– How long will replication take? - Too long!– Latency concerns? - Somewhat– Bandwidth constraints? - 100Mbps Do we move wholesale?– How long will it take to move 600TB? - Covered on next slide– Do we have to use EBS storage? - No– Can we just load data into S3 and use that? -Yes, with a little tweaking Can we run in parallel for a while?– Net-new data in AWS while moving historical data? - Ideal– Can we have a hybrid deployment? - Yes, some cost though How much downtime will end users experience? - Covered later
2019 SPLUNK INC.Compute MigrationAMI’s FTW AMI created per instance type Script to attach NVMe storage in RAID0 Configure AWS Splunk deployment configs Migrate all on-prem knowledge objects to AWS Redirect all deployment server and indexing functions to AWS tier Turn existing indexers into heavy forwarders and add tag for data being routed through them(unmanaged universal forwarders) Correct unmanaged universal forwarders routing through legacy index tier Redirect user traffic to AWS SHC
2019 SPLUNK INC.Data Migration OptionsHow to move 600TB Internet?– Only consistent 100Mbps bandwidth can be allocated on direct connect AWS network for this effort 598 Days, 20 hours, and 42 Minutes NOPE! Snowball?–––––Requires hybrid deployment approachTime to transfer 60TB 4.5 daysShipping from DC to AWS DC 2 daysAWS receives snowball to data loaded in S3 5 daysCost rounding error in project Bottom line– Never underestimate the bandwidth of a delivery truck
2019 SPLUNK INC.Migration PathCold Storage to SmartStoreCreate 12 S3 buckets and ordered Snowballs– 1 per on-prem indexer Load up 50TB from each of the 12 on-prem indexers– via 80TB snowballs Install s3fs on 12 new AWS indexers Mount S3 buckets to each of the 12 AWS indexers Update indexes.conf to use s3fs mount as cold storage– This effectively completes the migration from on-prem to AWS– Disconnect on-prem index cluster Create new SmartStore bucket to be shared across all indexers Update indexes.conf to enable SmartStore– Took about 3 days to move data from individual S3 buckets to common SmartStore bucket Disable cold storage via indexes.conf Unmount s3fs mounts and deleted legacy cold storage buckets
2019 SPLUNK INC.Finished Product
2019 SPLUNK INC.Red Hat’s OHC Deployment
2019 SPLUNK INC.Performance MetricsOn-Prem vs. AWSLegacy On-PremAWS 120K IOPS (FIO) 49M IOPS (FIO) 855,500 Events/sec 8,733,930 Events/sec 16 concurrent searches* 72 concurrent searches* 286s avg search** 22s avg search** 20% skipped search ratio 1% skipped search ratio*Before queueing begins**Well qualified searches with index AND source OR sourcetype defined
2019 SPLUNK INC.Lessons Learned
2019 SPLUNK INC.Lessons Learned When purchasing RI’s, pay attention to the type (conversion vs new) Snowballs arrive sporadically––––received out of orderreceived 2-3 per weeklast 6 snowballs delayed by 2 weeksall were ordered at the same time 80TB snowballs are heavy, 2 people to move On-prem hardware performance AWS Performance You can mount S3 storage via s3fs and use it as cold storage– This was our interim step before enabling SmartStore– FIO consistently reported 10K IOPS per bucket There should really be more than 1 Splunk Admin
2019 SPLUNK INC.Q&A
2019 SPLUNK INC.ThankYou!Go to the .conf19 mobile app toRATE THIS SESSION
Splunk App for AWS Comprehensive AWS Visibility AWS Data Sources AWS EC2 Amazon EMR Amazon Kinesis Amazon R53 Amazon VPC Amazon ELB Amazon S3 CloudFront AWS CloudTrail Amazon . Planning the Largest AWS Splunk Migration Do we age out? -Support dying infrastructure that is almost out of support for an additional 18 months?
As an alternative, an app can be uploaded using the corelight-client command line utility: corelight-client splunk list splunk delete Removes a previously uploaded Splunk App. splunk download Retrieves a previously installed Splunk App as a ZIP file. splunk list Returns a list of all installed custom Splunk Apps. splunk upload Uploads a new Splunk App from a ZIP file.
GSG-Monitoring-and-Diagnostics-101 sales@splunk.com www.splunk.com Try Splunk Cloud or Splunk Enterprise for free or learn more about IoT and industrial data. Already have Splunk? Download Splunk Apps on Splunkbase. 5 Connecting Splunk to Industrial Data and the IoT Kepware Industrial Data Forwarder for Splunk
Intellipaat's Splunk certification training includes the complete aspects of Splunk Developer and Splunk Administration. This Splunk course also includes various topics of Splunk, such as installation and configuration, Splunk Syslog, Syslog Server, log analysis, Splunk dashboard, and deploying Splunk search, monitor, index, report, and analysis.
Splunk Configuration 1. To install Splunk Apps, click the gear. 2. To install Splunk Apps, click the gear. Click Browse more apps and search for "Fortinet" 3. Install the Fortinet FortiGate Add-On for Splunk. Enter your splunk.com username & password. 4. Then install the Fortinet FortiGate App for Splunk. Enter your splunk.com username .
This is Intellipaat Master Program in Splunk tool includes Splunk Developer and Splunk Administration training. As part of this Splunk course, you will work on searching, sharing, saving Splunk results, creating tags, generating reports and charts, installing and configuring Splunk, monitoring, scaling and indexing large volumes of searches and analyzing it using the Splunk tool. Instructor Led Training 26 26Hrs of highly interactive
Splunk Documentation: docs.splunk.com Splunk Education & Training: education.splunk.com Third-Party Tools (not supported by Splunk) Search Examples: Big Book of Splunk Searches:bbosearch.com GoSplunk-A Search Repository: gosplunk.com Sizing Tool for Predicting Storage Requirements: splunk-sizing.appspot.com
Gain Insights into your Microsoft Azure Data using Splunk Jason Conger Splunk. Disclaimer 2 . Deploying Splunk on Azure Collecting Machine Data from Azure Splunk Add-ons Use cases for Azure Data in Splunk 3. Splunk available in Azure Marketplace 4. Splunk in Azure Marketplace
Core Dashboard & Splunk Investigate - dashboards@splunk.com Core Viz & Splunk Investigate - visualizations@splunk.com ITSI Experience - itsi-beta-gt-feedback@splunk.com Enterprise Experience - dashboardsbeta@splunk.com
