Oath Moloch Deployments
Oath Moloch DeploymentsAndy Wick
DeploymentsOath has three different network types that we monitor, each with their ownnetwork design and scale. Office - Employees, VPNs CiC - Backoffice in a data center 50 global offices, each with its own egress10 VPN concentratorsCentralized Elasticsearch clusterEach location with its own Elasticsearch clusterProd - Production traffic Each location with its own Elasticsearch clusterToo much Gbps to capture everythingSome traffic we don’t want to capture
Design AOL & Yahoo each had their own take on visibility Combined the best of both for OathZeek (Bro), Suricata, Moloch and other toolsRun all tools on each visibility box instead of specialized boxesUse a few hardware configurations so easy to reuseUse an NPB to load balance trafficWatch traffic to/from “internet”For production reduce traffic Analyze traffic for less then halfSave PCAP for even smaller percent
NPB Aggregates, filters, and load balances trafficNormal Arista switch, in a special mode Packets flow one directionStill need another switch for standard networkingInput: Span ports or IXIA optical tapsOutput: Visibility HostsOffice/CiC: 7150S-24, 7280SEProduction: 7508R 13RU, 6 power supplies, max11,484W
Why use a NPB? Easy to add Moloch capacityAllows the networking team and security team to act more independently Networking team can add more links at any time, just connect taps to NPBThe security team can add more tool capacity at any time, just connect tools to NPBMove the traffic filtering from a bpf to purpose built hardwareMultiple tools can see the same traffic (or subset), again making networkteam happy they aren’t involvedLoad balancingHandles HA issues of packets taking different paths as long as all paths hit the same NPB
Visibility Hosts Bro is a memory/cpu hogUse afpacket for everything requires a patch to BroWant enough memory to potential run other tools and scanners in the future2RU for space considerations, however boxes are deeper
Hardware Selected Keep number of configurations to a minimumArista NPBVisibility boxes New, Supermicro 6028R-E1CR24L24x10TB 128GB - Office, CiC24x12TB 256GB - ProdMoloches Used, most are 5 years old4x10TB 128GB - 1 node - Office, CIC4x12TB 256GB - 2 node - ProdSession replication
Office/CiC ArchitectureSpan ports mirror traffic to NPBMolochESHostnamemoloches-*eth0 - normalOS/managementHigh num eth portsEth24 and downeth1Low num eth portsEth1 and upMost sites only have 1 or 2 visibility serversHostname: visibilityNN
Prod ArchitectureThing1Thing2Each link monitored requires 2 NPB portsMolochES livesin data centerTORmolochesNNeth0 - normal OS/managementeth1visibilityNN
Reality
Things to watch for Hardware reliability Might require more ES replicationExtra capture nodesExtra hard drives on handConfigure multiple elasticsearch endpoints to handle failuresMake sure Elasticsearch is configured with shard awarenessIncrease thread pool.bulk.queue size setting in ESUse ES 6.4.2 not 6.2.4 if using replication and ES 6.xSecurity, use iptablesNumber of ACLs NPB can handle
Sizing Office visibility sizing is done by number of employees. Every site has an Arista NPBEach visibility box can handle 250 employees for desired retentionNPB is used for aggregationCiC & Prod sizing is done by avg Gbps Every site has an Arista NPBNPB aggregates trafficNPB is used to drop trafficMoloch rules are used to not save pcap
Example Sizing Sheet
Example Costing
Reality Cost Breakdown
Traffic Reduction NPB Drop by ip/portSimple perl script generates commands from CMDBMoloch Use rules to drop trafficDon’t save all the TLS packets Helps with ES - don’t save file pos Helps with Vis - reduces pcap storageDon’t save SYN scansDon’t save some ad network traffic to clouds
NPB Samplemail-list mx.aol.com tcpdefault ip access-list mail-listip access-list mail-list! file:mail.yahoo.com - (smtp):25 ips 100permit tcp any host 1.2.3.4 eq 25permit tcp host 1.2.3.4 eq 25 anypermit tcp any host 4.3.2.1 eq 9993 9995permit tcp host 4.3.2.1 eq 9993 9995 any259993 9995
Prod Rules - Drop TLS after 10 packets- name: "Drop tls"when: "fieldSet"fields:protocols:- tlsops:maxPacketsToSave: 10
Prod Rules - Drop SYN scans- name: "Drop syn scan"when: "beforeFinalSave"fields:packets.src: 1packets.dst: 0tcpflags.syn: 1ops:dontSaveSPI: 1
Prod Rules - Drop traffic to cloud- name: "Drop tls by hostname"when: "fieldSet"fields:host.http:- ad.doubleclick.net- foo.example.comprotocols:- tlsops:dontSaveSPI: 1maxPacketsToSave: 1dropByDst: 10
Other important high performance settings# IMPORTANT, libfile kills performancemagicMode basic# Enable afpacketpcapReadMethod tpacketv3tpacketv3BlockSize 8388608# Increase by 1 if still getting Input Dropstpacketv3NumThreads 2# Start with 5 packet threads, increase by 1 if getting thread drops.do NOT need 24 threads :)packetThreads 5You
Pcap Encryption at rest with Moloch Each pcap file has its own data encryption key (DEK)The DEK is encrypted using a key encryption key (KEK)The encrypted DEK, IV, and KEK id used for each file is stored in ESThe list of KEKs and currently used KEK are stored in the moloch config.inifile[default]pcapWriteMethod simplesimpleEncoding aes-256-ctrsimpleKEKId kekid1[keks]kekid1 Randomkekpassword1kekid2 Randomkekpassword2
QUESTIONS?
New, Supermicro 6028R-E1CR24L 24x10TB 128GB - Office, CiC 24x12TB 256GB - Prod Moloches Used, most are 5 years old 4x10TB 128GB - 1 node - Office, CIC 4x12TB 256GB - 2 node - Prod Session replication
The Thorny Devil Moloch horridus As Bianca has recently returned from a holiday to Central Australia, in this newsletter we have decided to feature her favourite new reptile, the Thorny Devil lizard, sometimes also known as the Moloch (Moloch horridus). A Thorny
The Moloch of Totalitarianism monument. For more than ten years, the day of the Russian New Martyrs and Confessors Assem-bly has been marked in freezing February. Pilgrims come from the Petersburg par- ishes, including many young people and children. After the requiem, there are baskets
Devil The Thorny Devil's scientific name Is Moloch horridus. Which is how he got his name 'Moloch.' The Thorny Devil is a reptile with many quirky adaptations to help them survive in their environment. They are covered in spikes, to scare off predators. They also have a strange bump on the back
Exchange Online and SharePoint Online comparisons: o Small deployments: 1,000 users o Medium deployments: 10,000 users o Large deployments: 100,000 users Functional units We analyzed the cloud services and on-premises deployments based on the functional unit for each cloud service—that is, the "useful output" offered by a deployment.
Apr 30, 2018 · Pre-Accession Interview 5-11 34 . Applicant Forms Review 5-12 35 . Report of Additional Information 5-13 36 . Pre-Oath Briefing 5-14 39 . Oath of Enlistment 5-15 40 . Accession and Delayed Entry/Enlistment Program Data Remain in USMIRS . After Oath of Enlistment 5-16 41 .
2013 Annual Report ONE OATh, ONE LAW, ONE MOvEMENT In the BSA’s earliest days, there was just one Scouting program— Boy Scouting—and it used the Scout Oath and Scout Law. Over the years, new programs have come along and with them new codes: the Cub Scout Promise, the Law of the Pack, the Venturing Oath, and the Venturing Code.
The reference architecture is intended to explain OATH's vision for authentication, as well as to provide a . CNG Microsoft Cryptographic API: Next Generation CMP Certificate Management Protocols . various industry regulations are met. For this reason, OATH believes that the path toward strong digital identity must start with strong .
3 Lorsqu’un additif présent dans un arôme, un additif ou une enzyme alimentaire a une fonction technologique dans la denrée alimentaire à laquelle il est adjoint, il est considéré comme additif de cette denrée alimentaire, et non de l’arôme, de l’additif ou de l’enzyme alimentaire ajouté et doit dès lors remplir les conditions d’emploi définies pour la denrée en question .
