High Throughput Data Acquisition At The CMS Experiment At
High throughput DataAcquisition at the CMSexperiment at CERNAndré G. HolznerUniversity of California San Diegoon behalf of the CMS Data Acquisition GroupHPC Advisory CouncilSwitzerland ConferenceLugano, Switzerland23rd-25th March 20151
Outline The Compact Muon Solenoid experiment at the Large HadronCollider from collisions to observations Event building in a nutshell Event builder upgrade technology choices first layer PC performance tuning second layer performance measurements High Level Trigger Summary2
3What are we doing ?thispresentationunderstandingfundamental laws ofnature at the smallestscalesreproduce conditionssimilar to early afterthe big bang in thelaboratoryhigher energy closer in time to bigbangImage credit:Particle Data Groupat Lawrence BerkeleyNational Lab.
Reproducing the early universe in the labMt. BlancLac Léman4Large Hadron Collider Aerial ViewGVA airportImage credit: Maximilien Brice (CERN) CC BY-SA 3.0
Detecting early universe interactions5
CMS detector In real lifehttps://goo.gl/maps/prT4N6
Collisions to look forhttps://cds.cern.ch/record/1406325
From collisions to observations2 Mbyte x100 kHzcustomprotocolsreadoutrequeststrigger data40 MHzPCs8Gridofflinereconstruction,data analysisStorage2 Mbyte x 1 kHzFPGAsFPGAsPCs576 x 10 GBit/sEthernetInfiniband FDR6 spine / 12 leaffolded ClosLevel 1trigger(electronics)fully assembledcollision data40 GBit/sEthernet1st stagedata aggregationPCsHigh leveltrigger(software)reduction 1:100PCs2nd stagedata aggregation
LHC offline computing grid9T0 CH CERNT2 CH CSCShttp://wlcg-public.web.cern.ch/
10Event building in a nutshell13134 3 2 11234PC1313PC4 3 2 11234123413134 3 2 1fullswitchingmatrix42PC4 3 2 12134213421344 3 2 1customelectronics4242PC4 3 2 1424242
Why upgrade ? Many pieces of equipment have reachedtheir end of life need to replacehardware more detector channels addedin the next years Increase sensitivity to new physicsphenomena by increasing beamenergy, intensity and focusing more collisions per beam crossing more parts of the detector traversedby particles higher data volume per collision11
Infiniband12DAQ1 TDR (2002) Pros: Designed as a High PerformanceComputing interconnect over shortdistances (within datacenters) Protocol is implemented in the networkcard silicon low CPU load 56 GBit/s per link (copper or optical),now 100 GBit/s available Native support for Remote DirectMemory Access (RDMA) No copying of bulk data between userspace and kernel (‘true zero-copy’) affordable Cons: Less widely known, API significantlydiffers from BSD sockets for TCP/IP more difficult to implement in anFPGA Fewer vendors than Ethernet Niche marketMyrinet1 Gbit/sEthernet10 Gbit/sEthernetInfiniband2013Top500.org share by Interconnect family
Run II event builder overview13data flow
From custom to standard protocols subdetectors use custom electronics10 GBit/s Ethernet outmodules (VME, uTCA) to10 GBit/s incommunicate with on-detector6 GBit/s in6 GBit/s inASICs many different designs due to manydifferent requirements want to use commercial off-the-shelfequipment as ‘early’ as possible first common element: ‘SLINK’ (copper,64 bit x 50 MHz 3.2 GBit/s),future: optical 6 / 10 GBit/s customprotocol output: 10 GBit/s Ethernet TCP sender implemented in amid-range FPGA usinga reduced TCP state machine*Front-End Readout Optical LinkFEROL*SLINK inSLINK inCompact PCImodule14
FEROLs in real life15
First layer switching network 16 switches with 48 x 10 GBit/s portsto FEROLs 12 x 40 GBit/s portsto first layer eventdata aggregationPCs aggregation layer, full connectivitynot needed in principle future: adding 40 GBit/s switchesto connect the 10/40 GBit/s switchesfor faster failover16
PC Performance tuning optimization on event buildingPCs of: assignment of Ethernet receivequeue interrupts to CPU cores TCP kernel settings assignments of softwarethreads to CPUs cores per thread local memoryallocation using libnumagraphical monitoring of IRQ activity17
Infiniband Clos network18 12 leaf, 6 spine switches 36 FDR (56 GBit/s) ports per switch 3 links between each leaf/spinepair 18 x 12 216 external ports 6 Tbit/s bisection bandwidth Subnet manager running ona switch full connectivity needed here all sources send to all destinations switching to Ethernet an option12 leaf switches6 spine switches
Infiniband throughput testleaves84 senders running a large scale test 84 sending PCs1950 receiversspines 50 receiving PCs 1 LID per host using off-the-shell software: qperf rc rdma write bw 84 x 50 4200 connections Obtain an upper limit on whatwe can get with our (in-house)event builder software and protocol achieved 37 GBit/s per receiver( 70% of linespeed after encoding)data flowlink occupancy during test
20Test with Event Buildingreceiving PCsmust alwayswait for slowestsenderoverhead dueto handshaking 32 GBit/sper receivingPC for 72senders x 54receivers(86% of qperfthroughput)
21High Level Trigger assembly of full collision dataon second layer of PCs multiple 36 x 40 GBit/s switches few to many distribution must reduce the rate of selected collisionsfrom 100 kHz to 1 kHz 15’000 cores 150 ms decision timeon average software of 3.8M C and 1.2M pythonlines of source code partial reconstruction of collision data finding clusters of high energy deposit 3D track fitting (Kalman filtering) from 3Dand 2D points matching of tracks to clusters Data exchange between 2nd stageevent building PCs and event filtering PCsvia files (NFS) allows decoupling of event building andfiltering software needs careful tuning of NFSFUsBUFUsBUFUsBU
Summary / Conclusions22 We presented the new Data Acquisition network (event builder) forfor the CMS detector at CERN for LHC Run II Multiple networking technologies are used: 10 / 40 GBit/s in the first aggregation layer 56 GBit/s FDR Infiniband in the second, fully connected layer 40 / 10 / 1 GBit/s in the output (filtering) layer Ready for LHC run II and looking forward to exploring new energies !Thank you for your attention !
23BACKUP
Accelerator complex24
Particle identification25image credit: CERN
High throughput Data Acquisition at the CMS experiment at CERN André G. Holzner University of California San Diego on behalf of the CMS Data Acquisition Group
Table 1. Cisco ACE to Avi Networks Cisco CSP 2100 Existing Cisco Model Migration to Cisco CSP Avi Vantage Ace 4710 Throughput: 0.5, 1, 2, 4 Gbps SSL Throughput: 1 Gbps SSL TPS: 7,500 Cisco CSP 4-core Avi SE Throughput: 20 Gbps SSL Throughput: 4 Gbps SSL TPS: 8,000 Ace 30 Service Module Throughput: 4, 8, 16 Gbps
NSA 5600 firewall only 01-SSC-3830 NSA 5600 TotalSecure (1-year) 01-SSC-3833 Firewall NSA 6600 Firewall throughput 12.0 Gbps IPS throughput 4.5 Gbps Anti-malware throughput 3.0 Gbps Full DPI throughput 3.0 Gbps IMIX throughput 3.5 Gbps Maximum DPI connections 500,000 New connections/sec 90,000/sec Description SKU NSA 6600 firewall only 01-SSC-3820
Rochester Institute of Technology Center for Imaging Science Computer Engineering. ABSTRACT High throughput data acquisition, whether from large image sensors or high-energy physics detectors, is characterized by a ‘setup’ phase, where instrumental parameters are set, a ‘wait’ phase, in which the system is idle until a triggering event occurs, an‘acquisition ’ phase, where a large .
SSL-VPN Throughput 4 Gbps Concurrent SSL-VPN Users (Recommended Maximum, Tunnel Mode) 10,000 SSL Inspection Throughput (IPS, avg. HTTPS) 3 5.7 Gbps SSL Inspection CPS (IPS, avg. HTTPS) 3 3,100 SSL Inspection Concurrent Session (IPS, avg. HTTPS) 3 800,000 Application Control Throughput (HTTP 64K) 2 16 Gbps CAPWAP Throughput (1444 byte, UDP) 20 Gbps
Expected throughput Data Rate x 0.65 In our case: 780 x 0.65 507 507 Mbps of throughput is what we may expect in good conditions in a lab with a single client. Measure Generally speaking, we can have two scenarios when we do a throughput test: APs are in Flexconnect local switching APs are in local mode or Flexconnect central switching
AA001.003.01 Data Acquisition Didactic Panel (NI USB 6003) AA001.002.01 Data Acquisition Didactic Panel (NI USB 6002) AA001.001.01 Data Acquisition Didactic Panel (NI USB 6001) AA001.000.01 Data Acquisition Didactic Panel (NI USB 6008) 10 11 Data acquisition, processing and Monitoring The didactic panels
Basics of Throughput Accounting Throughput The rate at which the system produces goal units. In business, this is the rate our system produces net, new dollars (euros, pesos, moneyessentially). Throughput can also be viewed as the value our organizations generate.
The Academic Phrasebank is a general resource for academic writers. It aims to provide the phraseological ‘nuts and bolts’ of academic writing organised according to the main sections of a research paper or dissertation. Other phrases are listed under the more general communicativ e functions of academic writing. The resource was designed primarily for academic and scientific writers who .
