Kernel TLS And Hardware TLS Offload In FreeBSD 13 By .
Kernel TLS and hardware TLS offload inFreeBSD 13byMellanox, Chelsio and Netflix
Why crypto? Bob and Alice and the secret message Mathematical dependance on a relativelysmall pre-shared key When used right: Prevents eavesdropping Prevents data tampering When used wrong: Makes denial of service easier
What is TLS ? Transport Layer Security, TLS Used behind https:// (TCP port 443) Supports multiple crypto codecs among others AES 128B / 256B Supports multiple key exchange protocols DiffieHellman, DH Ron Rivest, Adi Shamir, Leonard Adleman,RSA Most recent version is v1.3
What is TLS ?
TLS v1.2 Layout of a TLS record More detailed information at: https://tls.ulfheim.net/TLSREC(s)uint8 ttls type (data,handshake,alert)TCPHDRuint8 ttls vmajor (3)uint8 ttls vminor (3)uint16 ttls length (0.16K)uint8 ttls nonce[ ]uint8 ttls data[ ]IPv4/IPv6HDRETHHDR
TLS v1.3 Layout of a TLS record More detailed information at: RETHHDRuint8 ttls type (data 23)uint8 ttls vmajor (3)uint8 ttls vminor (3)uint16 ttls length (0.16K)uint8 ttls data[ ]
AES 128B / 256B Advanced Encryption Standard, AES See: https://en.wikipedia.org/wiki/Advanced Encryption Standard A 16-byte block cipher The stream version can stop and resumeencryption at any arbitrary point in the TLSrecord Supports the concept of a crypto cursor FreeBSD also supports CBC
TLS implementations Current FreeBSD alternatives (OpenSSL based) Generic user-space, AES-NI SW kernel TLS, AES-NI Open Crypto Framework kernel backend TCP Offload Engine for TLS NIC kernel TLS. vs .
A look inside OpenSSL Datapath is oriented around: typedef struct bio st BIO; BIO read() BIO write() All data must have a pointer in user-space inorder to be encrypted Based on the source and sink methodology Refer to the bio(3) manual page
OpenSSL and kTLS 16 patches have been submitted by:Boris Pismenny borisp@mellanox.com FreeBSD userspace APIs: #include sys/ktls.h setsockopt(TCP TXTLS ENABLE) setsockopt(TCP TXTLS MODE) FreeBSD kernel support added in r351522: https://svnweb.freebsd.org/changeset/base/351522
Netflix kTLS Kernel TLS Motivation Handle 100Gb/s of TLS with nginx Retain performance advantages of asyncsendfile(9) (fewer context switches, nonginx thread pool, no extra memory copy) Eliminate any possible inefficiency
New mbuf technologies Not ready flag Unmapped mbufs Send Tags
not ready mbuf flag mbuf flag M NOTREADY tell socket buffers if mbufs areready for transmission or not. Added to support async sendfile in r275329 Sendfile(9) adds mbuf to socket buffer markedM NOTREADY Until M NOTREADY is cleared, tcp cannot send it disk reads are issued into those mbufs M NOTREADY cleared and tcp usr ready() routine calledafter disk read is complete Allows a simple mbuf filter routine, like TLS encryption, toprocess the mbufs before they are submitted to the networkdriver via the TCP stack.
Netflix “unmapped” mbufs Called “unmapped” because they carry an array of pointers to unmapped physicaladdresses. Initially envisioned for sendfile, not TLS Dramatically reduces the length of socket buffer mbuf chains, thus reducing cachemisses. For a 16K TLS record, it compresses chains by about 6:1 (TLS hdr, trailerand 4 buffers). For unencrypted sendfile, it can compress mbuf chains up to 19:1 5-20% CPU reduction in Netflix unencrypted workloadsDescribes a TLS record entirely, including TLS header, trailer, message data, andpointers to kernel TLS session state in a single mbuf A single reference counted entity per TLS record is key for NIC TLS offload to beable to easily handle TCP retransmissions.
Netflix Software kTLSSoftware Kernel TLS Implementation, TLS 1.0 - TLS 1.3 Plaintext data passed to kernel via sendfile() or sosend(). The kernel frames TLS records into M NOMAP mbufs atsendfile() or sosend() time and places them into socketbuffers. Mbuf chains are marked with M NOTREADY Framed records are queued for encryption when theywould previously be marked “ready” Encryption is done by a pool of kernel threads (1 per core) Once encrypted, mbufs are marked “ready” & sent to TCP
mbuf send tags A property of mbufs which tell the underlyingnetwork interface about dedicated packetprocessing and queues. A quick and efficient way to demultiplex datatraffic. Allows for traversal through VLAN and LAGG(Link Aggregation). Safe against route changes.
mbuf send tag APIs Control path methods: struct mbuf snd tag *mst; structifnet Allocate(ifp, &mst)Modify(mst, arg)Query(mst, arg)Free(ifp, mst)*ifp;
mbuf send tags From Network Stack, NS, perspective: struct mbuf *mb; struct ifnet *ifp; m pkthdr.snd tag mst; m pkthdr.csum flag CSUM SND TAG; ifp- if output(mb);
mbuf send tags From Network Driver, ND, perspective: struct mbuf *mb; struct xxx send tag *st; st container of(m pkthdr.snd tag, ) select queue by st- queue;NSLAGGVLANoooND
Dataflow overview
Sendfile dataflow overviewUsing sendfile and software kTLS, datais encrypted by the host CPU.This increases our bandwidthrequirements by 25GB/s to /sCPU100Gb/s100Gb/s12.5GB/s12.5GB/sMemoryNetwork Card
Sendfile dataflow overviewUsing sendfile and inline kTLS, data isencrypted by the NIC.12.5GB/s5GB/s5GB/sDisksCPU12.5GB/sThis reduces our bandwidthrequirements by 25GB/s to roughly thesame as no TLS.100Gb/s100Gb/s12.5GB/s12.5GB/sMemoryNetwork Card
TLS before and after
NIC kTLS offload challenges Minor OSI model violation. Packets are sent containing full headers,except for un-encrypted payload. Prior to retransmission, crypto cursor needsupdate by re-transmitting off-the-wire parts ofthe TLS record, if any.
Benchmarks
Netflix Video Serving with TLSKernel TLS Performance: 90Gb/s, 68% CPU (SW), 35% CPU (T6 NIC kTLS) Original ( 2016) Netflix 100G NVME flash appliance E5-2697A v4 @ 2.60GHz (16 core / 32 HTT), 128GB DDR4 2400MT/s, 1x100GbE, 4xNVME
Mellanox NIC TLS
Mellanox NIC TLS support ConnectX-6 DX (coming October 2019) http://www.mellanox.com/page/ethernet cards overview16 000 000 simultaneous TLS connections (25, 50, 100 and 200 Gbit/s)
Chelsio HW TLS support T6 NIC TLS supports TLS v1.1 and v1.2 usingboth AES-CBC and AES-GCM. TOE TLS support for kTLS is in progress. ccr(4) can be used for AES-GCM via the OCFbackend.
Questions and AnswersQ/A
The stream version can stop and resume encryption at any arbitrary point in the TLS record Supports the concept of a crypto cursor FreeBSD also supports CBC. . typedef struct bio_st BIO; BIO_read() BIO_write() All data m
The TLS-5 is a portable unit weighing just over 4 pounds. A detachable power cord is supplied with the TLS-5A and TLS-5C; it is not supplied with the TLS-5B and TLS-5D. As shown in Figure 1, the front panel provides four modular RJ-11 ja
Anatomy of a linux kernel development Questions : – How to work kernel code? – How to write C code on the kernel? – How to building and install the kernel on old version linux? – How to release the linux kernel? – How to fixes bugs (patch) on kernel trees? Goal : –
The transition from TLS 1.1 to TLS 1.2 has been steady, with 27% more hosts making the move in 2017. Currently, 89% of hosts are using TLS 1.2. IETF's progress on TLS 1.3 has been slow for many reasons, not the least of which is debate about whether TLS 1.2 is really "broken" enough to require fixing.
n Linux is a modular, UNIX -like monolithic kernel. n Kernel is the heart of the OS that executes with special hardware permission (kernel mode). n "Core kernel" provides framework, data structures, support for drivers, modules, subsystems. n Architecture dependent source sub -trees live in /arch. CS591 (Spring 2001) Booting and Kernel .
Kernel Boot Command-Line Parameter Reference The majority of this chapter is based on the in-kernel documentation for the ichwerewrittenbythe kernel developers and released under the GPL. There are three ways to pass options to the kernel and thus control its behavior: When building the kernel.
What if Linux Kernel Panics Kexec: system call to load and boot into another kernel from the currently running kernel (4.9.74). crashkernel 128M [normal kernel cmdline] irqpoll, nosmp, reset_devices [crash kernel cmdline] --load-panic option Kdump: Linux mechanism to dump machine memory content on kernel panic.
In TLS-N, by the de nition of non-repudiation, message authen-tication and the identi cation of at least one TLS peer is guaran-teed. WecompareTLS-Nto existingnon-repudiationproposals and identify properties that non-repudiation solutions must possess for particular use cases. We implement and evaluate TLS-N as an extension of the new
TLS-450PLUS and/or TLS-XB or a maximum of 16 modules per system 16 Sold Separately (either Factory Installed or as a Spare Part Module) Universal Input/Output Interface Module (UIOM) for Relay Control and Input Signal Monitoring 332813-001 - Factory Installed Module 330020-620 - Spare Part Module Up to 4 for each TLS-450PLUS and/or TLS-XB
