Direct Universal Access: Making Data Center Resources .

Microsoft ResearchMicrosoft AzureDirect Universal Access:Making Data Center Resources Available to FPGARan Shu1, Peng Cheng1, Guo Chen2,Zhiyuan Guo1, 3, Lei Qu1, Yongqiang Xiong1,Derek Chiou4, Thomas Moscibroda4Microsoft Research1, Hunan University2,Beihang University3, Microsoft Azure41

FPGA Deployment in Data Centers Wide deployment– Major cloud service providers Microsoft, Amazon, Facebook, Alibaba, Tencent, Baidu, IBM, etc. Accelerated applications– Computation Web search ranking Deep neural networks Big data analytics– Networking Network processing– Database/Storage SQL Key-value storeImage from D. Firestone et al., NSDI 2018Image from A. Caulfield et al., Micro 20162

Resource Access Requirements Heterogeneous resources– CPU– Memory– Other FPGAs– GPU– SSDImage from A. Putnam et al., ISCA 2014 3ServerServerCPUGPUHostDRAMSSDPCIe fabricFPGA boardNICFPGAEthernetOnboardDRAM.CPUFPGA board. FPGAOnboardDRAMGPU.HostDRAMSSDPCIe fabricFPGA boardNICEthernetData center network fabricFPGAOnboardDRAM.FPGA board. FPGAOnboardDRAM

FPGA Board in Data CenterDDR 4FPGA chipQSFPPCIe Gen 3Image nference/

Current FPGA Communication ArchitectureFPGA BoardFPGA ChipApplicationApplicationApplication LayerDCQCN(in LTL)DDR 4QSFPDDR StackLTLDDR 4 IPQSFP IPTransport LayerHostDMADMANVMePCIe Gen 3 IPData Link LayerPhysical LayerPCIe Gen 35

Current FPGA Communication ArchitectureFPGA BoardFPGA ChipApplicationApplicationDCQCN(in LTL)?DDR 4QSFPDDR StackLTLDDR 4 IPQSFP IPHostDMADMANVMePCIe Gen 3 IPPCIe Gen 3MAC layerPhysical layerImage from L. Zhang etal., CCR 20146

Problem #1 – Programming InterfaceLines of code to use each interfaceFPGA ChipApplication 1DDR StackLTLDDR IPEthernet IPHost DRAM: 294Application2Host CPUprogram: 205Onboard DRAM: 517Remote FPGA: 1356HostDMANVMe StackPCIe IP7

Problem #2 – AccessibilityFPGA ChipDDR StackOn-boardnaming spaceDDR IPApplicationApplicationLTLHostDMAData centernaming spaceEthernet IPNVMe StackServer-areanaming spacePCIe IP8

Problem #3 – MultiplexingFPGA ChipApplication 1Application 2Host DMAMux/DemuxDDR StackLTLDDR IPEthernet IPHostDMANVMe StackPCIe IP9

Problem #3 – MultiplexingFPGA ChipApplication 1Application 2LTLMux/DemuxDDR StackLTLDDR IPEthernet IPHostDMANVMe StackPCIe IP10

Problem #3 – MultiplexingFPGA ChipDDR StackApplication 1Application 2LTLHostDMANVMe StackPCIe Mux/DemuxDDR IPEthernet IPPCIe IP11

Problem #4 – SecurityFPGA ChipDDR StackMaliciousApplicationLTLHostDMANVMe StackHostMemoryDDR IPEthernet IPPCIe IPAccessunauthorizedaddressPCIe Fabric12

Problem #4 – SecurityFPGA ChipDDR StackVictimApplicationLTLHostDMANVMe StackMaliciousCPUProgramDDR IPEthernet IPPCIe IPAttackapplicationPCIe Fabric13

Existing Problems Complex programming interfaceSeparate naming spaceNo general multiplexingSecurity issue14

Direct Universal AccessServerServerFPGA 3FPGA 2FPGA 1App 2App ectHostDMA②DDRQSFPPCIe Gen3DDRPCIe Gen3CPUPCIe Gen3Intra-server networking fabric③Datacenter networking fabric15

DUA OverviewDUA is an “IP layer” ServerServerFPGAFPGAAppAn abstract overlay networkLeverage all existing h/w stacks①Hierarchicaladdressing & LTLDDRQSFPFPGAConnectHostDMA②DDRQSFPPCIe Gen3DDRPCIe Gen3CPUPCIe Gen3Intra-server networking fabric③Datacenter networking fabric16

DUA OverviewDUA is an “IP layer” ServerServerFPGAFPGAAppAppEfficient RoutingAppAppApp④①DUA byDUADirect resource GA, totally bypass nectHostDMA②DDRQSFPPCIe Gen3DDRPCIe Gen3CPUPCIe Gen3Intra-server networking fabric③Datacenter networking fabric17

DUA OverviewDUA is an “IP layer” ServerEfficient pApp④①CompatibleDUABSD-socket Interface DUAfor both applications and communication Ie Gen3DDRPCIe Gen3CPUPCIe Gen3Intra-server networking fabric③Datacenter networking fabric18

DUA OverviewServerFPGAFPGAAppAppDUADDRaccessDUA is an “IP layer” ServerEfficient RoutingCompatible stDMAGeneral p④DUALTLFPGAConnectHostDMA②for both applicationsand communication stacksDDRQSFPPCIe Gen3DDRPCIe Gen3CPUDDRQSFPPCIe Gen3Intra-server networking fabric③Datacenter networking fabric19

DUA OverviewServerFPGAFPGAAppAppDUADDRaccessDUA is an “IP layer” ServerEfficient RoutingCompatible BSD-socketInterface PCIe sLTLFPGAConnectDDRQSFPAppPCIe Gen3CPUProtect against both inside and outside attacksHostDMAPCIe Gen3Intra-server networking fabric③Datacenter networking fabric20

System ArchitectureServerFPGADUA Data PlaneDUAdata planeDDR ControllerHost DMAAppAppDUAOverlayAppFPGA Host StackCPUNVMe StackFPGA ConnectCPU ControlAgentLTLFPGAControlAgentServerFPGADUA UnderlayCPUFPGADDRQSFPPCIe Gen3Intra-server networking fabricNICDatacenter networking fabricNIC21

System ArchitectureServerFPGADUA Data PlaneDUAcontrol planeDDR ControllerHost DMAAppAppDUAOverlayAppFPGA Host StackCPUNVMe StackFPGA ConnectCPU ControlAgentLTLFPGAControlAgentServerFPGADUA UnderlayCPUFPGADDRQSFPPCIe Gen3Intra-server networking fabricNICDatacenter networking fabricNIC22

DUA Control Plane Challenge: large-scale resourceand routing info dissemination– Limited h/w resource DUA solution– Hierarchical addressing– Hierarchical routing– Leverage existing infrastructure Fully distributed and lightweight– Need no global synchronizationUID(serverID:deviceID)Address /portResource 00019FFFF0000x80000000800080011st block of host DRAM2nd block of host DRAM1st block of FPGA onboard1st application on FPGA2nd application on FPGAResource tableSrc Resource (UID)FPGA 1 (192.168.0.2:1)FPGA 2 (192.168.0.2:2)Dst Resource (UID) / StackFPGA 2 (192.168.0.2:2) / FPGA ConnectHost DRAM (192.168.0.2:3) / DMAOnboard DRAM (192.168.0.2:4) / DDRFPGA 1 (192.168.0.2:1) / FPGA ConnectHost DRAM (192.168.0.2:3) / DMAResources on other servers (*:*) / LTLInterconnection table23

DUA Data Plane Overlay– Unified interface– Routing Stacks– Leverage all the existing (oradopt future) stacksFPGADUA data planeDDR AccessAppAppHost DMADUAoverlayAppFPGA Host stackNVMe StackFPGA Connect UnderlayLTL– Efficient multiplexing– SecurityDUA underlayDDRQSFPPCIe Gen324

DUA Data Plane – Overlay Efficient & extensible design– Switch fabric High capacity cross-bar switch– Connector All cached routing tablesDUAoverlayFPGA CA– TranslatorAppAppConnectorConnectorConnectorSwitch FabricConnector Protocol translation High performance data path– Line-rate– Near rLTLTranslatorDDRTranslatorHost DMATranslatorLTLDDRHost DMA25

FPGA 1FPGA 2FPGA 3FPGA 4APPDUA overlayDUA overlayAPPEvaluation – efficiencyFPGAConnectExtreme low latency ( 50 d Trip Time through FPGA Connect and DUA for 4 times, LTL twice4.54.0DUALTLFPGA ConnectLatency 1760.196641282562.01.51.00.50.0Packet Size (B)26

Evaluation – Logic OverheadDUA Overlay 2 Ports: 4.24% 4 Ports: 9.29% 8 Ports: 19.86%DUA Underlay 4 Stacks and 3 PHYInterfaces: 0.25%27

Evaluation – Deep CrossingSPMVSPMV640 64640 128128 640DMV64 640DMVDMVDMVSPMVSingle FPGA Board: Parall 32, 2 FPGA Board: Parall 6445.28% Latency Reduction28

4.51.68E 0744.19E 06through FPGA Connectthrough CPUPure CPU1.05E 063.52.62E 0536.55E 04Latency (us)Throughput (GB/s)Evaluation – Regex Matching1.64E 042.54.10E 0321.02E 031.5through FPGA Connect10.52.56E 026.40E 01through CPU1.60E 01Pure CPU4.00E 001.00E 0000200040006000800010000 12000 14000 16000 18000Input String Length (Byte)64128256512102420484096819216384Input String Length (Byte) Up to 105 107 higher than CPU, Up to 105 lower than CPU Up to 3 times throughput and up to 55% latency reduction compared tousing CPU to move data between FPGAs29

Conclusion Current FPGA communication architecture– No universal access DUA: build the “IP” layer for FPGA in data center– Leverage existing data center network– Efficient routing– Compatible BSD socket interface– Unified multiplexing– Security Open source soon30

Thank you!Questions?31

DUA Overview FPGA Server DDR QSFP App App LTL DDR access FPGA Connect Host DMA DDR PCIe Gen3 App App LTL DDR access FPGA Connect Host DMA Datacenter networking fabric QSFP FPGA DDR access Connect Host PCIe Gen3 PCIe Gen3 CPU DUA DUA Intra-server networking fabric DUA is an “IP layer” ③ ② ① ④ Efficient Routing Direct resource access .