2017 NSF IUCRC Biennial Meeng ARLINGTON, VA JULY 27, 2017

2017 NSF IUCRC Biennial Mee4ngARLINGTON, VA JULY 27, 2017Mul4-Site CentersStudent-centric Collabora1ve ExperiencesDereje Agonafer, Jenkins GarreJ ProfessorThe Na4onal Science Founda4on Center for Energy Smart Electronic Systems (ES2)12017 NSF IUCRC Biennial Mee4ng

Outline Background & MotivationInfrastructuresHistory of Working Relationships of Center DirectorsManaging DirectorProjectsES2 2016 – 2017 Projects – 21 projects – all led bystudentsSample ProjectES2 Student collaboration on projectsMonthly Project Meetings – led by studentsQuarterly Project Meetings – led by students2

Outline - cont IAB Meeting – led by studentsSample ProjectStudent collaboration on projects exampleStudents conference attendanceStudents internshipPostersIAB MeetingsThesis ExampleJobs - Recent PhD’s graduates3

Energy Usage in Typical Data Center! Data centers are big consumers of electricity! In 2014, 70 billion kWH electricity³! 1.8% of total U.S. electricity consumption! Cooling energy typically contributes 30-50% of Data Centeroverall energy consumptionFigure 3: Typical Data Center Layout¹! Environmental design criteria is importantTypical Data Center SystemHVAC54%HVAC ChilledWater Plant14%ComputerLoads38%HVAC AirMovement9%Energy Efficient SystemLigh4ng1%UPSLosses13%Ligh4ng2%UPS Losses6%Figure 4: Electrical Consump4on Distribu4on in two data centers²Source¹ : Google Data Center PUE es4ma4on, 2008Source²: Data Center Best Prac4ces Guide, 2012Source³: Lawrence Berkeley Na4onal Laboratory, United States Data Center Energy Usage Report, 2016ComputerLoads63%

Infrastructure – built in Phase I

History of Working Relationships of Center DirectorsOver 25 years of working rela1onshipA Brief Overview of Recent Developments inThermal Management in Data CentersSami Alkharabsheh, John Fernandes, Betsegaw Gebrehiwot,Dereje Agonafer, Kanad Ghose, Alfonso Ortega, Yogendra Joshi andBahgat SammakiaJ. Electron. Packag 137(4), 040801 (Sep 10, 2015) (19pages)IAB Mee4ng, UTA 2016 (alum Suma Kuravi)

History of Working Relationships of Center Directors –SemiTherm 2017Discussion: led by Bahgat Sammakia, BUPanelist:Dereje Agonafer, UTAKanad Ghose, BUYogendra Joshi, Georgia TechAl Ortega, Villanova2017 Thermi AwardPresented to Chandrakant PatelChief Engineer and Senior Fellow of HP Inc.PresentersDereje Agonafer, UTA (Nominator)Dr. Veerendra Mulay, General ChairFacebook

Managing Director of ES2 Andrea Palmeri serves as the Managing Director of ES2. Inmy opinion, she is critical to the functioning of the centerand Phase II would not have happened without herleadership. As a Managing Director, Andrea is responsiblefor all of the Center’s operations, including–––––––proposal coordination,business development,marketing, website,Center communications,development of policies and procedures,NSF reporting, andserving as the liaison between the university partners andindustry members.

Projects ES2 2016 – 2017 Projects – 21 projects –all led by students Sample Project ES2 Student collaboration on projects Monthly Project Meetings – led bystudents Quarterly Project Meetings – led bystudents IAB Meeting – led by students

ES2 2016-2017 Projects

ES2 2016-2017 Projects (cont)

PI: Dereje Agonafer(UTA)UTA StudentsDr. BetsegawGebrehiwot (Intel)Dr. Manasa Sahini(Intel)Vishnu SreeramSuhas SathyanarayanDigvijay SawantBU StudentDr. Husam Alissa(Microsoft)Collaborator: Bahgat Sammakia (BU)MentorsDavid Mendo and Simpson CumbaComcastDeepak Sivanandan, Mark Hendrix, andCommScopeTom CraftVeerendra MulayFacebookAkhil Docca and Mark SeymourFuture FacilitiesSaurabh Shrivastava and Yasin Makwana PanduitNaveen Kannan, James Hoverson, JimJagers, and Mike KalerMestexRichard Craig and Robert YurcikVerizon WirelessThe Center for Energy-Smart Electronic Systems

ES2 Student collaboration on projects, Example 1 Data Center Lab with raised- floor hotaisle containment configuration.– CRAC Units equipped with Variable SpeedDrives– SynapSense Wireless sensors to monitortemperature, humidity and static pressure– Remote connection that facilitates datalogging from all servers for experimentalanalysis Collaboration with ES2 students inBinghamton University– Interdisciplinary collaboration (CS, ME etc.)between ES2 graduate students– Access to research facilities and laboratoriesfor all ES2 research projectsData Center Lab at UTA

ES2 Student collaboration on projects, Example 2 Artificial Neural Networkapplications in data centercontrol systems A neural network developedand trained with CFD datamodelling the BU data centerlab. ANN technique successful inpredicting the output of thedata center in both steadystate and transient cases.Source: Neural network modeling in model-based control of a data centerPredic4on of server temperature using Neural NetworkOutput error when compared to CFD data

7/28/1715

IAB Meeting - April 29 - 30, 2015, Steel ORCA, NJExperimental Setup for Dynamic Liquid-Cooled MCM (Project 6)R. Kokate (Google), J. Fernandes (Facebook), M. Sahini (Intel) and D. AgonaferDepartment of Mechanical & Aerospace Engineering, University of Texas at Arlington, Arlington, TX - 76019Preparing for Experimental Tes4ngAbstractCon4nuing trends of increasing microprocessor power densi4es and nonuniform temperature distribu4ons pose a significant challenge to the coolingrequirements of a data center environment. With a view to minimizing energyconsump4on of cooling infrastructure, the objec4ve is to design a dynamic,energy-efficient and prac4cal cooling solu4on for high-power equipment. Amul4-chip module (MCM) is chosen as the pladorm to base the design of such aliquid-cooled solu4on.Determining prac4cal applica4on of the dynamic cold plate necessitatesexperimental tes4ng. In the absence of a MCM thermal test vehicle (TTV), anassembly consis4ng of copper blocks embedded with thick-film heaters installedin a plas4c substrate will serve to simulate a func4oning module. Opera4on ofthe mock-MCM and control of pumps will be previewed through arepresenta4ve, but simplified, air-cooling setup. Results show great promise forapplica4on to the TTV and suggested control schemes are discussed. Objec4ves}Original cold plate(OCP)Compare performance of original and dynamic solu4ons Cri4cal measurements Pumping power (Pp) Component temperatures Component power dissipa4on}MCM TTV is subjected to different loading schemesDynamic cold plate(DCP) OCP – Run tests at different flow rates DCP – Set target temperature in dead-band control schemeCounter effects of drij in chiller supplytemperature-10 C to 75 C Non-uniform power distribu4on at the die– Need to cool for maximum chip temperature Energy efficiency of servers– High efficiency near maximum u4liza4on only– Cooling accounts for 30% of overall power (S. Pelly, 2009)oQ – 1lpm,2lpm, 3lpm,4lpmInternalLoop(water)Arduino –heater powerdissipa4onPump ater)F1ChillerExternal loop Calculate total pumping power,FutureWorkTin – 35 C,45 C Experimental Tes4ng Develop an extendable, dynamic and prac4cal cold plate for energyefficient thermal management of high power devices Introduce controls and instrumenta4on to create a smarter solu4on withtargeted delivery of cooling resources based on local heat dissipa4on Op4mize cooling infrastructure to reduce energy consump4on and boostthe overall system efficiencyTin Agreed-upon benchmark device pladorm for inves4ga4on High-power MCM layout as reference(courtesyof Endico: Interconnect)Around 500W of powerdissipa4on in a 78mm x 78mm (60 sq.cm)footprintQuan4ty Power112113740W5W-PinPoutToutoutPump2Tin Test matrix} While tes4ng both OCP and DCP solu4ons MCM TTV is subjected to different loads Uniform loading (5 cases) Idling: ASICs and FPGA each dissipate 5W Higher loads: ASICs each dissipate 10, 20, 30 and 40W FPGA power dissipa4on is always 5W Non-uniform loading (12 cases) Each block is set to dissipate 40W in isola4on Remaining blocks are set to idling power (5W)Pla[ormSubstrateASICFPGALIGAOCPclampedto MCMSensor powersupplyF2Provide cooling as required to reduce power consump'onComponent 𝑃 Dataacquisi4onunit(DAQ)Sensor –electricalcircuitPump1Source: R. Mahajan, Proc. IEEE,August 2006Goals Measuring impedance curve for each sec4on Use flow rate to determine pressure drop at opera4ng pointMCM TTV power supply The Interna4onal Technology Roadmap for Semiconductors (2010) predicts– Power density of high performance processors to more than double by 2024– Allowable junc4on temperature to decrease from 90 C to 70 C– Total thermal resistance will need to decrease by almost a factor of fourDetermine pumping power for each sec4on by Liquid cooling test benchCool for the highest temperature across all devices Coolant circuit for tes4ng both original and dynamic cold platesBackground/Need of StudyWhile tes4ng DCP Different chiller supply temperatures: 15 C, 25 C, 35 C and 45 C Inves'ga'ng warm water cooling is a priority Results of OCP tes4ng used to determine target device temperature Setup control scheme to achieve selected temperature Results: Device temperatures vs. total flow rate} Uniform power distribu4on Non-uniform power distribu4on Including dynamic loading}Preparing for Experimental Tes4ng}}While tes4ng OCP Maximum permissible device temperature is 85 C Different chiller supply temperatures: 35 C and 45 C Inves'ga'ng warm water cooling is a priority Chiller bypass loop is modulated to deliver specific flow rates to the cold plate Different flow rates considered: 1lpm, 2lpm, 3lpm and 4lpm All 17 loading cases tested at each flow rate Results: Device temperatures vs. pumping power– Establish reference points by tes4ng performance of original cooling solu4on– Determine improvement over baseline with employment of dynamic solu4onAcknowledgements Thanks to the industry mentors for their con4nuous input and supportMentorAffilia4onPeter BeucherWolverine / MicroCoolSy-Jenq LoongWolverine / MicroCoolScoJ HollandWolverine / MicroCoolMark SeymourFuture Facili4esChris AldhamFuture Facili4esSteven SchonQuantaCoolVidhya ShankarIBMBenson ChanEndicor InterconnectYasin MakwanaPat McGinnPanduitCoolIT Systems

7/28/1717

Lieutenant Alex PriesserDan Furman7/28/1719

Disserta4on Defense ExamExperimental and Computational Study of Multi-levelCooling Systems at ElevatedCoolant Temperatures in Data CentersRoom LevelManasa SahiniRack Level,PhD CandidateFacebookMechanical Engineering CoolITUniversity of Texas at ArlingtonSupervising ProfessorDereje AgonaferServer Level, CISCOFacebookWinterfell OC1

Jimil Shah presen4ng7/28/1727

Jobs - Recent ES2 PhD’s graduatesStudent NameUniversityGradua4on (date)Internship(list company, denote IAB members)Fahad MirzaUTA2014Global FoundriesSaeed GhalamborUTA2014Tianyi GaoBU2015Faculty, Sichuan University – Pirsburgh Ins4tute,Chengdu, ChinaBaiduZhihang ZangBU2015Faculty member in ChinaAnjali ChauhanBU2015MicrososJohn FernandesUTA2015FacebookRick EilandUTA2015DellMarianna VallejoUTA2015CH2MFurat AfranBU2016NVIDIAHusam AlissaBU2016MicrososKourosh Nema4BU2016Future Facili4esBetsegaw GebrehiwotUTA2016IntelJeff LuJrellUTA2016UTA FacultyOluwaseun AweUTA2016Business OwnerA SakibUTA2016NXP SemiconductorsTyler StacheckiBUABDBloombergMarcelo del ValleVU2016ComcastManasa SahiniUTA2017Intel

Thank you!

Jul 06, 2017 · Presented to Chandrakant Patel Chief Engineer and Senior Fellow of HP Inc. Presenters Dereje Agonafer, UTA (Nominator) Dr. Veerendra Mulay, General Chair Facebook Managing Director of ES2 Andrea Palmeri serves as the Managing Director of ES2. In