IBM XIV At Red Hat

IBM XIV at Red HatArchitecture, Performance and TuningWill FosterSenior Sysadmin and IT Storage Lead, Red HatJune 4, 2012Will Foster, Red Hat Inc.

RED HAT: WHAT WE DO.We offer a range of mission-critical software and services covering:CLOUDMIDDLEWAREVIRTUALIZATIONHOW WE DO IT.We develop everything usingan OPEN SOURCE model.OPERATING SYSTEMSTORAGETHE BENEFITS.FlexibilityFaster technology innovationShared development reducescosts & accelerates innovation.Open collaboration offers productsthat genuinely meet customers'requirements.Better qualityBetter price & performanceAlignment to your needsWill Foster, Red Hat Inc.

IBM XIV Footprint - ArchitectureArchitecture1,800 Servers2 Datacenters602TB XIV Capacity6 IBM XIV Frames 5 x Gen2 1 x Gen32079 Volumes2.13PB Storage356 attached hosts 21 cluster groups Cisco MDS SAN Fabric 9513 core switches 110 BC switches IBM HS22, 22V IBM X Series forDatabasesXIV is our standard for SAN/FC and transactional environmentswith moderate to high performance requirementsWill Foster, Red Hat Inc.

IBM XIV Footprint - Environment100% VirtualizedNon-productionEnvironment42% VirtualizedProductionDatabase Environments Oracle 11g Oracle RAC MySQL & PostgreSQLKVM-based Dev Cloud iDataplex RHEL5 & 6 SoNASPerformance Data-sets Business Intelligence Data Warehouse Enterprise Service BusRHEV Virtualization Production Stage Test/Dev/QAWill Foster, Red Hat Inc.

Disruptive Growth in Compute and Storage2012 – 2.13PBNew Datacenter 73%2011 – 1.23PBTwo new Datacenters 102%Problem:Problem:Explosive Growth inStorage CapacitySurge in BusinessDemand and DeliveryScaleable StorageRHEV & Bladecenter ArchitectureAutomation and Config Management 25002000 43%1500Terabytes2010 – 622TBNew Datacenter2009 – 432TB 44%10005000200820092010Year2008 – 300TBWill Foster, Red Hat Inc.20112012

Virtualization, Cloud and Self-Service“Everything is different, but thesame. things are moremoderner than before.bigger, and yet smaller.it's computers.”- Ox, Bill and Ted's Excellent Adventure (1989)Will Foster, Red Hat Inc.

Virtualization, Cloud and Self-ServiceAppAppSolution omationToolingSelf-ServiceWill Foster, Red Hat Inc.RequestApp

RHEV Design (per Hypervisor)CustomersWill Foster, Red Hat Inc.

RHEV on XIV in the DatacenterWill Foster, Red Hat Inc.

Performance in Transactional WorkloadsWe use 1-3TB sized LUNS Databases range from 1-2TB in size Average 2 to 5ms latency or service timeWith XIV grid architecture: Do not have to balance X sized LUNs in order toachieve optimal cache and spindles, chunks go to every diskLVM Concatenation To grow datasets: New LUN as a 'PV' Extend the VolumeGroup and LogicalVolume (non-disruptive)Filesystem: Ext3 or Ext4 (on top of LVM Logical Volume)Logical VolumeXIV LUN 1XIV LUN 2VolumeGroupLogical VolumeLogical VolumeXIV LUN 3Logical VolumeWill Foster, Red Hat Inc.

XIV SAN Multipath Internals on Red HatExample output of 'multipath -ll' on Red Hat with XIV SANmpath2 (20017380004130b5f) dm-1 IBM,2810XIV[size 1.0T][features 1 queue if no path][hwhandler 0][rw]\ round-robin 0 [prio 8][active]\ 8:0:0:3 sdab 65:176 [active][ready]\ 8:0:1:3 sdah 66:16 [active][ready]\ 10:0:0:3 sdan 66:112 [active][ready]\ 10:0:1:3 sdat 66:208 [active][ready]\ 3:0:0:3 sdd 8:48 [active][ready]\ 3:0:1:3 sdj 8:144 [active][ready]\ 5:0:0:3 sdp 8:240 [active][ready]\ 5:0:1:3 sdv 65:80 [active][ready] On XIV, all paths are active at the same time IOPS balance across all paths so queue depth needs tobe tuned lower than traditional active/passive arrays Application uses may determine queue depth as welli.e. number of Oracle DB writers/threads etc.Will Foster, Red Hat Inc.

XIV SAN Multipath Internals on Red Hat dm-multipath is the default RHEL mpath subsystem Supports active-active and active-passive round-robin Some configurations support load shifting to different paths For optimal performance, tune the “min rr io” parameter “min rr io” determines how many requests to servicebefore moving to the next path For transactional environments, keep min rr io between10-32 (or 10 less than your queue depth) For sequential workloads, keep min rr io around 100-200 Experiment to see what works best for youWill Foster, Red Hat Inc.

Active: Active SAN pathing (4paths)SAN-1 FC3/7SAN-2 FC3/7SAN-1 FC4/7SAN-2 FC4/7Will Foster, Red Hat Inc.

Tuning Queue Depth on your HBA Tuning HBA and host queue depthDetermine your HBA defaultscat /sys/class/scsi device/*/device/queue depth Use a tool like “nmon” to monitor in-flight queues You may need to increase if: ›Queue depth buffer gets consistently saturated on your paths›You experience paths bouncing around erratically›Tune your multipathing subsystem appropriately›Multiply #paths against your queue depth to obtain optimal settingsExample: 4 x paths, HBA queue depth of 64 256 queue depthWill Foster, Red Hat Inc.

Tuning Queue Depth on Local Disk Tuning HBA and host queue depth Monitor your local disk queue depth in the same way If it is too low, performance may be affected You can easily increase this on the fly via:echo "128" /sys/block/sda/device/queue depth For best practices Consult your HBA and disk manufacturer Consider what makes sense for your workload (sequential/random, etc.)Will Foster, Red Hat Inc.

Using NMON on Linux to Monitor Performance Look at “InFlight” queues to determine your profile Balance disk & HBA queue depth with your multipath settings Test, Test, Test! Tools like hdparm, dd, bonnie and iozone can test general IOTools like orion can simulate database load.Will Foster, Red Hat Inc.

Summary XIV Footprint: Architecture XIV Footprint: Environment The “Growth Problem” Virtualization, Cloud and Self-Service Performance in Transactional Workloads Linux and Multipath TuningWill Foster, Red Hat Inc.

THANK ideos@RedHatNewstwitter.com/RedHatNewsRed Hatfacebook.com/RedHatIncRed Hatlinkedin.com/company/3545Will Foster, Red Hat Inc.

Jun 02, 2012 · IBM XIV Footprint - Architecture 2 Datacenters 6 IBM XIV Frames 5 x Gen2 1 x Gen3 Cisco MDS SAN Fabric 9513 core switches 110 BC switches IBM HS22, 22V IBM X Series for Databases Architecture 1,800 Servers 602TB XIV Capacity 2079 Vo