Hitachi Solution For Databases - Virtualized Oracle RAC On .
Hitachi Solution for Databases - Virtualized OracleReal Application Cluster (RAC) on VMware vSphereusing Hitachi Virtual Storage Platform E990 andHitachi Advanced Server DS220Reference Architecture GuideBy Hitachi VantaraNovember 2020
FeedbackHitachi Vantara welcomes your feedback. Please share your thoughts by sending an email message toSolutionLab@hitachivantara.com. To assist the routing of this message, use the paper number in the subject and the title ofthis white paper in the text.Revision HistoryRevisionMK-SL-209-00ChangesInitial releaseDateNovember 17, 2020
Table of ContentsSolution Overview2Business Benefits2High Level Infrastructure3Key Solution Components5Solution Design7Storage Architecture7Oracle ASM Configurations11Management Server Configurations12Server and Application Architecture12SAN Architecture13Network Architecture14Physical Network Configuration15Engineering Validation18Database Configuration18Testing Environment18Test Methodology19Test Results20Conclusion22Product Descriptions22Hitachi Storage Virtualization Operating System RF23Hitachi Advanced Server DS22023Hitachi Advanced Server DS12023Hitachi Ops Center23Oracle Linux24Oracle Database With the Real Application Clusters Option24Oracle Automatic Storage Management24VMware ESXi24VMware vCenter Server Appliance24Brocade Switches25Cisco Switches25
1Hitachi Solution for Databases - Virtualized Oracle RealApplication Cluster (RAC) on VMware vSphere using HitachiVirtual Storage Platform E990 and Hitachi Advanced Server DS220Reference Architecture GuideThe use of Hitachi Solution for Databases with Oracle Real Application Cluster (RAC) virtualized on VMware vSphereprovides significant dynamic scalability, high availability, and reliability benefits while processing small to large Oracleworkloads and maintaining predictable performance capabilities for varying deployment types. This reference architectureprovides a validated virtualization solution for Oracle Database 19c running on VMware ESXi 6.7 while keepingvirtualization overhead to a minimum, with virtual machines using Oracle Linux 7.8 as Oracle RAC instances.Hitachi Solution for Databases with virtualized Oracle RAC on VMware vSphere is a fast, agile offering from Hitachi Vantarathat makes efficient use of deployed resources. Oracle on VMware aims for simplifying IT transformation, as it offersincreased efficiency, quality of service, predictable performance, flexible journey to minimize cost, dynamic scalability,disaster recovery with VMware vSphere and-most importantly-it is intuitive to implement.This solution integrates many innovative technologies from Hitachi Vantara, Oracle, and VMware. To prepare virtualizedOracle RAC on VMware vSphere, this environment includes the following: Hitachi Virtual Storage Platform (VSP E990) Hitachi Advanced Server DS220 for storage and computing resources Hitachi Advanced Server DS120 for management servers Oracle Real Application Cluster 19c VMware vCenter Server Appliance VMware ESXi VMware vSphere Running Oracle databases on VMware vSphere provides the following benefits: Dynamic scalability Predictable performance High availabilityThis solution provides the flexibility to select storage and compute resources based on unique requirements. Deploy smalldatabases as well as large databases, depending on resource availability.This document is for the following audiences: Database administrators Storage administrators System administrators1
2To use this reference architecture guide, you need familiarity with the following: Hitachi Virtual Storage Platform E990 Hitachi Advanced Server DS220 servers Hitachi Advanced Server DS120 servers Storage Area Networks Oracle Database administration Oracle Database 19c Release 3 Oracle Linux VMware vCenter Server Appliance VMware ESXiNote — Testing of this configuration was in a lab environment. Many things affect production environments beyondprediction or duplication in a lab environment. Follow the recommended practice of conducting proof-of-concepttesting for acceptable results in a non-production, isolated test environment that otherwise matches your productionenvironment before your production implementation of this solution.Solution OverviewUse this reference architecture to implement Hitachi Solution for Databases with virtualized Oracle Real Application Cluster(RAC) on VMware vSphere using DS220 as VMware ESXi hosts and Hitachi Virtual Storage Platform E990 (VSP E990) withVMware vCenter Server Appliance. This solution is engineered, pre-tested, and qualified to provide high performance, highavailability, and high reliability in demanding, dynamic Oracle environments. This reference architecture implementsHitachi Solution for Databases for Oracle Real Application Clusters database on two VMware ESXi hosts using VSP E990,which addresses high availability, performance, and scalability requirements for OLTP and OLAP workloads.Business BenefitsThe Oracle RAC solution for VMware vSphere virtualization provides numerous benefits to database administratorsbecause it creates a layer of abstraction between application, operating system, and the underlying hardware. VMwarevCenter Server Appliance manages the node resources jointly such that the cluster owns all the CPU and memory of allnodes.The following are some of the benefits of this reference architecture: Achieve high Oracle Real Application Clusters database performance with VSP E990 storage. Provide a solution for customers who are looking for converged Oracle products – not covered by physical servers forOracle RAC. Create a VMware ESXi cluster with multiple Hitachi Advanced Server DS220 servers to deploy virtual machines for anOracle Real Application Clusters database deployment. Add or remove ESXi hosts in the VMware ESXi cluster based on business requirements. Using the Oracle solution with VMware ESXi reduces space, power consumption, and IT administrative requirementswhile driving high-speed performance.2
3High Level InfrastructureHitachi Solution for Databases with virtualized Oracle Real Application Cluster (RAC) on VMware vSphere includes thefollowing components: Hitachi Advanced Server DS220 servers Hitachi Advanced Server DS120 servers Hitachi Virtual Storage Platform E990 Brocade G620 32 Gb/s SAN infrastructure Cisco 10/25 GbE LAN infrastructureThe configuration of Virtual Storage Platform E990 and Advanced Server DS220 has the following characteristics: Fully redundant hardware Dual fabric connectivity between hosts and storageFigure 1shows the high-level infrastructure for this solution.3
4Figure 1To avoid any performance impact to the production database, Hitachi Vantara recommends using a configuration with thefollowing characteristics: A dedicated storage system for the production database A dedicated storage system for storing backup data, if neededThe uplink speed to the corporate network depends on the customer environment and requirements. The Cisco Nexus93180YC-EX switches used in this reference architecture can support uplink speeds of 40 GbE or 100 GbE if higherbandwidth is required.4
5Key Solution ComponentsThe key solution components for this solution are listed in Table 1, Table 2, and Table 3. Detailed component information isprovided in the Product Descriptions section.TABLE 1. HARDWARE COMPONENTSHardwareHitachi VirtualStorage PlatformVSP E990Detail DescriptionTwo Controllers: 8 32Gb/s Fibre -60/0012 Intel Xeon Platinum 8276L 28C CPU @2.20GHzBIOS:S5BH3B16.H002768 GB (64GB 12) DIMM DDR4Synchronous Registered (Buffered) 2666MHzBMC: 4.64.062 Intel XXV710 dual port 25 GbE NIC cardsDriver: i40e1024 GB cache memory16 1.9 TB Backend NVMe SDDsHitachi VirtualStorage PlatformVSP G900Two Controllers4 32 Gbps Fibre Channel ports16 12 Gbps backend SAS ports512 GB cache memory4 6.0 TB 7.2K RPM SAS drivesHitachi AdvancedServer DS220CPLD: 11Driver Version:1.7.17Firmware: 6.8Intel X722 Dual Port 10 GbE SFP Driver: i40eDriver Version:1.7.17Firmware: 4.02 Emulex LightPulse LPe31002-M6 2-Port Driver: lpfc16Gb Fibre Channel AdaptersDriver Version:12.0.193.14.Firmware:12.0.193.135
6TABLE 1. HARDWARE COMPONENTS (CONTINUED)HardwareHitachi AdvancedServer DS120Detail DescriptionVersion2 Intel Xeon Gold Processor 6240, 18-core, BIOS:2.60GGHz, 85WS5BH3B16.H00Quantity2256 GB (32 GB 8) DIMM DDR4 Synchronous BMC: 4.64.06Registered (Buffered) 2666 MHzCPLD:111 64 GB MLC SATADOM for boot1 Dual Port 25 GbE NIC Intel XXV710 PCIecardDriver Version:1.7.17Firmware: 6.80(1.2007.0)1 Emulex LightPulse LPe31002-M6 2-Port Firmware:16 Gb Fibre Channel Adapters12.0.193.13Driver version:12.0.193.14Brocade G620 Fibre 24 48 port Fibre Channel switchesChannel switches32 Gb/s SFPsKernel: 2.6.34.6Cisco Nexus93180YC-EXBIOS: version07.61Cisco Nexus3048TP48 10/25 GbE fiber ports6 40/100 Gbps Quad SFP (QSFP28) ports1 GE 48-Port Gb Ethernet Switch2Fabric OS: v8.2.0b2NXOS: version7.0(3)I7(6)BIOS: version4.0.01NXOS: version7.0(3)I4(7)Note — The solution was tested with PCIe and OCP mezzanine NIC cards. Using all PCIe cards is recommended forconsistency and better NIC bonding options. SATADOM, SAN boot, or local boot can be used for the boot option.6
7TABLE 2. SOFTWARE COMPONENTS FOR COMPUTE NODESSoftwareVersionFunctionOracle Linux7.8 with UEK kernel updateOperating system for Oracle VM5.4.17-2011.4.6.el7uek.x86 64Oracle Database19c (19.3.0.0)Database softwareOracle Grid Infrastructure19c (19.3.0.0)Volume management, file system software, andOracle automatic storage managementTABLE 3. SOFTWARE COMPONENTS FOR MANAGEMENT NODESSoftwareVersionFunctionVMware ESXi6.7.0 U3Hypervisor for management serverVMware vCenter Server Appliance6.7.0 U3VMware cluster management serverHitachi Device Manager- StorageNavigatorMicrocode dependentStorage management softwareHitachi Ops Center10.2.0Hitachi infrastructure management softwareSwingbench2.6.0Oracle database stress test toolSolution DesignThis describes the reference architecture environment to implement Hitachi Solution for Databases with virtualized OracleRAC on VMware vSphere. The environment uses Hitachi Virtual Storage Platform E990.The infrastructure configuration includes the following: VMware ESXi server – There are two hosts configured in an Oracle database environment. Management Node – One VMware vCenter Server Appliance is used to manage all ESXi hosts. Storage System – There are virtual volumes mapped to each port that are presented to the server as LUNs. SAN Connection – There are SAN connections to connect the Fibre Channel HBA ports to the storage through BrocadeG620 switches.Storage ArchitectureThis describes the storage architecture for this solution.Storage ConfigurationThe storage configuration takes into consideration Hitachi Virtual Storage Platform, Oracle, and VMware recommendedbest practices for the design and deployment of database storage.Figure 2 illustrates the high-level storage configuration for this solution.7
8Figure 28
9Table 4 shows the storage pool configuration used for this solution.TABLE 4. DYNAMIC PROVISIONING POOL FOR HITACHI VIRTUAL STORAGE PLATFORM E990Pool IDVMware-ESXi-ClusterPool TypeDynamic Provisioning PoolRAID Group1-1 to 1-4RAID LevelRAID-10 (2D 2D)Drive Type1.9 TB SSDNumber of Drives 16Number of LDEVs 70LDEV Size(s)880 GBPool Capacity13.74 TBIn this solution, servers use SAN boot with RAID 10 protection. Table 5 shows the logical storage configuration used in thissolution.TABLE 5. VSP E990 LOGICAL STORAGE CONFIGURATIONASM disk Total numbergroupof dynamicprovisionvolumesDynamicprovisioningvolume sizesPurposeStorage PortDATA16200 GBOLTP application1A, 2A, 1B, 2B, 1C, 2C, 1D,tablespaces, system sysaux, 2Dundo, tempREDO810 GBOnline redo logs and controlfilesTEMP440 GBTemp logsFRA2200 GBIncremental backups,archived redo logs, controlfile and auto backupsOCR360 GBOracle cluster registry andvoting diskN/A2200 GBSAN Boot host OS volumesN/A2200 GBSAN Boot Oracle RAC VM OSvolumes9
10Table 6 shows the VSP G900 configuration for management servers.TABLE 6. VSP G900 CONFIGURATION FOR MANAGEMENT SERVERSItemPurposeValue or DescriptionVMware datastoresCCI deviceRAID LevelRAID-10 (2D 2D)Drive Type6.0 TB 7.2K RPM SASNumber of Drives4Number of Spare Drives0Number of LDEVs3LDEV Size(s)3000 GBNumber and Size of CCI device1 100 MBStorage Port for Management Servers7A, 7B, 8A, 8BThere is an additional RAID group consisting of four 3 TB 7.2k RPM SAS drives configured as RAID-10 (2D 2D). This is usedas shared storage for the management server cluster.Additional LUNs can be mapped if required. While the test environment was configured using a dedicated SAS RAID groupfor the management server cluster, this can be configured as a dedicated SSD RAID group, a dedicated dynamicprovisioning pool, and capacity on the dynamic provisioning pool configured for the Oracle environment.Database LayoutThe database layout design uses recommended practices from Hitachi Vantara for Hitachi Virtual Storage Platform E990 forsmall random I/O traffic, such as OLTP transactions. The layout also considers Oracle ASM best practices when usingHitachi Vantara storage.Base the storage design for database layout needs on the requirements of the specific application implementation. Thedesign can vary greatly from one implementation to another, based on the RAID configuration type and number of drivesused in the implementation.The components in this solution have the flexibility for use in various deployment scenarios to provide the right balancebetween performance and ease of management for a given scenario.The database layout has two nodes with Oracle RAC instances. The Oracle Linux 7.8 operating system is installed on twovCenter VMs with Oracle RAC database installed on them.Shared LUNs from storage are assigned to two VMs as shared virtual machine disks (VMDK) for the Oracle RAC database.The shared LUN is mapped to a datastore cluster, and virtual machine disks (VMDK) are utilized for OS SAN boot of VMs.10
11Oracle ASM Configurations Data and Indexes Tablespace - Assign an ASM diskgroup with external redundancy for the data and index tablespaces. TEMP Tablespace - Place the TEMP tablespace in this configuration in the TEMP ASM diskgroup. Undo Tablespace - Create an UNDO tablespace in this configuration within the Oracle Data ASM diskgroup. Online Redo Logs - Create an ASM diskgroup with external redundancy for Oracle online redo logs. Oracle Cluster Registry and Voting Disk - Create an ASM diskgroup with normal redundancy to contain the OCR andvoting disks and to protect against single disk failure to avoid loss of cluster availability. Place each of these files in theOCR ASM diskgroups.Oracle Initial ParametersTable 7 lists the Oracle Database settings.TABLE 7. ORACLE DATABASE SETTINGSEnvironmentValueRACYesASMYes – to support Oracle RAC DatabaseOracle ASM Disk MappingsTable 8 lists the details of the disk mappings from the LUNs to the ASM disk groups for Oracle Database tablespaces for the2 TB database size. This is an example with a single instance database virtual machine. Adjust parameters accordinglywhen multiple virtual machine pairs are used.TABLE 8. LUN DISK MAPPINGS TO ASM GROUPSASM DiskGroupASM DiskUDEV RulesLUN DetailsPurposeN/AN/A/dev/xvd[a-b]12 200 GBOS and Oracle DatabaseOCRN/A/dev/xvd[c-e]13 60 GBOracle cluster registry andvoting diskDATAN/A/dev/xvd[f-u]116 200 GBApplication dataREDON/A/dev/xvd[v-z]1, /dev/xvda[a-c]18 10 GBOnline REDO log groupFRAN/A/dev/xvda[d-e]14 200 GBFlash recovery areaTEMPN/A/dev/xvda[f-i]14 40 GBTemp11
12Management Server ConfigurationsTable 9 lists management server virtual machine (VM) configuration details.TABLE 9. MANAGEMENT SERVER VIRTUAL MACHINE CONFIGURATIONVirtual MachinevCPUVirtual MemoryDisk capacityIP AddressvCenter Server Appliance210 GB300 GB192.168.242.xxHitachi Ops Center Administrator416 GB100 GB192.168.242.xxHitachi Ops Center Analyzer432 GB800 GB192.168.242.xx410 GB110 GB192.168.242.xxHitachi Ops Center Analyzer detailviewServer and Application ArchitectureThis reference architecture uses two Hitachi Advanced Server DS220 servers for each storage system. This provides thecompute power for the VMware infrastructure that is used for the Oracle RAC database deployment.Two Hitachi Advanced Server DS120 servers are used for VMware ESXi management server configuration as well asadditional virtual machines to manage the environment.Table 10 lists the details of the server configurations for this solution.TABLE 10. HITACHI ADVANCED SERVER DS220 AND DS120 SPECIFICATIONSServerName andModelDS220DS120Server HostNameRoleCPU CoreRAMVMware ESXiserver 1VMware ESXi server 56 (2 28C)for Oracle RACdatabase VM768 GB (64 GB 12)VMware ESXiserver 2VMware ESXi server 56 (2 28C)for Oracle RACdatabase VM768 GB (64 GB 12)VMware ESXi 1Management server 16for Hitachi16managementapplications256 GB (32 GB 8)VMware ESXi 2256 GB (32 GB 8)12
13SAN ArchitectureMap the provisioned LDEVs to multiple ports on Hitachi Virtual Storage Platform E990 (VSP E990). These LDEV portassignments provide multiple paths to the storage system from the host for high availability. This reference architectureuses two dual port Emulex HBAs per Advanced Server DS220. Compute Servers 8 SAN switch connections are used for VSP E990 Fibre Channel ports. 8 SAN switch connections are used for server HBA ports. Management Servers 4 SAN switch connections are used for VSP G900 Fibre Channel ports. 4 SAN switch connections are used for management server HBA ports.Table 11 lists details of the Fibre Channel switch connect configuration on Hitachi Virtual Storage Platform E990 SAS ports.TABLE 11. SAN HBA CONNECTION CONFIGURATION TO HITACHI VIRTUAL STORAGE PLATFORM E990ServerHBADS220Server 1HBA1CN31CN31 HBA1 1CN31 HBA1 1 ASE44 66 1AVSP E990 1ASANswitch 1HBA2CN31CN31 HBA1 2CN31 HBA1 2 ASE44 66 2A2ASANswitch 2HBA3CN31CN31 HBA2 1CN31 HBA2 1 ASE44 66 1B1BSANswitch 1HBA4CN31CN31 HBA2 2CN31 HBA2 2 ASE44 66 2B2BSANswitch 2HBA1CN32CN32 HBA1 1CN32 HBA1 1 ASE44 66 3A3ASANswitch 1HBA2CN32CN32 HBA1 2CN32 HBA1 2 ASE44 66 4A4ASANswitch 2HBA3CN32CN32 HBA2 1CN32 HBA2 1 ASE44 66 3B3BSANswitch 1HBA4CN32CN32 HBA2 2CN32 HBA2 2 ASE44 66 4B4BSANswitch 2DS220Server 2HostGroupNameHost NameSwitch ZoneStorageSystemStoragePortBrocadeG620Switch13
14TABLE 11. SAN HBA CONNECTION CONFIGURATION TO HITACHI VIRTUAL STORAGE PLATFORM E990 (CONTINUED)ServerHBADS120Server1HBA1MN33MN33 HBA1MN33 HBA1 ASE42 43 1AVSP G900 1ASANswitch 1HBA2MN33MN33 HBA2MN33 HBA2 ASE42 43 3A3ASANswitch 2HBA1MN34MN34 HBA1MN34 HBA1 ASE42 43 2A2ASANswitch 1HBA2MN34MN34 HBA2MN34 HBA2 ASE42 43 4A4ASANswitch 2DS120Server2HostGroupNameHost NameSwitch ZoneStorageSystemStoragePortBrocadeG620SwitchNote — In a production environment, use separate storage ports for the management servers to avoid impact ondatabase performance. While shared storage ports can be used, monitor port utilization to avoid performance issues inhigh performance environments.Network ArchitectureDo the following when configuring networks in your environment: Use NIC bonding to provide failover and load balancing of interconnections within a server. Set all NICs to full duplex mode.Configure each VMware ESXi server node with at least the bonding interfaces for the following: Management network Public network Private networkEach virtual machine has public, private and management vNICs. Use separate VLANs for the following: VMware vCenter server management network Private Oracle database network Public network14
15Table 12 describes the network configuration for VMware vCenter virtual machines installed with Oracle RAC database.TABLE 12. NETWORK CONFIGURATION FOR VMWARE VCENTER VIRTUAL MACHINESVMsOracle RACVM 1 andOracle RACVM 2vSwitchIDSE-DSwitchNetworkIP link 110.76.32.xxUplink 2Public Network10.76.32.xxUplink 410.76.3
provides a validated virtualization solution for Oracle Database 19c running on VMware ESXi 6.7 while keeping virtualization overhead to a minimum, with virtual machines using Oracle Linux 7.8 as Oracle RAC instances. Hitachi Solution for Databases with virtualized Oracle RAC on VMware vSphere is a fast, agile offering from Hitachi Vantara
Bruksanvisning för bilstereo . Bruksanvisning for bilstereo . Instrukcja obsługi samochodowego odtwarzacza stereo . Operating Instructions for Car Stereo . 610-104 . SV . Bruksanvisning i original
82 hitachi zx200-3, zx225usrlc-3 1033091 sprocket sp-zx200-3 83 hitachi zx240-3 1032489 sprocket 84 hitachi ex300-1/2, ex270-1/2 1010467 sprocket 80 sp-ex300-1 85 hitachi ex300-3,ex270-5 1017928 sprocket 80 sp-ex300-3 86 hitachi ex300-5, ex330-5, zx270, zx330, zx350, zx370 1022168 sprocket 85 sp-ex300-5 87 hitachi
Hitachi Solution for Databases - Oracle Real Application Clusters Database 12c based on Hitachi Advanced Server DS220 and . computing performance, sophisticated power and thermal design to avoid unnecessary OPEX with quick deployment. For this solution two DS120 servers are used. The two DS120 servers are configured as a VMware vCenter cluster.
10 tips och tricks för att lyckas med ert sap-projekt 20 SAPSANYTT 2/2015 De flesta projektledare känner säkert till Cobb’s paradox. Martin Cobb verkade som CIO för sekretariatet för Treasury Board of Canada 1995 då han ställde frågan
service i Norge och Finland drivs inom ramen för ett enskilt företag (NRK. 1 och Yleisradio), fin ns det i Sverige tre: Ett för tv (Sveriges Television , SVT ), ett för radio (Sveriges Radio , SR ) och ett för utbildnings program (Sveriges Utbildningsradio, UR, vilket till följd av sin begränsade storlek inte återfinns bland de 25 största
Hotell För hotell anges de tre klasserna A/B, C och D. Det betyder att den "normala" standarden C är acceptabel men att motiven för en högre standard är starka. Ljudklass C motsvarar de tidigare normkraven för hotell, ljudklass A/B motsvarar kraven för moderna hotell med hög standard och ljudklass D kan användas vid
LÄS NOGGRANT FÖLJANDE VILLKOR FÖR APPLE DEVELOPER PROGRAM LICENCE . Apple Developer Program License Agreement Syfte Du vill använda Apple-mjukvara (enligt definitionen nedan) för att utveckla en eller flera Applikationer (enligt definitionen nedan) för Apple-märkta produkter. . Applikationer som utvecklas för iOS-produkter, Apple .
appointment issued by the Bank Group, the terms and conditions of the letter of appointment will prevail. . Before accepting an assignment, STC/STT are required to acquaint themselves with the restrictions on relatives’ employment contained in SR 4.01, par. 5.03 and report to HR Operations (Bank and MIGA appointments) or Client Services (IFC appointments) any close relatives working for .
