Big Data And Oracle Data Integration

Bridging the Big Data Divide with Oracle Data IntegrationMilomir Vojvodic,Business Development Manager, EMEA DIS

Diverse Data SetsInformation Architectures Today:Decisions based on transactional datatransactions, applications, structured DataInformation Architectures Today:Decisions based on all your dataVideo and ImagesDocumentsSocial DataMachine-Generated Data

ArchitectureOracle DataPrinciplesIntegration SolutionsandBestDataPracticesfor Big

Integrated ArchitectureMaster &Ref eOrganizeODSDataWarehouseSocialMediaText, ImageVideo, AudioKey-ValueData StoreAlertingEPMBI ApplicationsText Analyticsand SearchCDCHadoopCluster wMapReduceData MartsIn-DatabaseAnalyticsReal-TimeStreaming(CEP Engine)MessageBasedGovernReporting &DashboardsDB nalyticsVisualDiscoveryManagementSecurity, ure

Integrate Big Data with DW and TransactionalData StoresOracleBig Data OrganizeAnalyze & Visualize Load from big data processing into your data warehouse for further analysis Access your customer information while you process through your big data in order to look for patterns

Oracle Data Integration Solutions Complete and best-of-breedapproach to address enterpriseintegrationLegacySourcesOracle Enterprise Data QualityApplicationSourcesRelational andNon-RelationalOracle Data IntegratorOracle GoldenGate Maximum performance withlower cost of ownership, ease ofuse, and reliability. Certified for leading technologiesto deliver fast time to value Oracle customers report:–80% lower TCO–Five times higher performance–70% reduction in development costs

ArchitecturePrinciplesDB Replica andCDC withinandBestPracticesDataIntegrationLayer

What is Oracle GoldenGate?OGGSource DBTarget DB

What is Oracle GoldenGate?First OGG DifferentiatorAccessing directly transaction logsOGGSource DBTarget DBSecond OGG DifferentiatorMoving only committed transactions

Oracle DIS Use Cases - OGGMigrations&ConsolidationsOGGOGGOGG ADGZero DowntimeMigrations & UpgradesActive/ActiveDB DeploymentDisaster RecoveryReporting DatabaseNew DB/HW/OS/APPFully Active Distributed DBReporting Database and/or DR databaseOGGDW SynchronizationData Warehouse

OGG is Log Based ReplicaORHoursreach 5 days withthe current HW150100500Year1 Year2 Year3 Year4 Year5Currently during the End Of Dayutilizes the Server CPU by 40-50%and the IO by 90%. Probably the IOis the bottleneck.NO OF CPUs REQUIRED FOR SAMEPERFORMANCE*Required No.CPUs can beDisaster RecoverydoubledTestNo Of Required CPUs120100806040200and DevelopmentPrimary SiteYear1 Year2 Year3 Year4 Year5ESTIMATED COSTS FOR SERVER ANDLICENSE**Estimated Cost of Purchase in USDMillionsTIME REQUIRED FOR THE END OF DAYPROCEDUREDaily load time can 3 2 2 1 1 -Costs can beOracle License doubledCostsYear1 Year2 Year3 Year4 Year5

OGG Moves Only Committed TransactionsORBegin, TX 1Insert, TX 1Begin, TX 2Begin, TX 2PumpCheckpointBegin, TX 2Update, TX 1Insert, TX 2Insert, TX 2Insert, TX 2Commit, TX 2Commit, TX 2Commit, TX 2CaptureCheckpointBegin, TX 3Begin, TX 3Insert, TX 3Insert, TX 3Commit, TX 3Begin, TX 4Commit, TX 3Delete, TX 4DeliveryCheckpoint

ArchitecturePrinciplesETL and DataQuality withinandBestPracticesDataIntegrationLayer

ODI is centralizing all ETL rtingDataMigrationData PerformanceDataWarehousingData MartsData HubsBatch ScriptsData AccessSQLJavaCustomOLTP & ODSSystemsDataWarehouse, Data MartDataFederationOraclePeopleSoft, Siebel, SAPCustom AppsFilesExcelXMLOLAP

ODI is centralizing all ETL le Data IntegratorOLTP & ODSSystemsDataWarehouse, Data MartOraclePeopleSoft, Siebel, SAPCustom AppsFilesExcelXMLOLAP

Why is ODI different?First ODIDifferentiatorTransformationsusing the power ofthe Target Database– no staging serverODI E-LTSecond ODI DifferentiatorODI Declarative Design and ODI Knowledge Modulesfor reusing already written down level SQL codeStaging ServerODIOGGData WarehouseODI Knowledge d fromCDC SourceLoadFromSources efore LoadIntegrateTransformand Move toTargetsStaging TablesCheckServiceExpose DataandTransformatiWW Won ServicesSS SIntegrateServicesTarget TablesError TablesSample out-of-the-box Knowledge ModulesSAP/R3SiebelBenefitsSQLOracleOracleJMSCheck MS geTriggersServicesOracleDB2DB2CheckType IISiebel EIM DB2 rODI Declarative DesignODI Declarative Design1DefineWhatYou Want22AutomaticallyGenerateDataflowLog MinerDefine How : Built - in Templates

Oracle DIS Use Cases – ODI and EDQMigrations&ConsolidationsEDQOGGOGGOGG ADGOGGODIZero DowntimeMigrations & UpgradesActive/ActiveHigh AvailabilityNew DB/HW/OS/APPFully Active Distributed DBQuery Off-Loadingand Disaster RecoveryReporting Database and/or DR databaseODI EDQBI&DW Synchronizationand LoadingData Warehouse

Why Do We Need Data Quality?Customer IDCustomer NameAD23298Mr Peter MayhewAddress 19407 Main StVS38611Dr Ellen Van Der Heijde144 E Grove StDC18223Jalila Abdul-Alim (Do Not Call)4548 Pennsylvania AveCO9387ATayside Computers Inc.4912 E 41st NTZ35019Mr Zachary P Jahn98-1731 Ipuala LoopCB27843Mrs Edith Y Baba JuniorBaba Real Est. Corp.OX80306Andrew & Mary Baxter14 Oxbridge WayJP70210RD48107Mr RJ & Mrs FB MacDonaldMr Andy Baxter57 Hadleigh Close14 Oxbridge Wy19 Attributes non-standard,missing or invalidAbbreviations(often ambiguous)Inconsistent formatsAddress 2FairfaxStateVAZip22031-4001CountryUSABirth Date02/23/61KingstonPA18704US07/12/57Kansas CityMO64111-3349USA02/23/63FIdaho ted States 06/12/86Male209 Stony Point 28/67FSwindon SN5 9BZMilfordMANH3056USAUSA01/01/01YMApt 205WestleaCityCompound NamesMis-Fielded DataEmbedded Additional InformationErroneous DataMixed Business & Personal NamesInternational Date FormatsMultiple NamesDefault or Dummy Data 2011 Oracle CorporationGenderMWidespreadduplication(often hardto spot)

Why Do We Need Data Quality?10hp motor 115V Yoke mountMOT-10,115V, 48YZ,YOKEmtr, ac(115) 10 horsepower 115voltsThis 10hp yoke mounted motor is rated for115V with a 5 year r2610160010 horsepower115Yoke10 Caballos, Motor, 115 VoltiosTEAO HP 10.0 1725RPM 115V 48YZ YOKE MTRProduct data is much more variable and unpredictable than other data typesMotor, TEAO, 1725 RPM, 48YZ, 15 Voltios,Montaje de Yugo, hp 1020

Oracle Enterprise Data Quality Profile, Audit, Transform, Parse, Cleanse, Standardize, Match withinOne Unified Solution

EDQ Address Verification300 Berry #1210 SF CaliforniaLatitude37.775837Longitude ryValidate300Berry StStep 1 Extract pieces ofthe addressStep 2 Check the piecesagainst the information in theGlobal Knowledge Repositoryto complete and find thecorrect abbreviationsSubPremise#1210Unit 1210LocalitySFSan FranciscoAdministrativeAreaCaliforniaCAStep 3 Change character set –transliterate - if necessary94158-1670Step 4 Find LocationPostCode 2012 Oracle Corporation – Proprietary and Confidential22

ArchitectureOracle DataPrinciplesIntegratorandBestDataPracticesfor Big 2012 Oracle Corporation – Proprietary and Confidential23

ODI for Big DataHeterogeneous Integration to Hadoop Environments Supports Hadoop standardsTransformsVia MapReduce Easy to configure UI forgenerating MapReduceOracle DataIntegratorLoads

ODI for Big Data to OracleOptimized Integration to Oracle ExadataOracle Big Data ConnectorsTransformsVia MapReduceOracle DataIntegratorActivatesOracle Loaderfor HadoopLoadsHadoop ClusterOracle Database,Oracle ExadataOracle Big Data Appliance

Oracle Data Integrator for Big DataPutting Together the Unique AdvantagesSimplifies creation of Hadoop and MapReduce code to boostproductivityIntegrates big data heterogeneously via industry standards:Hadoop, MapReduce, Hive, NoSQL, HDFSUnifies integration tooling across unstructured/semi-structuredand structured dataOptimizes loading of big data to Oracle Exadata using Oracle BigData ConnectorsEngineered for running on and integrating with Oracle Big DataAppliance via Big Data Connectors

and Best Practices Oracle Data Integrator for Big Data . ODI for Big Data Heterogeneous Integration to Hadoop Environments Transforms Via MapReduce Loads Oracle Data Integrator Supports Hadoop standards Easy to configure UI for generating MapReduce .