Introduction And Database Technology - Universiteit Leiden

The Future of 1997 (Gray)Conclusions Unifying object-oriented concepts with therelational model. New datatypes (image, document, drawing) arebest viewed as the methods that implement themrather than the bytes that represent them. By adding procedures to the database system,one gets active databases, data inference, anddata encapsulation. The object-oriented approachis an area of active research.(4/4)September 11, 2012Databases and Data Mining33

The Future of 1996Database Research: Achievements andOpportunities Into the 21st Century.Silberschatz, M. Stonebraker, J. Ullman Eds.SIGMOD Record, Vol. 25, No. 1pp. 52-63March 1996September 11, 2012Databases and Data Mining34

New Database Applications (1996) EOSDIS (Earth Observing System Data andInformation System) Electronic Commerce Health-Care Information Systems Digital Publishing Collaborative DesignSeptember 11, 2012Databases and Data Mining35

EOSDIS (Earth Observing SystemData and Information System)Challenges: Providing on-line access topetabyte-sized databasesand managing tertiarystorage effectively. Supporting thousands ofinformation consumerswith very heavy volume ofinformation requests,including ad-hoc requestsand standing orders fordaily updates. Providing effectivemechanisms for browsingand searching for thedesired data,September 11, 2012Databases and Data Mining36

Electronic CommerceHeterogeneous informationsources must beintegrated. For example,something called a"connector“ in one catalogmay not be a "connector“in a different catalog "schema integration“ is awell-known and extremelydifficult problem.Electronic commerce needs: Reliable Distributed Authentication Funds transfer.September 11, 2012Databases and Data Mining37

Health-Care Information SystemsTransforming the healthcare industry to takeadvantage of what isnow possible will have amajor impact on costs,and possibly on qualityand ubiquity of care aswell.Problems to be solved: Integration of heterogeneous forms of legacy information. Access control to preserve the confidentiality of medicalrecords. Interfaces to information that are appropriate for use byall health-care professionals.September 11, 2012Databases and Data Mining38

Digital Publishing Management and delivery of extremely largebodies of data at very high rates. Typical dataconsists of very large objects in the megabyte togigabyte range (1996) Delivery with real-time constraints. Protection of intellectual property, including costeffective collection of small payments andinhibitions against reselling of information. Organization of and access to overwhelmingamounts of information.September 11, 2012Databases and Data Mining39

The Information SuperhighwayDatabases and database technologywill play a critical role in thisinformation explosion. AlreadyWebmasters (administrators ofWorld-Wide- Web sites) arerealizing that they are databaseadministrators September 11, 2012Databases and Data Mining40

Support for Multimedia Objects (1996) Tertiary Storage (for petabytestorage) New Data Types timely and realistic presentation ofthe data?gracefully degradation service? Canwe interpolate or extrapolate someof the data? Can we reject newservice requests or cancel old ones?Multi-resolution Queries The operations available for eachtype of multimedia data, and theresulting implementation tradeoffs.The integration of data involvingseveral of these new types.Quality of Service Tape silosDisk juke-boxesContent Based RetrievalUser Interface SupportSeptember 11, 2012Databases and Data Mining41

New Research Directions (1996) Problems associated with putting multimediaobjects into DBMSs.Problems involving new paradigms for distributionof information.New uses of databases New transaction models Data MiningData WarehousesRepositoriesWorkflow ManagementAlternative Transaction ModelsProblems involving ease of use and managementof databases.September 11, 2012Databases and Data Mining42

Conclusions of the Forum (1996)The database research community has a foundational role in creating the technologicalinfrastructure from which database advancements evolve. New research mandate because of the explosions inhardware capability, hardware capacity, andcommunication (including the internet or "web“ andmobile communication). Explosion of digitized information require the solution tosignificant new research problems: support for multimedia objects and new data typesdistribution of informationnew database applicationsworkflow and transaction managementease of database management and useSeptember 11, 2012Databases and Data Mining43

“One Size Fits All”:An Idea Whose Time Has Come and Gone.M. Stonebraker, U. CetintemelProceedingsofThe 2005 International Conferenceon Data EngineeringApril 2005http://ww.cs.brown.edu/ ugur/fits all.pdfSeptember 11, 2012Databases and Data Mining44

DBDMS Services OverviewSeptember 11, 2012Databases and Data Mining45

DBDMS Services OverviewSeptember 11, 2012Databases and Data Mining46

DBMS: “One size fits all.”Single code line with all DBMS Services solves: Cost problem: maintenance costs of a single codeline Compatibility problem: all applications will runagainst the single code line Sales problem: easier to sell a single code linesolution to a customer Marketing problem: single code line has an easiermarket positioning than multiple code lineproductsSeptember 11, 2012Databases and Data Mining47

DBMS: “One size fits all.” To avoid these problems, all the major DBMSvendors have followed the adage “put all woodbehind one arrowhead”.In this paper it is argued that this strategyhas failed already, and will fail moredramatically off into the future.September 11, 2012Databases and Data Mining48

Data Warehousing In the early 1990’s, a new trend appeared: Enterpriseswanted to gather together data from multiple operationaldatabases into a data warehouse for business intelligencepurposes.A typical large enterprise has 50 or so operationalsystems, each with an on-line user community who expectfast response time.System administrators were (and still are) reluctant toallow business-intelligence users onto the same systems,fearing that the complex ad-hoc queries from these userswill degrade response time for the on-line community.In addition, business-intelligence users often want to seehistorical trends, as well as correlate data from multipleoperational databases. These features are very differentfrom those required by on-line users.September 11, 2012Databases and Data Mining49

Data WarehousingData warehouses are very different from OnlineTransaction Processing (OLTP) systems: OLTP systems have been optimized for updates, as themain business activity is typically to sell a good orservice.In contrast, the main activity in data warehouses is adhoc queries, which are often quite complex.Hence, periodic load of new data interspersed with adhoc query activity is what a typical warehouseexperiences.September 11, 2012Databases and Data Mining50

Data WarehousingThe standard wisdom indata warehouseschemas is tocreate a fact table:“who, what, when,where” about eachoperational transaction.September 11, 2012Databases and Data Mining51

Data Warehousing Data warehouse applications run much betterusing bit-map indexes OLTP (Online Transaction Processing) applicationsprefer B-tree indexes. materialized views are a useful optimization tacticin data warehousing, but not in OLTP worlds.September 11, 2012Databases and Data Mining52

Data WarehousingBitmapsAs a first approximation, mostvendors have a warehouse DBMS (bit-mapindexes, materialized views, starschemas and optimizer tactics forstar schema queries) ale015Male016Female10 OLTP DBMS (B-tree indexesand a standard cost-basedoptimizer), which are unitedby a common parserSeptember 11, 2012Databases and Data Mining53

Emerging ApplicationsSome other examples that show:Why conventional DBDMs willnot perform on the current emerging applications.September 11, 2012Databases and Data Mining54

Emerging Sensor Based Applications Sensoring Army Battalionof 30000 humans and12000 vehicles x.10 6sensorsMonitoring TrafficAmusements Park TagsHealth CareLibrary booksEtc.September 11, 2012Databases and Data Mining55

Emerging Sensor Based Applications There is widespreadspeculation thatconventional DBMSs willnot perform well on thisnew class of monitoringapplications.For example: Linear Road,traditional solutions arenearly an order ofmagnitude slower than aspecial purpose streamprocessing engineSeptember 11, 2012Databases and Data Mining56

Example: An existing application:financial-feed processingMost large financial institutions subscribe to feedsthat deliver real-time data on market activity,specifically: News consummated trades bids and asks etc.For example: Reuters Bloomberg InfodyneSeptember 11, 2012Databases and Data Mining57

Example: An existing application:financial-feed processingFinancial institutions have a variety of applicationsthat process such feeds. These include systemsthat produce real-time business analytics,perform electronic trading,ensure legal compliance of all trades to the variouscompany and SEC rulescompute real-time risk and marketexposure to fluctuations in foreign exchange rates.The technology used to implement this class ofapplications is invariably “roll your own”, becauseno good off-the-shelf system software productsexist.September 11, 2012Databases and Data Mining58

Example: An existing application:financial-feed processingExample Detect Problems in Streaming stock ticks: Specifically, there are 4500 securities, 500 ofwhich are “fast moving”.Defined by rules: A stock tick on one of these securities is late if itoccurs more than five seconds after the previoustick from the same security. The other 4000 symbols are slow moving, and atick is late if 60 seconds have elapsed since theprevious tick.September 11, 2012Databases and Data Mining59

Stream ProcessingSeptember 11, 2012Databases and Data Mining60

Performance The example application was implemented in theStreamBase stream processing engine (SPE) [5],which is basically a commercial, industrialstrength version of Aurora [8, 13].On a 2.8Ghz Pentium processor with 512 Mbytesof memory and a single SCSI disk, the workflow inthe previous figure can be executed at 160,000messages per second, before CPU saturation isobserved.In contrast, StreamBase engineers could only get900 messages per second from an implementationof the same application using a popularcommercial relational DBMS.September 11, 2012Databases and Data Mining61

Why?: Outbound vs Inbound ProcessingRDBMS(Outbound Processing)September 11, 2012StreamBase(Inbound Processing)Databases and Data Mining62

Inbound ProcessingSeptember 11, 2012Databases and Data Mining63

Outbound vs Inbound Processing DBMSs are optimized for outbound processing Stream processing engines are optimized forinbound processing. Although it seems conceivable to construct anengine that is either an inbound or an outboundengine, such a design is clearly a research project.September 11, 2012Databases and Data Mining64

Other Issues: Correct Primitives forStreams SQL systems contain a sophisticated aggregation system, whereby a user canrun a statistical computation over groupings of the records from a table in adatabase. When the execution engine processes the last record in the table, itcan emit the aggregate calculation for each group of records. However, streams can continue forever and there is no notion of “end oftable”. Consequently, stream processing engines extend SQL with the notion oftime windows. In StreamBase, windows can be defined based on clock time, number ofmessages, or breakpoints in some other attribute.September 11, 2012Databases and Data Mining65

Other Issues: Integration of DBMS Processingand Application Logic (1/2) Relational DBMSs were all designed to have clientserver architectures. In this model, there are many client applications,which can be written by arbitrary people, andwhich are therefore typically untrusted. Hence, for security and reliability reasons, theseclient applications are run in a separate addressspace from the DBMS.September 11, 2012Databases and Data Mining66

Other Issues: Integration of DBMS Processingand Application Logic (2/2) In an embedded processing model, it isreasonable to freely mix application logiccontrol logic andDBMS logicThis is what StreamBase does.September 11, 2012Databases and Data Mining67

Other Issues: High Availability It is a requirement of many stream-basedapplications to have high availability (HA) and stayup 7x24. Standard DBMS logging and crash recoverymechanisms are ill-suited for the streaming world The obvious alternative to achieve high availabilityis to use techniques that rely on Tandem-styleprocess pairs Unlike traditional data-processing applications thatrequire precise recovery for correctness, manystream-processing applications can tolerate andbenefit from weaker notions of recovery.September 11, 2012Databases and Data Mining68

Other Issues: Synchronization Traditional DBMSs use ACID transactions betweenconcurrent transactions submitted by multipleusers for example to induce isolation. (heavyweight) In streaming systems, which are not multi-user, aconcept like isolation can be effectively achievedthrough simple critical sections, which can beimplemented through light-weight semaphores.ACID Atomicity, Consistency, Isolation(transactions can be executed in isolation),DurabilitySeptember 11, 2012Databases and Data Mining69

One Size Fits All?September 11, 2012Databases and Data Mining70

One Size Fits All?Conclusions Data warehouses: store data by column ratherthan by row; read oriented Sensor networks: flexible light-way databaseabstractions, as TinyDB; data movement vs datastorage Text Search: standard RDBMS too heavy weightand inflexible Scientific Databases: multi dimensional indexing,application specific aggregation techniques XML: how to store and manipulate XML dataSeptember 11, 2012Databases and Data Mining71

The Fourth ParadigmeScienceand thewww.fourthparadigm.org(2009)September 11, 2012Databases and Data Mining72

Four Science Paradigms (J.Gray, 2007) Thousand years ago:science was empiricaldescribing natural phenomena Last few hundred years:theoretical branchusing models, generalizations Last few decades:a computational branch 2 a 4π Gρc2 Κ 2a 3a .simulating complex phenomena Today:data exploration (eScience)unify theory, experiment, and simulation Data captured by instrumentsOrgenerated by simulator Processed by software Information/Knowledge stored in computer Scientist analyzes database / filesusing data management and statisticsSeptember 11, 2012Databases and Data Mining73

The eScience ChallengeNovel Tools needed for: Data Capturing Data Curation Data Analysis Data Communication and PublicationInfrastructureSeptember 11, 2012Databases and Data Mining74

Gray’s LawsDatabase-centric Computing in ScienceHow to approach data engineering challengesrelated to large scale scientific datasets[1]: Scientific computing is becoming increasinglydata intensive.The solution is in a “scale-out” architecture.Bring computations to the data, rather than datato the computations.Start the design with the “20 queries.”Go from “working to working.”[1] A.S. Szalay, J.A. Blakeley, The 4th Paradigm, 2009September 11, 2012Databases and Data Mining75

VLDB 2010 J. Cho, H. Garcia Molina, Dealing with Web Data:History and Look ahead, VLDB 2010, Singapore,2010. D. Srivastava, L. Golab, R. Greer, T. Johnson, J.Seidel, V. Shkapenyuk, O. Spatscheck, J. Yates,Enabling Real Time Data Analysis, VLDB 2010,Singapore, 2010. P. Matsudaira, High-End Biological ImagingGenerates Very Large 3D and Dynamic Datasets,VLDB 2010, Singapore, 2010.September 11, 2012Databases and Data Mining76

VLDB 2011Keynotes T. O’Reilly, Towards a Global Brain,. D. Campbell, Is it still “Big Data” if it fits in mypocket?”,.Novel Subjects Social Networks,MapReduce (Hadoop) , Crowdsourcing, and MiningInformation Integration and Information RetrievalSchema Mapping, Data Exchange, Disambiguation ofNamed EntitiesGPU Based Architecture and Column-store indexingSeptember 11, 2012Databases and Data Mining77

VLDB 2012 Keynotes g the Spatial Web billions of queries/weekWeb objects that are near the location where query was issuedNew challenges on: Spatial web data management Relevance ranking based on text and location Low latencyData Science for Smart Systems Integration of information and control in complex systemsSmart bridges, transportation systems, health care systems,supply chains, etc.Data characteristics: heterogeneous, volatile, uncertainNew data management techniquesNew data analyticsSeptember 11, 2012Databases and Data Mining78

VLDB 201210 Year Best Paper Award Approximate Frequency Counts over Data StreamsG. Singh Manku (Google Inc. USA), R. MotwaniData Stream Algorithms research started late 90s.Sensor networksStock dataSecurity monotoringRecently: personal data stream analysisSeptember 11, 2012Databases and Data Mining79

VLDB2012 Subjects Spatial QueriesMap ReduceBig DataCloud DatabasesCrowdsourcingSocial Networks and Mobility in the CloudeHealthWeb databasesMobilityData Semantics and Data MiningParallel and Distributed DatabasesGraphsString and Sequence ProcessingPrivacyProbabilistic DatabasesData Flow; Hardware; Indexing; Query Optimization; Streams; September 11, 2012Databases and Data Mining80

VLDB 2012 Spatial QueriesT. Lappas et al.On theSpatiotemporalBurstiness of Terms. Burst identificationSpatial, temporalUnusual high frequency

September 11, 2012 Databases and Data Mining 2 DBDM Introduction Databases and Data Mining Projects at LIACS Biological and Medical Databases and Data Mining CMSB (Phenotype Genotype), DIAL CGH DB Cyttron: Visualization of the Cell GRID Computing VLe: Virtual Lab e-Science environments DAS3/DAS4 super computer Research on Fundamentals of Databases and Data