November 2013 ALSO INSIDE - Telliant

Really Big DataDr. Dobb’s JournalNovember 2013ALSO INSIDEDo All Roads Lead Back to SQL? UnderstandingWhat Big DataCan DeliverApplying the Lambda Architecture From the Vault:Easy Real-Time Big Data Analysis UsingStorm www.drdobbs.com

More on DrDobbs.comJolt Awards: The Best BooksFive notable books every serious programmer should read.Dr. Dobb’s JournalCONTENTSNovember 2013COVER ARTICLE8 Understanding WhatBig Data Can Deliver23 From the Vault: Easy Real-TimeBig Data Analysis Using StormBy Aaron KimballBy Shruthi Kumar and Siddharth PatankarIt’s easy to err by pushing data to fit a projected model. Insightscome, however, from accepting the data’s ability to depict what isgoing on, without imposing an a priori bias.If you're looking to handle big data and don't want to traverse the Hadoop universe, you might well find that usingStorm is a simple and elegant solution.GUEST EDITORIAL3 Do All Roads Lead Back to SQL?6 News BriefsBy Seth ProctorRecent news on tools, platforms, frameworks, and the stateof the software development world.After distancing themselves from SQL, NoSQL products are moving towards transactional models as “NewSQL” gains popularity.What happened?By Adrian Bridgwater7 Open-Source DashboardA compilation of trending open-source projects.FEATURES15 Applying the Big Data Lambda Architecture 34 LinksBy Michael HausenblasA look inside a Hadoop-based project that matches connections insocial media by leveraging the highly scalable lambda architecture.www.drdobbs.comSnapshots of interesting items on drdobbs.com including alook at the first steps to implementing Continuous Deliveryand developing Android apps with Scala and Scaloid.November 2013http://www.drdobbs.com/240162065A Massively Parallel Stack for Data AllocationDynamic parallelism is an important evolutionary stepin the CUDA software development platform. With it,developers can perform variable amounts of workbased on divide-and-conquer algorithms and in-memorydata structures such as trees and graphs — entirelyon the GPU without host duction to Programming with ListsWhat it’s like to program with immutable lists.http://www.drdobbs.com/240162440Who Are Software Developers?Ten years of surveys show an influx of younger developers, more women, and personality profiles at oddswith traditional stereotypes.http://www.drdobbs.com/240162014Java and IoT In MotionEric Bruno was involved in the construction of the Internet of Things (IoT) concept project called “IoT InMotion.” He helped build some of the back-end components including a RESTful service written in Javawith some database queries, and helped a bit with thefront-end as well.http://www.drdobbs.com/2401621892

[GUEST EDITORIAL]IN THIS ISSUEGuest Editorial News Open-Source Dashboard What Big Data Can Deliver Lambda Storm Links Table of Contents Do All Roads Lead Back to SQL?After distancing themselves from SQL, NoSQL products are moving towards transactional models as“NewSQL” gains popularity. What happened?By Seth Proctoruch has been made in the past several years about SQLversus NoSQL and which model is better suited to modern, scale-out deployments. Lost in many of these arguments is the raison d’être for SQL and the difference between model and implementation. As new architectures emerge, thequestion is why SQL endures and why there is such a renewed interestin it today.MBackgroundIn 1970, Edgar Codd captured his thoughts on relational logic in a paper that laid out rules for structuring and querying data(http://is.gd/upAlYi). A decade later, the Structured Query Language(SQL) began to emerge. While not entirely faithful to Codd’s originalrules, it provided relational capabilities through a mostly declarativelanguage and helped solve the problem of how to manage growingquantities of data.Over the next 30 years, SQL evolved into the canonical data-management language, thanks largely to the clarity and power of its underlying model and transactional guarantees. For much of that time,deployments were dominated by scale-up or “vertical” architectures,in which increased capacity comes from upgrading to bigger, individwww.drdobbs.comNovember 2013ual systems. Unsurprisingly, this is also the design path that most SQLimplementations followed.The term “NoSQL” was coined in 1998 by a database that providedrelational logic but eschewed SQL (http://is.gd/sxH0qy). It wasn’t until2009 that this term took on its current, non-ACID meaning. By then, typical deployments had already shifted to scale-out or “horizontal” models. The perception was that SQL could not provide scale-out capability,and so new non-SQL programming models gained popularity.Fast-forward to 2013 and after a period of decline, SQL is regainingpopularity in the form of NewSQL (http://is.gd/x0c5uu) implementations. Arguably, SQL never really lost popularity (the market is estimated at 30 billion and growing), it just went out of style. Either way,this new generation of systems is stepping back to look at the last 40years and understand what that tells us about future design by applying the power of relational logic to the requirements of scale-out deployments.Why SQL?SQL evolved as a language because it solved concrete problems. Therelational model was built on capturing the flow of real-world data. If apurchase is made, it relates to some customer and product. If a song is3

[GUEST EDITORIAL ]IN THIS ISSUEGuest Editorial News Open-Source Dashboard What Big Data Can Deliver Lambda Storm Links Table of Contents played, it relates to an artist, an album, a genre, and so on. By definingthese relations, programmers know how to work with data, and the system knows how to optimize queries. Once these relations are defined,then other uses of the data (audit, governance, etc.) are much easier.Layered on top of this model are transactions. Transactions areboundaries guaranteeing the programmer a consistent view of the“Many NoSQL systems tout the benefit of having no(or a limited) schema. In practice, developers still need somecontract with their data to be effective”database, independent execution relative to other transactions, andclear behavior when two transactions try to make conflicting changes.That’s the A (atomicity), C (consistency), and I (isolation) in ACID. To saya transaction has committed means that these rules were met, andthat any changes were made Durable (the D in ACID). Either everythingsucceeds or nothing is changed.Transactions were introduced as a simplification. They free developers from having to think about concurrent access, locking, or whethertheir changes are recorded. In this model, a multithreaded service canbe programmed as if there were only a single thread. Such programming simplification is extremely useful on a single server. When scalingacross a distributed environment, it becomes critical.With these features in place, developers building on SQL were able tobe more productive and focus on their applications. Of particular importance is consistency. Many NoSQL systems sacrifice consistency for scalwww.drdobbs.comNovember 2013ability, putting the burden back on application developers. This tradeoff makes it easier to build a scale-out database, but typically leaves developers choosing between scale and transactional consistency.Why Not SQL?It’s natural to ask why SQL is seen as a mismatch for scale-out architectures, and there are a few key answers. The first is that traditionalSQL implementations have trouble scaling horizontally. This has led toapproaches like sharding, passive replication, and shared-disk clustering. The limitations (http://is.gd/SaoHcL) are functions of designingaround direct disk interaction and limited main memory, however, andnot inherent in SQL.A second issue is structure. Many NoSQL systems tout the benefit ofhaving no (or a limited) schema. In practice, developers still need somecontract with their data to be effective. It’s flexibility that’s needed —an easy and efficient way to change structure and types as an application evolves. The common perception is that SQL cannot providethis flexibility, but again, this is a function of implementation. Whentable structure is tied to on-disk representation, making changes tothat structure is very expensive; whereas nothing in Codd’s logicmakes adding or renaming a column expensive.Finally, some argue that SQL itself is too complicated a language fortoday’s programmers. The arguments on both sides are somewhatsubjective, but the reality is that SQL is a widely used language with alarge community of programmers and a deep base of tools for taskslike authoring, backup, or analysis. Many NewSQL systems are layeringsimpler languages on top of full SQL support to help bridge the gapbetween NoSQL and SQL systems. Both have their utility and their usesin modern environments. To many developers, however, being able to4

[GUEST EDITORIAL ]IN THIS ISSUEGuest Editorial News Open-Source Dashboard What Big Data Can Deliver Lambda Storm Links Table of Contents reuse tools and experience in the context of a scale-out databasemeans not having to compromise on scale versus consistency.Where Are We Heading?The last few years have seen renewed excitement around SQL.NewSQL systems have emerged that support transactional SQL, builton original architectures that address scale-out requirements. Thesesystems are demonstrating that transactions and SQL can scale whenbuilt on the right design. Google, for instance, developed F1(http://is.gd/Z3UDRU) because it viewed SQL as the right way to address concurrency, consistency, and durability requirements. F1 is specific to the Google infrastructure but is proof that SQL can scale andthat the programming model still solves critical problems in today’sdata centers.Increasingly, NewSQL systems are showing scale, schema flexibility,and ease of use. Interestingly, many NoSQL and analytic systems arenow putting limited transactional support or richer query languagesinto their roadmaps in a move to fill in the gaps around ACID and declarative programming. What that means for the evolution of these systems is yet to be seen, but clearly, the appeal of Codd’s model is asstrong as ever 43 years later.— Seth Proctor serves as Chief Technology Officer of NuoDB Inc. and has more than15 years of experience in the research, design, and implementation of scalable systems.His previous work includes contributions to the Java security framework, the Solarisoperating system, and several open-source projects.www.drdobbs.comNovember 20135