Agile Data Warehousing Incorporating Agile Principles

3EXECUTIVE REPORTcompany’s existing IT staff hadother high-priority responsibilities,the development team did nothave extensive knowledge of thebusiness or existing operationalsystems. The development teamdid, however, have open accessto business and technical expertswithin the company as well astechnology experts from thecompany’s software vendor.While initial discovery effortswere challenging, all stakeholdersparticipated extensively.Customer role. The company’sfinance division filled the roleof the primary “customer” forthe new data warehouse, andthe CFO sponsored the project.Together, the customer team andthe CFO had a relatively focusedbusiness goal of gaining more reliable access to revenue and profitability information. They also hada substantial amount of existingreports used in business analysison a routine basis, offering a reasonable basis for requirementsanalysis.Project management. CorporateIT handled project managerresponsibilities using traditionalProject Management Institute(PMI) practices. The IT group wassimultaneously involved in twoother large development projects,both of which had direct or indirect impact on the scope of thedata warehouse project.Hosted environment. Due to limited resources and infrastructure,the company’s IT leadership had 2004 CUTTER CONSORTIUMrecently decided to partner withan ASP to provide hosting servicesfor newly developed productionsystems. The data warehouse wasexpected to reside at the hostingfacility located on the West Coastof the US, while company headquarters were located on the EastCoast. Because legacy systemsremained on the corporate infrastructure, separate geographiclocations for the warehouse andheadquarters created complications — though not insurmountable ones — for the movement oflarge volumes of data.Project OutcomeThe original project plan called foran initial data warehouse launchwithin 90 days. As any experienced data warehouse developerknows, such a deadline imposesan ambitious, but not impossible,schedule (assuming appropriatescope definition). But the datawarehouse launch for this projectcame a full eight months afterproject start. User acceptancetesting did not go well. Userswere already annoyed with project delays, and when they finallysaw the promised features, therewas a significant gap betweenuser expectations and the endproduct. As is common with lateprojects, staff members wereadded to the development teammidstream to try and get the project back on track. So project costsfar exceeded the planned budget,and the project was placed onhold until further planning couldbe done to justify continueddevelopment.RetrospectiveSo who was to blame? Usersthought that developers hadmissed the mark and failed toimplement all their requirements.Developers believed that userexpectations had not been properly managed and thus projectscope had grown out of control.Project sponsors thought that thevendor and the consulting firmhad overpromised and underdelivered. The vendor and consultingfirm believed that internal politicsand organizational issues were toblame. Finally, many members ofthe company’s IT staff felt threatened by their own lack of ownership on the project and secretlycelebrated the failure.The project degenerated into aseries of meetings to review contracts and project documents todetermine who should be heldresponsible. And guess what?Everyone involved was partiallyto blame. In addition to the common technical challenges of dataextraction, integration, and cleansing, the following were identifiedas root causes of project failure:n The project contract did notsufficiently define the projectscope.n Requirements were incomplete, vague, and open-ended.n There were conflicting interpretations of the previouslyVOL. 4, NO. 12

4approved requirements anddesign documents. Developers put in long nightsand weekends in a chaoticattempt to respond to userchanges and new demands. Developers did not fully understand user requirements orexpectations and did notmanage requirementschanges well. Users had significant misconceptions about the purpose ofa data warehouse since theprevailing understanding of awarehouse was based on theprevious reporting application(which was not a good model). Vendors made ambitiouspromises that developers couldnot fulfill in the time available. The project manager did notmanage user expectations. IT staff withheld importantinformation from developers. The ASP partner did not provide the level of connectivityand technical support thatdevelopers expected.Hindsight truly is 20/20, and in thewaning days of this project, several realities became apparent:1. A higher degree of interactionbetween developers, users,stakeholders, and internal ITexperts would have ensuredaccurate understandingbetween all participants.2. Early and frequent workingsoftware, no matter howVOL. 4, NO. 12BUSINESS INTELLIGENCE ADVISORY SERVICERegardless of who is to blame,the root cause of data warehousingproject failure is a disconnectbetween user and developerexpectations.simplistic, would have greatlyreduced users’ misconceptionsand increased the accuracy oftheir expectations.3. Greater emphasis on usercollaboration would havehelped to avoid conflictinginterpretations of requirements.4. A project plan that focusedon adapting to changes ratherthan on meeting a fixed andarbitrary deadline would havegreatly improved user satisfaction with the end product.In the end, and regardless of whois to blame, for this data warehousing project and many otherfailures, the root cause is a disconnect between user and developer expectations.ADW BACKGROUND(Ken Collier)Until 2003, upon meetingAgileAglliance cofounder JimHighsmith, I was only superficiallyaware of the Alliance, a group ofthought leaders from the softwareengineering community. I hadfocused on advancements in datawarehousing and BI rather thanon software engineering. Butafter meeting Jim, I began to seethe connection between agileprinciples and data warehousingpractices.Jim and I, along with another colleague, met weekly to share coffee, experiences, and ideas. Jimwould talk about his Agile ProjectManagement (APM) principles asthey unfolded, and I would lamentmy most recent data warehouseproject for being understaffed andoverscoped and for having everchanging user requirements. Mydevelopment team would workovertime, and yet users werenever satisfied.But during a weekly meeting,I had an “aha!” experience: itdawned on me that all this agilestuff that Jim and others havetalked and written about has adirect application to data warehousing and BI in general. I realized it made sense to establish aset of values, principles, and practices for infusing agility into all BIprojects, from canned reportingto data visualization, data miningand quantitative analytics, and ofcourse, data warehousing. I havesince put ADW principles intopractice and am enthusiasticabout the results.Using ADW practices, I was ableto lead three developers with littleexperience in data warehousingto complete a data warehouse intwo and a half months withoutour team having to pull a singleall-nighter. Although our resultingdata warehouse will mature overtime, the complete end-to-endarchitecture has been established,and multiple OLAP reports areavailable for use. Author LukeHohmann likens early-stage agilewww.cutter.com

5EXECUTIVE REPORTprojects to babies: they are complete but immature. This is a useful way to think about ADW. Ourgoal is to get a working systemin the hands of users as early aspossible so that we can start getting feedback and improving thesystem.In the spirit of full disclosure,ADW borrows unabashedly fromthe works of agile pioneers suchas Ambler, Beck, Martin Fowler,Highsmith, Hohmann, CutterConsortium Senior ConsultantsAlistair Cockburn and KenSchwaber, and others. My contribution is the adaptation ofthese thought leaders’ ideas toindustry-standard data warehousing best practices. Using my ownknowledge and experience, andwith much coaching from Jim,my aim is to establish a practical,applicable, and more successfuldata warehouse developmentalternative.ADW AND APM(Jim Highsmith)Ken and I met less than two yearsago and were introduced by amutual friend. Many of our earlyconversations concerned politics,restaurants, and especially skiing,but eventually discussion turnedto the project Ken was workingon and my work in agile softwaredevelopment and project management. As Ken talked about hisdata warehousing project and others that exhibited similar waterfallcharacteristics, we discussed how 2004 CUTTER CONSORTIUMan agile approach might havemitigated some of the problems.Since then, and as discussed inthis report, Ken began to use agilepractices — and in particular,APM practices — with successfulresults. We have also workedtogether implementing agiledevelopment and project management with a joint client on adata warehouse–like project. As Ilearned more about data warehousing and BI from Ken, and helearned more about agile development from me, we came to themutual conclusion that agile practices could solve some of the illsplaguing traditional data warehouse and BI projects.Historically, these projects arethe epitome of big design up front.Even worse, the usual implementation sequence for these projectsleaves user reporting until late inthe project — after all the datastaging, cleansing, reporting database development, and other“technically” challenging portionsof the project are complete. Inmany cases, unfortunately, whenusers finally see the results, theydon’t like what they see. Doublyunfortunate, by the time userssee the results that fall short oftheir expectations, most of theproject’s resources have beenspent and options have becomelimited. With the noose on theproject tightening, the blamegame begins, because everyonehas already lost.Ken and I believe that combiningagile practices with data warehouse development has tremendous potential to ensure thatthese projects deliver value tocustomers early and often in aproject. In place of a huge projectthat goes on for 12 to 18 monthsbefore users have access to valuable results, this approach has thepotential to deliver results in justa few months, with subsequentquarterly deliveries.Note: Ken has authored the bulkof this report; I’ve chipped in hereand there, especially in the APMsection below.APMAn Agile SynopsisIn 2001, the AgileAlliance outlineda set of core values, principles,and practices for developing software that better meets users’needs and expectations. The mostsignificant outgrowth of the firstAgileAlliance workshop in 2001was a statement of shareddevelopment values called theManifesto for Agile SoftwareDevelopment [2], which states:We are uncovering betterways of developing softwareby doing it and helping others do it. Through this workwe have come to value:Individuals and interactions over processesand toolsWorking software over comprehensive documentationVOL. 4, NO. 12

6Customer collaborationover contract negotiationResponding to change overfollowing a planThat is, while there is valuein the items on the right[roman typeface], we valuethe items on the left [boldtypeface] more. [2]For practitioners like me withtechnical roots in the rigors of prescriptive methodologies, the AgileManifesto is as powerful as it issimplistic. After years of practicing“heavyweight process” development, I find these agile values liberating. In my experience, whenpush comes to shove, we instinctively adopt the values expressedin the Agile Manifesto. Unfortunately, when push comes toshove, we have often crossedthe boundary between order andchaos. By adopting these valuesfrom the outset of a project, wecan maintain order from thebeginning and gain satisfactionin the knowledge that our time isspent productively on creatingworking software.From Agile Principles toBusiness ObjectivesAPM is characterized by envisioning and exploring processes ratherthan planning and doing processes. APM addresses projectsin which the solution is unknownin the beginning and the projectteam must explore the problemand solution space to createa product that meets the customer’s vision and deliversVOL. 4, NO. 12BUSINESS INTELLIGENCE ADVISORY SERVICEcustomer value. When a projecthas significant risk and uncertainty, it requires an envisionàexplore process. Explorationincludes experimenting withdifferent solutions to see whichones work.Early on in high exploration–factorprojects, the team may not knowthe complete feature list. It mayhave a reasonable product vision,various business constraints (e.g.,target schedule, target costs), ageneral idea of key features, andsome ideas about the overallarchitectural skeleton, but thedetails will emerge as the teamdelivers completed features everycouple of weeks. The team’sprocess is one of evolution andadaptation, not planning and optimization; the process embraceschange.While most people recognizeindustry volatility, they have nottruly modified either their mindsetor their management practices toacknowledge change. For example, the perceived high cost ofchange has driven methodologiesto strive to limit change by engaging in extensive front-end planningand analysis. The problem is thatno matter how thorough this earlywork, stuff happens; change happens. So rather than allow thehigh cost of change to dictateprocess, agile developers focuson reducing costs; low-costchange enables a developmentcycle of minimum up-front workfollowed by a succession of shortdevelopment iterations. When weadequately reduce the cost ofchange, the entire economics ofsoftware development changes;the process shifts from one basedon anticipation (define, design,and build) to one based on adaptation (envision, explore, andrefine).While planàdo processes workfor low exploration–factor projects, they are no match for today’svolatile business environment. Sosoftware development and projectmanagement processes must begeared toward mobility, experimentation, and speed. But first,they must be geared toward business objectives.Reliable InnovationThere are five key businessobjectives for agile softwaredevelopment and APM: (1) continuous innovation (delivering oncurrent customer requirements);(2) product adaptability (delivering on future customer requirements); (3) reduced deliveryschedules (meeting market windows and improving ROI); (4)people and process adaptability(responding rapidly to productand business change); and (5)reliable results (supporting business growth and profitability).Continuous InnovationDeveloping new software applications, such as those for data warehousing and BI, requires amindset that fosters innovation.www.cutter.com

7EXECUTIVE REPORTBusiness value isn’t derived fromthese applications by followingprescribed paths but by blazingnew trails.While many organizations strivefor standardization of processand practices, they shouldstrive for consistency.Product AdaptabilityNo matter how well a data warehouse application is developed,the future always brings surprises.As an enterprise changes, as newexecutives use the application,as business processes evolve, asnew markets are pursued, theneeds of data warehouse and BIapplications change as well. Theonly way to survive is to strive foradaptability — a critical design criterion for a data warehouse application. Agile technical practicesfocus on reducing the cost ofchange (or adaptation) as anintegral part of the developmentprocess.Reduced Delivery SchedulesReducing delivery schedulesremains a high-priority businessgoal for executives, but providingincremental business value is justas important. Would you ratherhave a complete data warehouseapplication in 18 months or incremental data marts every threemonths? Further, incrementaldelivery can significantly increasethe ROI of projects as well as generate valuable feedback from thedevelopment process throughincremental releases. APM contributes to reducing deliveryschedules in two key ways:focus and streamlining. 2004 CUTTER CONSORTIUMFirst, the constant focus on features and the priority for theirrelease in short, iterative timeboxes forces teams (comprisingcustomers and developers) toconsider both the number of features to include and the depth ofthose features. This reduces theoverall workload by eliminatingmarginally beneficial features.Second, APM streamlines thedevelopment process by concentrating on value-adding activitiesand eliminating overhead.People and Process AdaptabilityHave you ever worked on twoprojects that were identical: thatis, they shared the same people,same problem, same constraints,same customers, and the sameexecutives? Why, then, do manyorganizations insist that the sameprocesses and practices shouldapply to every project? Whilemany organizations strive forstandardization of process andpractices, they should strive forconsistency. What we need inmost instances is a consistentframework with local variationsin process and practices toaccommodate differences amongprojects. We must also buildadaptable teams whose membersare comfortable with change andview change as integral to thrivingin a dynamic project environmentrather than as an obstacle.Reliable ResultsManufacturing processes aredesigned to be repeatable and todeliver the same result time aftertime. They are predictable andtherefore repeatable. However,because of the uncertainty andrisk, exploration processes aredifferent. They can deliver on acustomer’s vision and accordingto a customer’s requirements asthese requirements evolve, butthey can’t deliver a completelyprespecified result. APM deliversconsistent, reliable results.The APM FrameworkAs shown in Figure 1, the APMframework supports the businessobjectives of continuous innovation, product adaptability, reduceddelivery schedules, people andprocess adaptability, and reliableresults through its five phases:envision, speculate, explore,adapt, and close.In practice, projects have twoiterative cycles of collaborativeplanning and collaborative development, as shown in Figures 2and 3. In APM, as in all agile methods, both planning and development are done collaboratively.The collaborative planning cycle,which focuses on product vision,project scope and boundaries,and the overall project releaseplan, can — and should — beused throughout the project whenVOL. 4, NO. 12

8BUSINESS INTELLIGENCE ADVISORY sFinalproductCloseCloseFigure 1 — Highsmith’s Agile Project Management framework. (Source: [7].)ProductvisionIterationplandoing so is the product visionbox. The team designs a “box”in which the application will bepackaged, which forces the teamto focus on exactly who the customer is and the key three orfour features that will sell theapplication. The second aspect ofvisioning is to create a picture ofthe entire project community —of customers, product managers,project team members, and stakeholders — and how these partiesintend to work together. Otherspecific practices in the envisionphase include (1) generatingproduct architecture and guidingprinciples, (2) creating the projectdata sheet, and (3) process andpractice tailoring.SpeculateP r o je c t D a t a S h e e tP ro je c t N a m e:P ro je c t S t a rt D a te :C lie nt s:P r o je ct Le ad er :E x e c u tiv e S p on s o r:C li e n t B e n e fits :P ro je c t O b j ec tive S tat e m en t:P e r fo r m an ce /Q ua lity A tt ri b u te s :F oc u s M at ri x :E xc e lI m pr o veAc ce p tT a rg e tS c op eS c he d u l eD e fec tsR e so u rc eF ea tu r es : ( A bil ity to S tat em en ts )A r ch ite ct u re :M ajor M ile sto n e s:Is s u e s/ R is k s :Date1.2.3.4.1 9 92 -9 7 K n o w led ge S t ru c t ures , In c.Cycle 0Architecture1.0Cycle 1ArchitectureArchit ecture1.11.1Cycle 2Cycle 3Cycle 4Archit ectureArchitecture1.21.2Feature 1Requri ement sRequirementsCardscardsF e a t u re /C o m p o ne n t Req u ir e m en t s Car dF e at ur e/ Co m pone nt ID :F e at ur e/ Co m pone nt N am e :F e at ur e/ Co m pone nt T yp e:F e at ur e/ Co m pone nt D es c rip tio n :P lann ed Cy c le :Feature 3F ea t u r e/ Com po n en t R eq u ire m e n ts C ardFe at ure /Co m po n en t I D:Fe at ure /Co m po n en t N am e :Fe at ure /Co m po n en t T y pe :Fe at ure /Co m po n en t D es cr ipt i on:E s t. Wo rk E f fort :R equ irem ent s Un c ert ain ty (H ,

data warehouse implemented, data warehouse developers, and IT executives whose job it is to oversee such projects. By combin-ing experience in data warehous-ing (Dr. Ken Collier) and in agile development practices (Jim Highsmith), the authors have developed a new approach to data warehousing projects called agile data warehousing (ADW).