A Practitioner View Of CMMI Process Performance Models

Background Information on the DataHistorical data on escaped defects, by type, across lifecyclephases was recorded.pFor each historical project, software size was recorded, aswell,ll tto hhelpl normalizeli ththe ddefectsf t iinjectedj t d andd ffound,dthereby producing injection and detection rates.Robert StoddardRusty Young 2008 Carnegie Mellon University38

A modern spreadsheet for escapeddefect analysis before beingtransformed into a CMMI ProcessPerformance ModelRobert StoddardRusty Young 2008 Carnegie Mellon University39

Let’s look at thematrix showing“Phase Injected”vs “Phase Found”Robert StoddardRusty Young 2008 Carnegie Mellon University40

For example, anaverageg of 2000design defectswere found duringthe design activityRobert StoddardRusty Young 2008 Carnegie Mellon University41

Let’s look at the“Phase Containment”& “Phase Screening”ratesRobert StoddardRusty Young 2008 Carnegie Mellon University42

Here, 2080 Requirementsand Design defects werecaught during DesignHere, 2000 Design defectswere caughtg duringg Designg% of all defects entering andinjected in Design caught inDesign% of Design defects caughtin DesignRobert StoddardRusty Young 2008 Carnegie Mellon University43

Let’s look at the PhaseInjection and EscaperatesRobert StoddardRusty Young 2008 Carnegie Mellon University44

Here, 4200 Design defects were injected with2200 of them escaping the Design activity;Additionally, 2450 total defects (injectedduring Design or inherited from upstreamactivities) escaped past the Design activityRobert StoddardRusty Young 2008 Carnegie Mellon University45

Here, 36% of all defects in a project are expected to be Designdefects; 52% of Design defects are expected to escape pastDesign; and 54% of all types of defects in the Design activity(i j t d during(injectedd i DesignD i or inheritedi h it d fromfupstreamtactivities)ti iti )are escaping past the Design activityRobert StoddardRusty Young 2008 Carnegie Mellon University46

Now, let’s transformthis spreadsheetmodel into a validCMMI processperformance model!Robert StoddardRusty Young 2008 Carnegie Mellon University47

Robert StoddardRusty Young 2008 Carnegie Mellon University48

Robert StoddardRusty Young 2008 Carnegie Mellon University49

Each of the greencells have receiveduncertaintydistributions basedon historical dataRobert StoddardRusty Young 2008 Carnegie Mellon University50

Robert StoddardRusty Young 2008 Carnegie Mellon University51

Each of these bluecells were identified asoutcomes whoseresulting distributionswill be studiedRobert StoddardRusty Young 2008 Carnegie Mellon University52

Each of these blue cells wereidentified as outcomes whoseresulting distributions will be studiedRobert StoddardRusty Young 2008 Carnegie Mellon University53

Robert StoddardRusty Young 2008 Carnegie Mellon University54

StandardsimulationsummaryresultsRobert StoddardRusty Young 2008 Carnegie Mellon University55

We are 95%Wconfident that nomore thanapprox. 4,786Design defectswill be injectedby a projectRobert StoddardRusty Young 2008 Carnegie Mellon University56

We are 95%confident that nomore than 39%of all types ofdefects will beDesign defectsRobert StoddardRusty Young 2008 Carnegie Mellon University57

We are 95%confident that nomore than 22,351351Design defectswill escape theDesign activityRobert StoddardRusty Young 2008 Carnegie Mellon University58

We are 95%Wconfident that nomore than 61% ofDesign defects willescape the DesignactivityRobert StoddardRusty Young 2008 Carnegie Mellon University59

We are 95%confident that nomore than 2,607defects (injected( jduring Design orinherited fromupstream activitiesof Design) willescape the Designactivityi iRobert StoddardRusty Young 2008 Carnegie Mellon University60

We are 95%confident that nomore than 62% ofthe total defects(injected duringDesign or inheritedfrom upstreamactivities of Design)will escape theDesign activityRobert StoddardRusty Young 2008 Carnegie Mellon University61

We are 95%W9 % confidentfidthat no less than 1,499total defects (injected( jduring Design orinherited fromupstream activities ofDesign) will be foundduring the Designactivityti itRobert StoddardRusty Young 2008 Carnegie Mellon University62

We are 95% confidentthat no less than 38%of total defects(injected during Designor inherited fromupstream activities ofDesign) will be foundduring the DesignactivityRobert StoddardRusty Young 2008 Carnegie Mellon University63

We are 95% confidentthat no less than 47Design defects will befound during theDesign activityRobert StoddardRusty Young 2008 Carnegie Mellon University64

We are 95% confidentthat no less than 39% ofDesign defects will befound during the Designactivityti itRobert StoddardRusty Young 2008 Carnegie Mellon University65

We are 95% confidentthat no more than 46% ofall defects will escape atleast one phaseRobert StoddardRusty Young 2008 Carnegie Mellon University66

We are 95% confidentthat escaping defectswill not escape, onaverage more than 1average,1.88phasesRobert StoddardRusty Young 2008 Carnegie Mellon University67

Using SensitivityAnalysis, we canlearn more aboutwhich factors inour model aremostt contributingt ib tito our outcome.Robert StoddardRusty Young 2008 Carnegie Mellon University68

What Have We Accomplished?We transformed a model that used only historical averagesand substituted uncertaintyy distributions for each of theinjection and found ratesBy doingBd i thithis, we can establisht bli h confidentfid t conclusionsl iaboutb tour outcomes using their resulting distributions: Defect Injectionjrates byy Phase Phase Containment and Screening EffectivenessWe alsoWl usedd sensitivityiti it analysisl i tto ddecideid whichhi h ffactorstttotackle first to improve each outcomeRobert StoddardRusty Young 2008 Carnegie Mellon University69

Planning theRequirements Buildup 2008 Carnegie Mellon University

Generate PlanGenerate schedule based on higher level PPMs which helpdetermine milestones and variation base slackGenerate detailed PPMs to predict performance during thisphaseNote: these steps will be repeated periodically (such as atphase or other selected milestones) and on an as neededbasisRobert StoddardRusty Young 2008 Carnegie Mellon University71

Elicitation – Requirements Buildup – Predicted120Generalized Logistic (or Richard's)10080 60 40 20 0x time.timeA controls the lower asymptote,C controls the upper asymptote,M controlst l ththe titime off maximumigrowth,B controls the growth rate, andT controls where maximum growthoccurs - nearer the lower or upperasymptote1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26Robert StoddardRusty Young 2008 Carnegie Mellon University72

Elicitation – Requirements Buildup – Predicted120Generalized Logistic (or Richard's)100These typically would befunctions with controllable factorssuch as elicitation method (JAD,60prototyping, reengineering, etc.),team experience,p, domain40experience, # staff, etc.80 20 0x time.timeA controls the lower asymptote,C controls the upper asymptote,M controlst l ththe titime off maximumigrowth,B controls the growth rate, andT controls where maximum growthoccurs - nearer the lower or upperasymptote1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26Robert StoddardRusty Young 2008 Carnegie Mellon University73

Elicitation – Requirements Buildup – Predicted120Generalized Logistic (or Richard's)10080Calibration of this model for60 different domains, customers, etc.and the effects of the controllablefactors is critical40 20 0x time.timeA controls the lower asymptote,C controls the upper asymptote,M controlst l ththe titime off maximumigrowth,B controls the growth rate, andT controls where maximum growthoccurs - nearer the lower or upperasymptote1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26Robert StoddardRusty Young 2008 Carnegie Mellon University74

Risk in PlanUse PPMs to judge overall risk in the planMay use Monte Carlo simulation in the schedule to betterunderstand Schedule based sources of risk EffectsEffoff risksi k on theh scheduleh d lRobert StoddardRusty Young 2008 Carnegie Mellon University75

Elicitation – Requirements Buildup – Monitor120 100 80 6040 2001 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26Monitor the buildupFlattening means you arereaching the point of diminishingreturns for elicitationSignificant difference betweenpredicted and actual upperasymptote indicate a potentialmisunderstanding of the systemto be builtIf actuals show significantvariation from predicted, re fitcurve for new prediction– Calculate an appropriatePrediction Interval (PI) to aid indetection of anomalous conditionsRobert StoddardRusty Young 2008 Carnegie Mellon University76

Elicitation – Requirements Buildup – Example 22324-20Robert StoddardRusty Young 2008 Carnegie Mellon University7725

Elicitation – Requirements Buildup – Example 2a120100The Process is notperforming at its’ historicalLevels. Investigate andcalibrate 4-20Robert StoddardRusty Young 2008 Carnegie Mellon University7825

Elicitation – Requirements Buildup – Example 2b1401201008060The refit curve hasextended the predictedbuildup time byapproximately 20% (31vs 627282930-20Robert StoddardRusty Young 2008 Carnegie Mellon University7931

Elicitation – Requirements Buildup – NotesRequires a strong consistent requirements elicitationpprocess Different standard curves for different elicitation processes such asJAD, prototyping, etc. Curve shape parameters can be influenced by context -- rereengineering vs green field, well understood vs non-well understooddomain, experience, etc.Consider a measurement systems error analysis – perhapsusing Gage R&R to ensure consistent buildup countsRequires a good prediction of sizeCan be beneficial with good size prediction and then fittingthe curve as you gather data It will take time before variation in the buildup time minimizesRobert StoddardRusty Young 2008 Carnegie Mellon University80

Another ExampleProcess PerformanceModel (PPM) in theRequirements Phase 2008 Carnegie Mellon University

The Situation in the Requirements PhaseOur products are comprised of 40-60 featuresWe assign each feature a small development team todevelop the feature “cradle to grave”These feature teams operate in overlapping lifecycles withinan overall product incremental waterfall lifecycle model (thus(thus,different features will be added in each new increment)OUR NEED: A PPM that will let each feature team predictthe number of requirements defects to be experiencedthroughout the lifecycle of the feature developmentRobert StoddardRusty Young 2008 Carnegie Mellon University82

Details of the Requirements Phase PPMThe outcome, Y, is the predicted number of Requirementsgiven feature teamdefects for a gThe x factors used to predict the Requirements defects are:x1: Req’ts Volatility (continuous data)x2: Risk of Incomplete Req’ts (nominal data)x3:3 RiRiskk off AAmbiguousbiRReq’ts’t ((nominali lddata)t )x4: Risk of Non-Testable Req’ts (nominal data)x5: Risk of Late Req’tsReq ts (nominal data)Robert StoddardRusty Young 2008 Carnegie Mellon University83

Background Information on the DataWe collected historical data (of the Y and the x’s) for a largevolume of feature teamsFor the x2 thru x5 factors, the feature team leader wouldhi t i ll checkhistoricallyh k offff as ““yes”” or ““no”” ddependingdi on whetherh ththey felt that the specific risk significantly impacted thefeature team cost, schedule or qualityOperational definitions and training were conducted toensure consistencyi tandd repeatabilityt bilit among ththe ffeaturetteam leadsRobert StoddardRusty Young 2008 Carnegie Mellon University84

Development of the Req’ts Phase PPM - 1All of the risks are ratedX1: Volatilityeither 0 or 1, with 0 beingThe Y outcomeshown inthe absence of the riskIs the Numberdecimal formand 1 being the presenceOf Reqtsof the riskDefectsRobert StoddardRusty Young 2008 Carnegie Mellon University85

Development of the Req’ts Phase PPM - 2Robert StoddardRusty Young 2008 Carnegie Mellon University86

Development of the Req’ts Phase PPM - 3Thi willThisill accomplishli hDummy VariableRegression to handleX factors that areContinuous andDiscreteRobert StoddardRusty Young 2008 Carnegie Mellon University87

Development of the Req’ts Phase PPM - 4Robert StoddardRusty Young 2008 Carnegie Mellon University88

Development of the Req’ts Phase PPM - 5Robert StoddardRusty Young 2008 Carnegie Mellon University89

Development of the Req’ts Phase PPM - 6Robert StoddardRusty Young 2008 Carnegie Mellon University90

Intended Use of this Req’ts PPMOnce we decide on the final form of the PPM, we will use itprimaryy ways:yin two p1)At the beginning of each feature team kickoff, the teamwillill anticipateti i t ththe valueslffor ththe x ffactorst( 1 x5).(x15)They will evaluate the PPM at these values to predict thenumber of Req’ts defects. If this prediction isunacceptable, they will take immediate action to addressone or more of the x factors.2)During the development, the feature team willperiodically re-assess the anticipated values of the xfactors and repeatpthe actions of stepp 1 above.Robert StoddardRusty Young 2008 Carnegie Mellon University91

Updating this Req’ts PPMAs more feature teams develop features, they will continueto record the data for the x factors and the resultinggYoutcome of number of Req’ts DefectsWhen a group of feature teams have finished the lifecycleand have recorded their data, the organization may chooseto add their data to the existing data set and then repeat theexercise of developingp g the dummyy variable regressiongequation.Ultimately, the organization may want to segment thefeature teams by type and conduct this analysis for eachsegment.Robert StoddardRusty Young 2008 Carnegie Mellon University92

An Example ProcessPerformance Model(PPM) during theDesign Phase 2008 Carnegie Mellon University

The Situation in DesignThe Design team is faced with modifying legacy software inp g new software.addition to developingA major issue that can have disastrous effects on projects isth ideatheid off “b“brittleness”ittl” off software.ftInI a nutshell,t h ll softwareftbecomes more “brittle” over time as it is changed andexperiences a drifting usage model.OUR NEED: A PPM used by each feature team duringd i tto predictdesigndi t hhow “b“brittle”ittl ” softwareftiis, andd subsequentlybtlto make the correct design decisions regarding degree ofmodification vs rewrite from scratch.Robert StoddardRusty Young 2008 Carnegie Mellon University94

Details of the Software Brittleness PPMThe outcome, Y, is the measure of software brittleness,g ),measured on an arbitraryy scale of 0 ((low)) to 100 ((high),which will be treated as continuous dataThe x factorsThf tusedd iin thithis predictiondi ti examplel are ththefollowing: Unit ppath complexitypyUnit data complexityNumber of times the unit code files have been changedNumber of unit code changes not represented in Design documentupdatesRobert StoddardRusty Young 2008 Carnegie Mellon University95

Background Information on the DataWe collected historical data from feature teams on theircode units. The data,, related to the first four x factors,, aremaintained by the CM system using automated toolstracking this data each time new code file versions arechecked in. Unit path complexity Unit data complexity NumberN b off titimes ththe unitit coded filfiles hhave bbeen changedhdWe also have access to problem reporting and inspectiondatabases which pprovide us with a number of issuesreported against individual code units. Finally, we have“Brittleness” values for each unit of code that were assignedby a different empirical exercise with domain experts.Robert StoddardRusty Young 2008 Carnegie Mellon University96

Development of the Brittleness PPM - 1Robert StoddardRusty Young 2008 Carnegie Mellon University97

Development of the Brittleness PPM - 2Robert StoddardRusty Young 2008 Carnegie Mellon University98

Development of the Brittleness PPM - 3Robert StoddardRusty Young 2008 Carnegie Mellon University99

Development of the Brittleness PPM - 4Robert StoddardRusty Young 2008 Carnegie Mellon University100

Development of the Brittleness PPM - 5Robert StoddardRusty Young 2008 Carnegie Mellon University101

Development of the Brittleness PPM - 6Robert StoddardRusty Young 2008 Carnegie Mellon University102

Intended Use of this Brittleness PPM - 1Once we decide on the final form of the PPM, we will use itprimaryy ways:yin two p1)As each feature team begins software design, the teamwill collect the x factor information for the software unitsto be worked on, and then evaluate the PPM at thesevalues to predict the

JMP is interactive, comprehensive, visual software from SAS. It . Reader Input Input & Output Inquiry Report Sales Report Output Output Store System Inventory Database CRUD Logical File Interface Input & Output Cash Register Input Bar Code Label Maker Output Output Cas