Practical Software Measurement: Measuring For Process .

11Practical SoftwareMeasurement: Measuringfor Process Managementand ImprovementWilliam A. FloracRobert E. ParkAnita D. CarletonApril 1997GUIDEBOOKCMU/SEI-97-HB-003

GuidebookCMU/SEI-97-HB-003April 1997Practical Software Measurement:Measuring for Process Management and ImprovementWilliam A. FloracRobert E. ParkAnita D. CarletonSoftware Engineering Measurement and AnalysisUnlimited distribution subject to the copyrightSoftware Engineering InstituteCarnegie Mellon UniversityPittsburgh, PA 15213

This report was prepared for theSEI Joint Program OfficeHQ ESC/AXS5 Eglin StreetHanscom AFB, MA 01731-2116The ideas and findings in this report should not be construed as an official DoD position. It is published in theinterest of scientific and technical information exchange.FOR THE COMMANDER(signature on file)Thomas R. Miller, Lt Col, USAFSEI Joint Program OfficeThis work is sponsored by the U.S. Department of Defense.Copyright 1997 by Carnegie Mellon University.Permission to reproduce this document and to prepare derivative works from this document for internal use isgranted, provided the copyright and “No Warranty” statements are included with all reproductions andderivative works.Requests for permission to reproduce this document or to prepare derivative works of this document forexternal and commercial use should be addressed to the SEI Licensing Agent.NO WARRANTYTHIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIALIS FURNISHED ON AN “AS-IS” BASIS. CARNEGIE MELLON UNIVERSITY MAKES NOWARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTERINCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE ORMERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL.CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITHRESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT.This work was created in the performance of Federal Government Contract Number F19628-95-C-0003 withCarnegie Mellon University for the operation of the Software Engineering Institute, a federally fundedresearch and development center. The Government of the United States has a royalty-free governmentpurpose license to use, duplicate, or disclose the work, in whole or in part and in any manner, and to have orpermit others to do so, for government purposes pursuant to the copyright license under the clause at 52.2277013.This document is available through Research Access, Inc., 800 Vinial Street, Suite C201, Pittsburgh, PA 15212.Phone: 1-800-685-6510. FAX: (412) 321-2994. RAI also maintains a World Wide Web home page. The URL ishttp://www.rai.comCopies of this document are available through the National Technical Information Service (NTIS). Forinformation on ordering, please contact NTIS directly: National Technical Information Service, U.S.Department of Commerce, Springfield, VA 22161. Phone: (703) 487-4600.This document is also available through the Defense Technical Information Center (DTIC). DTIC providesaccess to and transfer of scientific and technical information for DoD personnel, DoD contractors andpotential contractors, and other U.S. Government agency personnel and their contractors. To obtain a copy,please contact DTIC directly: Defense Technical Information Center / Attn: BRR / 8725 John J. KingmanRoad / Suite 0944 / Ft. Belvoir, VA 22060-6218. Phone: (703) 767-8274 or toll-free in the U.S. — 1-800 2253842).Use of any trademarks in this report is not intended in any way to infringe on the rights of the trademarkholder.

It is not surprising to encounter not just a few, but manyindividuals who have been entrusted with continuousimprovement responsibilities who cannot define an in-controlprocess, who cannot accurately distinguish between processcontrol and process capability, who cannot distinguishbetween process capability and product capability, who do notunderstand the basic structure of a control chart, who do nothave practical knowledge of the fundamental theorem ofstatistical process control, and who do not understand thesignificance and relative importance of various signals ofspecial causes of variation.Robert Hoyer & Wayne Ellis, 1996

Table of ContentsAcknowledgmentsixPrefacexiGuidebook Roadmapxiii1The Role of Measurement in Process Management1.1 Why Measure?1.2 Process Measures Are Driven by Goals and Issues1.3 What Is a Software Process?1.4 What Is Software Process Management?1.5 Responsibilities and Objectives of Software Process ManagementDefine the ProcessMeasure the ProcessControl the ProcessImprove the Process1.6 The Relationship of Process Management to Project Management1.7 Integrating Measurement with Software Process Management1125678991011122The Perspectives of Process Measurement2.1 Performance2.2 Stability2.3 Compliance2.4 Capability2.5 Improvement and Investment1517202428303Planning Measures for Process Management3.1 Identifying Process IssuesSteps for Identifying Process IssuesThe Role of Mental ModelsCommon Process Issues3.2 Selecting and Defining MeasuresSelecting Process MeasuresDefining Process MeasuresCriteria for Operational DefinitionsExamples of Operational DefinitionsCreating Your Own Definition Frameworks3.3 Integrating Measures with the Software ProcessAnalysis of Existing Measurement ActivitiesDiagnosis of Existing MeasuresAction to Integrate 97-HB-003i

Tasks for Defining Your Measurement ProcessesAction Plans45455Applying Measures to Process Management—Part 1: Collecting andRetaining Data4.1 General Principles4.2 Collecting DataProcess Performance DataProcess Compliance Data4.3 Retaining DataDatabase Planning IssuesDatabase Operation IssuesDatabase Management Issues5758596061626365655Applying Measures to Process Management—Part 2: Analyzing Data5.1 Separating Signals from Noise5.2 Evaluating Process StabilityThe Importance of StabilityStability Concepts and PrinciplesThe Structure of Control ChartsThe Distinction Between Variables Data and Attributes DataDetecting Instabilities and Out-of-Control SituationsThe Stability Investigation Process5.3 Control Charts for Variables DataX-Bar and Range ChartsIndividuals and Moving Range Charts for Continuous DataIndividuals and Moving Range Charts for Discrete Data5.4 Frequency Histograms and Natural Process Limits5.5 Control Charts for Attributes DataDistributional Models and Their Relationships to Chart TypesU ChartsZ ChartsXmR Charts for Attributes Data5.6 Evaluating Process CapabilityCapability HistogramsFraction NonconformingCapability IndicesSpecification TolerancesA Procedure for Assessing Process 1081081101121121156Applying Measures to Process Management—Part 3: Acting on theResults6.1 General PrinciplesThe Additional Roles of MeasurementThe Need for Additional Information117117117119iiCMU/SEI-97-HB-003

6.2Establishing StabilityFinding and Correcting Assignable CausesNoncompliance as an Assignable CauseImproving the ProcessWhere to Look When Seeking ImprovementsEffects of Changing a Process: Look Before You LeapAfter the Change: Examine the ResultsConditions for SuccessTools for Finding Root Causes and SolutionsScatter DiagramsRun ChartsCause-and-Effect DiagramsHistogramsBar ChartsPareto ChartsTechnologies and Methodologies for Changing or Improving 135138141142More About Analysis and Use of Process Measures7.1 Statistical Inference as a Basis for ActionEnumerative StudiesAnalytic Studies7.2 Reviewing and Assessing Collected DataCriterion 1: VerifiedCriterion 2: SynchronousCriterion 3: Self ConsistentCriterion 4: Valid7.3 Assessing Predictive Validity7.4 Control LimitsWhy 3 Sigma?The Central Limit Theorem and the Role of the NormalDistributionGetting Started: Constructing Control Charts with Limited DataRevising and Updating Control LimitsTesting for and Sustaining Statistical Control7.5 The Problem of Insufficient Granularity in Recorded Values7.6 Rational Sampling and Rational SubgroupingRational SamplingRational Subgrouping7.7 Aggregation and Decomposition of Process Performance Data7.8 World-Class 184iii

89Principles of Successful Process Measurement8.1 Process Measures Are Driven by Business Goals8.2 Process Measures Are Derived from the Software Process8.3 Effective Measurement Requires Clearly Stated OperationalDefinitions8.4 Different People Have Differing Measurement Views and Needs8.5 Measurement Results Must Be Examined in the Context of theProcesses and Environments That Produce Them8.6 Process Measurement Spans the Full Life Cycle8.7 Retaining Data Provides Factual Baselines for Future Analyses8.8 Measures Are the Basis for Objective Communication8.9 Aggregating and Comparing Data Within and Across ProjectsRequires Care and Planning8.10 Structured Measurement Processes Improve Data 92193Appendix A: Locating Key Information195Appendix B: Template for a Measurement Action Plan199Appendix C: Example—A Practitioner’s ReportC.1 BackgroundC.2 ProblemC.3 Selecting a Set of DataC.4 Data Analysis—Round 1C.5 Data Analysis—Round 2C.6 Round 2 Wrap-UpC.7 Data Analysis—Round 3C.8 Round 3 Wrap-UpC.9 ConclusionsC.10 Future 7Index223ivCMU/SEI-97-HB-003

List of FiguresFigure1-1:Business Goals, Project and Process Issues, and RelatedMeasurable Attributes4Figure1-2:Definition of Process6Figure1-3:The Four Key Responsibilities of Process Management8Figure1-4:The Relationship Between Process Management and ProjectManagement11Measurement Activities and Their Relationship to the KeyResponsibilities of Process Management12Figure1-5:Figure2-1:The Control-Process Responsibility15Figure2-2:Guidelines for Selecting Product and Process Measures19Figure2-3:The Concept of Controlled Variation21Figure2-4:The Concept of Uncontrolled or Assignable Cause Variation22Figure2-5:Control Chart for the Backlog of Unresolved Problems24Figure2-6:Examples of Entities and Attributes that Can Be Measured toAddress Process Compliance26Figure2-7:Inspection Process Use27Figure2-8:Inspection Use by Document Type27Figure2-9:Histogram Reflecting Process Capability28Figure 2-10:Aligning Process Performance to Process Requirements29Figure 2-11:Lynch and Cross’s Performance Pyramid for Business OperatingSystems30Figure 2-12:Examples of Indicators for Business Operating Systems31Figure 2-13:Sources of Early Indicators for Customer Satisfaction, Flexibility,and Productivity31Measurement Planning and the Key Responsibilities of ProcessManagement33Figure3-1:Figure3-2:The Principal Activities of Measurement Planning33Figure3-3:A Generic Process Model36Figure3-4:A Simple Process Model for Defect Tracking36Figure3-5:Example of a Control-Flow Diagram37Figure3-6:Measurement Planning Activities—Step 239Figure3-7:Measurable Entities in a Software Process43Figure3-8:Measurable Attributes Associated with Software Process Entities44Figure3-9:A Checklist-Based Definition for Counting Defects (page 1 of 2)49Measurement Planning Activities—Step 351Figure 3-10:CMU/SEI-97-HB-003v