Final Verification And Validation (V&V) Report For RACER . - WBDG

3m ago
9 Views
2 Downloads
520.01 KB
78 Pages
Last View : 27d ago
Last Download : 1m ago
Upload by : Gia Hauser
Transcription

Final Verification and Validation (V&V)ReportforRACER Services and Verification andValidation (V&V)Contract Number: W91ZLK-07-D-0002Delivery Order: 0008Prepared for:U.S. Army Environmental CommandATTN: IMAE-CDP5179 Hoadley Road, Bldg E-4480Aberdeen Proving Ground, MD 21010-540123 September 2009

Table of Contents1. PROBLEM STATEMENT. 11.1Intended Use. 21.2Model Overview . 31.3Model Application . 41.4Accreditation Scope . 41.5V&V Scope . 52. MODEL REQUIREMENTS AND ACCEPTABILITY CRITERIA . 83. MODEL ASSUMPTIONS, CAPABILITIES, LIMITATIONS, & RISKS/IMPACTS. 103.1Model Assumptions. 103.2Model Capabilities . 103.3Model Limitations. 113.4Model Risks/Impacts . 114. V&V TASK ANALYSIS. 134.1Data V&V Task Analysis . 134.1.1 Data Verification. 134.1.2 Data Validation Task Analysis . 134.2Conceptual Model Validation Task Analysis . 134.3Design and Implementation Verification Task Analysis . 134.4Results Validation Task Analysis . 164.5V&V Reporting Task Analysis . 185. V&V RECOMMENDATIONS. 196. KEY PARTICIPANTS. 226.1Accreditation Participants . 226.2V&V Participants . 236.3Other Participants . 247. ACTUAL V&V RESOURCES . 257.1Planned V&V Tasking and Funding . 257.2Actual V&V Timeline . 26APPENDIX A MODEL DESCRIPTIONAPPENDIX B SIGNIFICANT CHANGES TO THE RACER SYSTEM SINCE RACER2002APPENDIX C BASIS OF COMPARISONAPPENDIX D REFERENCESAPPENDIX E ACRONYMSAPPENDIX F DISTRIBUTION LISTAPPENDIX G V&V PLANAPPENDIX H TEST INFORMATION

1.PROBLEM STATEMENTU.S. Government agencies use parametric models to estimate future environmentalcleanup costs; these cost estimates are then used as the basis for reporting outstandingenvironmental liabilities, as well as program and budget requirements. DefenseEnvironmental Restoration Program (DERP) Management Guidance (September 2001),requires that computer models used for estimating costs for environmental liabilitiesare verified, validated, and accredited in accordance with the requirements specified inDepartment of Defense Instruction (DoDI) 5000.61, “DoD Modeling and Simulation(M&S) Verification, Validation and Accreditation” (13 May 2003).The Remedial Action Cost Engineering and Requirements (RACER) cost estimatingsystem is a parametric cost estimating tool used to develop estimates of outstandingenvironmental liabilities. RACER was originally developed in 1991 in response to the1990 Chief Financial Officer’s (CFO) Act, which, along with subsequent legislation,required federal agencies to improve financial management and reporting, and toprovide accurate, complete, reliable, timely, and auditable financial information.Enhancements and new technologies have been added to the RACER system over thepast 18 years.1 The version of RACER proposed for accreditation, RACER 2008, is asingle-user desktop application developed using Microsoft (MS) Visual Basic (VB) 6.0and MS Access.M&S tools are classified by DoDI 5000.61 as either Common-use, General-use, or Jointuse. Additionally, DoDI 5000.61 states:“Each DoD Component shall be the final authority for validating representationsof its own forces and capabilities in common-, general-, or Joint-use M&Sapplications and shall be responsive to the other DoD Components to ensure itsforces and capabilities are appropriately represented.”DoDI 5000.61 defines three categories of M&S tools: Common-use M&S tools are “M&S applications, services, or materials provided by aDoD Component to two or more DoD Components.” General-use M&S are “Specific representations of the physical environment orenvironmental effects used by, or common to, many models and simulations; e.g., terrain,atmospheric, or hydrographic effects.” Joint-use M&S are “Abstract representations of joint and Service forces, capabilities,equipment, materiel, and services used in the joint environment by two, or more,Military Services.”1 Final Software Testing Plan, RACER 2008 Maintenance and Support, Earth Tech, Inc., Greenwood Village, CO, August 2007Final V&V ReportContract: W91ZLK-07-D-0002, TO 0008Page 1 of 27

The RACER software is categorized as a “Common-use M&S,” and is subject toVerification, Validation, and Accreditation (VV&A) standards of the fundingDepartment of Defense (DoD) component. In the case of RACER, VV&A activities aredually funded by the Army and Air Force, thus the following three regulations apply2to RACER Verification and Validation (V&V) activities: Air Force Instruction (AFI) 16-1001 Army Regulation (AR) 5-11 DoDI 5000.61.The purpose of this V&V report is to document verification and validation activities forthe RACER 2008 system in accordance with DoDI 5000.61, AFI 16-1001, and AR 5-11.1.1Intended UseU.S. Government agencies are required to develop estimates of outstandingenvironmental liabilities. RACER is a parametric cost estimating tool used to createthese estimates. The benefit of using RACER to create environmental liability estimatesis that it provides an automated, consistent, and repeatable method.In 2001, the Government engaged PricewaterhouseCoopers LLP to verify and validateRACER 2001, Version 3.0.0. Based on the 2001 V&V evaluation, Headquarters (HQ) AirForce Civil Engineer Support Agency (AFCESA) accredited RACER for the followingintended use:“To provide an automated, consistent, and repeatable method to estimate and document theprogram cost for the environmental cleanup of contaminated sites and to provide a reasonableestimate for program funding purposes consistent with the information available at the timeof the estimate preparation.”In the 1990s, Congress passed sweeping financial management reform legislationincluding the CFO Act of 1990, the Government Performance and Results Act (GPRA)of 1993, the Government Management Reform Act (GMRA) of 1994, and the FederalFinancial Management Improvement Act (FFMIA) of 1996. Such legislation aims toimprove financial management, promote accountability and reduce costs, andemphasize results-oriented management. These Acts require each executive agency toprepare and submit to the Director of the Office of Management and Budget a complete,accurate, and auditable financial statement for the preceding fiscal year. Environmentalliability estimates are one source of the financial information reported on agencies’annual financial statements as well as on the DoD Annual Report to Congress. As such,the environmental liability estimates must be accurate, complete, reliable, timely, andauditable.2 AFI 16-1001 and AR 5-11 are nearly identical to DoDI 5000.61; there are no conflicting instructions. The Performance WorkStatement (PWS) for this TO identifies requirements as written in AFI 16-1001; therefore, the standards, definitions, andprocesses as documented in AFI 16-1001 are referenced throughout this report.Final V&V ReportContract: W91ZLK-07-D-0002, TO 0008Page 2 of 27

Cost-to-complete (CTC) estimates form the basis of the environmental liability lineitems reported in the annual financial statements and must be updated annually.Environmental liabilities are reported on "Environmental Liabilities and EnvironmentalDisposal Liabilities," Note 14 to each Agency’s balance sheets. For the DoD agencies,RACER is one of the primary methods used to create standardized cost estimates forcurrent and future environmental liabilities.1.2Model OverviewRACER employs a patented parametric cost modeling methodology using over 113technology-specific cost models (technologies) that represent various applicationsrelated to site remediation.3 Each of the technologies is based on generic engineeringsolutions for environmental projects, technologies, and processes. These genericengineering solutions were derived from historical project information, industry data,government laboratories, construction management agencies, vendors, contractors, andengineering analysis. When creating an estimate in RACER, the user enters site-specificinformation to tailor the generic engineering solutions to reflect project-specificconditions and requirements. The tailored design is then translated into specificquantities of work, and the quantities of work are priced using current price data.Assemblies in the RACER database correlate with costs reported in the GovernmentCost Book, published by the Tri-Service Automated Cost Engineering Systems(TRACES) Committee.To aid in localizing RACER estimates, national average unit costs for assemblies in theRACER database are derived primarily based on the Government Cost Book (formerlythe Unit Price Book, or UPB). The area cost factor (ACF) for the estimate and a safetylevel cost adjustment are applied to calculate the adjusted unit price for each assemblyto arrive at the adjusted direct cost. Direct costs are marked up using a series of factorsrelating to various aspects of the work.Suggested changes to RACER are considered and processed according to the followingtwo plans: the Software Configuration Management Plan for RACER Software System(Version 4.0, dated February 26, 2003 - DRAFT) and the RACER Change ManagementPlan (Version 2.01, dated July 2007). The Configuration Management Plan applies tochanges to the structure of the software (source code, underlying data, requirements,model algorithms, software versioning, etc.), whereas the Change Management Plandescribes the relevant parties and their roles and responsibilities. The ChangeManagement Plan is one of three documents that describe the overall businessmanagement of RACER.43 There are 113 RACER cost models available to the standard RACER user. U.S. Air Force users that are approved to use theMilitary Munitions Response Program (MMRP) Supplemental Investigation technology have 114 cost models available.4 The three documents are (1) RACER Change Management Plan, (2) RACER Business Management Plan, and (3) RACERQuality Management Plan. Complete reference information is provided in Section 7 of this document.Final V&V ReportContract: W91ZLK-07-D-0002, TO 0008Page 3 of 27

Changes that have been approved are included in the annual software update. TheChange Management Plan provides a process where all participating federal agencieshave involvement and RACER continues development in a consistent manner to fulfillthe needs of actively participating agencies. All enhancements and revisions tosoftware, systems, processes, and documentation can be fully coordinated withparticipating federal agencies through use of the Change Management Plan.For RACER, Change Management involves identifying the configuration of workproducts and their descriptions at given point in time, employing a process tosystematically control changes to the configuration, and maintaining the integrity andtraceability of the configuration throughout the entire project lifecycle.1.3Model ApplicationThe RACER system is a cost estimating tool that can be applied to all phases ofremediation. It operates through a number of technology-specific cost models whichallow the user to input data which correlates with the anticipated work, resulting inassembly quantity calculations.The categories of remediation which can be estimated using the RACER system are: Pre-Studies Studies Removal Action/Interim Corrective Measures Remedial Design/Corrective Measures Design Remedial Action/Corrective Measure Implementation Operations and Maintenance Long Term Monitoring Site Close-outAfter completing the estimate, users can generate a wide variety of reportsdocumenting the estimated cost for the project. Additionally, estimates can beimported into the U.S. Army Environmental Command (USAEC) and the U.S. ArmyCorps of Engineers (USACE) management systems. Generating reports and importingestimate information into management systems are the two most common methodsused by agencies for documenting and tracking CTC information.1.4Accreditation ScopeThe following excerpt from the RACER V&V Plan5 prepared by Earth Tech, Inc. (May2008) provides the following justification for accreditation of RACER:5 RACER Verification & Validation Plan, Earth Tech, Inc., Greenwood Village, CO, May 2008Final V&V ReportContract: W91ZLK-07-D-0002, TO 0008Page 4 of 27

“There are four primary reasons for getting RACER Accredited. The first three reasons listeddeal with meeting regulatory requirements. The final reason listed deals with increasingconfidence in decision making. The Air Force Audit Agency found that RACER did not conform to DoDI 5000.61 –DoD Modeling and Simulation Verification, Validation, and Accreditation”. DoDI 5000.61 requires that M&S used to support the major DoD decision makingorganizations and processes (DoD Planning, Programming, and Budgeting System)shall be accredited for that use. (AF) 16-1001 requires accreditation. Increases credibility in the M&S outputs and reduces the risk of using the M&S. Overallthis increases the confidence level of decisions made based on the outputs.”The RACER system has undergone a number of changes since the 2001 V&V evaluationand system accreditation. A listing of the changes to the RACER system from 2002 to2008 is included in Appendix B of this document, and in Appendix B of the FinalRACER V&V Plan.5 The Final RACER V&V Plan also focuses on the current state of thecost models and other RACER functionality.Recent RACER releases have included the elimination of obsolete cost models and thedevelopment of new cost models. Available reports have also been expanded. Themost frequently used models were re-engineered for RACER 2008 based on thecollection of and comparison to historical project cost data. The default markuptemplate and the markup process were completely redefined as well.Each release of RACER includes updated assembly prices, area cost factors, per diemrates, and escalation factors. The RACER 2008 release includes extensive redefinitionand updating of assembly costs using information from the 2006 version of theGovernment Cost Book. Each assembly has been defined using Cost Book line itemsthat improve documentation and maintainability of cost data. Except for assemblies forwhich costs are provided by USACE or the Air Force, all assemblies were defined usingCost Book line items. Previous RACER releases included a mix of assemblies definedusing the Cost Book and assemblies that relied on other data sources. Some assemblieshave no pre-defined unit cost, but were priced when used in a model (for example,Other Direct Costs and per diem).1.5V&V ScopeV&V activities, their results, and recommendations are included in this report. Thisdocument will be maintained by the V&V Manager as part of the M&S VV&A history,and used to support current and future accreditation decisions, feasibility assessments,and future enhancements to RACER.Final V&V ReportContract: W91ZLK-07-D-0002, TO 0008Page 5 of 27

The following definitions, presented by the DoD M&S Coordination Office (M&S CO),are utilized in DoDI 500.61, AFI 16-1001, and AR 5-11.Verification1. The process of determining that a model implementation and its associateddata accurately represent the developer's conceptual description andspecifications.2. The process of determining that a model or simulation faithfully representsthe developer's conceptual description and specifications. Verificationevaluates the extent to which the model or simulation has been developedusing sound and established software and system engineering techniques.Verification is performed by the Verification Agent. For RACER, this is the ArmyRACER Point of Contact (POC).6The goal of the Verification portion of the V&V was to evaluate RACER and itsunderlying cost models to determine whether it correctly and completely represents thedeveloper’s conceptual description and specifications. Verification activities for the2008 version of the RACER software were performed by the software developmentcontractor, Earth Tech, Inc. with oversight and approval provided by the RACERTechnical Review Group (TRG). Earth Tech, Inc.7 was awarded two task ordersthrough the U.S. AFCESA which included annual maintenance and support as well asreengineering of thirteen cost models. Under these task orders Earth Tech, Inc. alsomaintained and updated the internal control documents listed in Section 7.0 of thisreport. Both task orders included verification activities.Validation1. The process of determining the degree to which a model and its associateddata are an accurate representation of the real world from the perspective ofthe intended uses of the model.2. The process of determining the fitness of a model or simulation and itsassociated data for a specific purpose.Validation is performed by the Validation Agent. For RACER 2008, this is the ArmyRACER POC.6The primary objective of the validation portion of the V&V was to provide sufficientdocumentation to support validation of the RACER 2008 cost models and underlyingdatabases by documenting a comparison of RACER-generated costs against associatedactual historical costs for current technologies. On September 25, 2008, USAEC6 Guidance for Verification and Validation (V&V) of Remedial Cost Engineering and Requirements Software, March 20067 The RACER 2008 software developer was Earth Tech, Inc.; Earth Tech, Inc. is now known as AECOM.Final V&V ReportContract: W91ZLK-07-D-0002, TO 0008Page 6 of 27

awarded a contract to Booz Allen Hamilton (Booz Allen) (W91ZLK-07-D-0002, TaskOrder (TO) 0008) to “validate the RACER 2008 (version 10.0.2) cost models andunderlying databases.” The contract directs Booz Allen to “document comparison ofRACER-generated costs with associated actual project costs on present models and oncecomparisons are completed, a new V&V report will be developed.”8 The opportunity tocompare actual project costs with RACER cost estimates represents a best practice in thedevelopment of parametric models and will allow continued enhancement of RACER asa calibration tool.To compare RACER 2008 cost models (technologies) against actual project cost data,project information was collected from a variety of Government offices. The types ofproject information collected included technical reports and contracting documents forenvironmental remediation projects executed by the Government within the past fiveyears. Under the USAEC TO, and in support of Validation activities, Booz Allentraveled to four Government offices to collect project information. In addition, BoozAllen conducted similar visits in 2007 and 2008 under a TO of a contract issued by theAir Force Center for Engineering and the Environment (AFCEE).98 Contract Order W91ZLK-07-D-000 TO 0008 page 5, dated September 25, 20089 Global Engineering, Integration, and Technical Assistance 2005 (GEITA05), FA8903-05-D-8729 TO 0372 (Mod 2, dated 19August 2008))Final V&V ReportContract: W91ZLK-07-D-0002, TO 0008Page 7 of 27

2.MODEL REQUIREMENTS AND ACCEPTABILITY CRITERIAVerificationSoftware testing must follow approved methods and standards; also, when tested, themodels must meet these design specifications. For RACER, the Software Testing Plan10describes the testing process for the software; the Software Test Results Report11describes the results of the three phases of testing (alpha, beta, and final acceptance).The testing goals, as outlined in the Software Testing Plan, are shown in Table 2.0below. These goals also serve as acceptability criteria for the verification portion of theV&V.# Allowed in AlphaBuild# Allowed in BetaBuild# Allowed inReleasedVersion320NecessaryNo stated goalNo stated goal0Cosmetic1263DefectClassificationCriticalTable 2.0. Defect Goals for RACER testing, as Stated in the Software Testing Plan12ValidationThe Tri-Service Parametric Model Specification Standard’s13 purpose is to establishcriteria and standards for developing and updating parametric cost models like thoseused in RACER. The ranges of accuracy, as stated by the Association for theAdvancement of Cost Engineering (AACEI), and as also reported in the Tri-ServiceParametric Model Specification Standard, for preliminary (order of magnitude),secondary (budget), and definitive estimates are displayed in the Table 2.1, below.DescriptionPreliminary (Order of Magnitude)Secondary (Budgetary)Definitive 50% 30% 15%Rangetototo- 30%- 15%- 5%Table 2.1. AACEI Ranges of Accuracy10 Final Software Testing Plan, RACER 2008 Maintenance and Support, Earth Tech, Inc., Greenwood Village, CO, August200711 Software Test Results Report for RACER 2008, Final Acceptance Testing Results, Earth Tech, Inc., Greenwood Village, CO,October 200712 Section 3.1.2 of the Final Software Testing Plan13 Tri-Service Parametric Model Specification Standard, Project Time & Cost, Inc., April 1999Final V&V ReportContract: W91ZLK-07-D-0002, TO 0008Page 8 of 27

Per the Tri-Service Parametric Model Specification Standard:“Due to the lack of information in environmental remediation work a parametric costmodel would be used as a Preliminary or Order of Magnitude Estimate and should beevaluated as such. However, in some instances, including more complicated models thatinvolve secondary parameters, it may be contained in the Secondary or Budget Estimatecategory.”13Therefore, the acceptability criteria for the validation portion of the V&V are thatRACER estimates should fall within -30% and 50% of actual costs.Final V&V ReportContract: W91ZLK-07-D-0002, TO 0008Page 9 of 27

3.MODEL ASSUMPTIONS, CAPABILITIES, LIMITATIONS, &RISKS/IMPACTS3.1Model AssumptionsThe system uses a patented methodology to generate parametric cost estimates.RACER technologies are based on generic engineering solutions for environmentalprojects, technologies, and processes. These solutions, and the resulting RACERestimates, are constrained by the amount and the accuracy of the project data gatheredto create each of the cost models in the software. The project data used to supportmodel (technology) development (“Historical Cost Data”) is collected by the softwaredevelopment contractor, reviewed by the RACER TRG, and incorporated into a“technology addendum.” A “technology addendum” is created by the softwaredevelopment contractor for each RACER cost model, and reviewed for accuracy by theRACER TRG.The accuracy of RACER estimates is further constrained by several additional factors: The User. The user preparing the estimate must be knowledgeable (i.e., officiallytrained) in the use of the RACER software. What Was Known About the Project. The user must know, at a minimum, all ofthe “required parameters” to be entered into each cost model. If assumptions aremade about the values of required parameters, the accuracy of the assumptionswill impact the accuracy of the resulting estimate. Inaccurate Use of the Software. Individual users will inevitably segregateproject components differently. One user might, for example, add individualassemblies to account for waste disposal; a different user might employ theResidual Waste Management technology to account for these costs; a third usermight employ the Load & Haul technology. Agencies can increase consistencyamongst estimates by ensuring all of its users are uniformly trained andknowledgeable about the RACER software. Changes in Project Scope. RACER estimates are designed to be point-in-timeestimates. If the project scope changes between estimate preparation and projectexecution, the accuracy of the estimate may be subject to change. Changes in Design Standards. The RACER software is continually updated toincorporate field-proven techniques for environmental remediation. Newertechnologies, unique approaches, and experimental methods are not available asparametric models in RACER. If a project employs such techniques, the projectmay not accurately be estimated in RACER.3.2Model CapabilitiesIn 2001, the Government engaged PricewaterhouseCoopers LLP to verify and validateRACER 2001, Version 3.0.0. Based on the 2001 V&V evaluation, HQ AFCESAaccredited RACER for the following intended use:Final V&V ReportContract: W91ZLK-07-D-0002, TO 0008Page 10 of 27

“To provide an automated, consistent, and repeatable method to estimate and document theprogram cost for the environmental cleanup of contaminated sites and to provide a reasonableestimate for program funding purposes consistent with the information available at the timeof the estimate preparation.”For the 2008 version of the software addressed in this report, the intended useremains the same.3.3Model LimitationsThe accuracy of the RACER models is constrained by the following: The amount of project data gathered to create each of the cost models in thesoftware The accuracy of project data gathered to create each of the cost models in thesoftware The accuracy of the algorithms employed in each RACER model The accuracy of the data used to populate the parameters of each cost model The training level/knowledge of the user preparing the estimate The methodology employed by the user to segregate project components andcorrelate those components to individual RACER cost models Whether the remediation technologies employed in the actual project areavailable for cost modeling in the RACER software.RACER creates a point-in-time estimate based on generic engineering solutions that isknown at the time. Unknowns can contribute to decreased accuracy.3.4Model Risks/ImpactsThe risk associated with developing and utilizing RACER for its intended use (creationof parametric cost estimates) is that the estimates will not be accurate enough to meetthe standard for a preliminary estimate (-30%/ 50%), as described in Section 2 of thisdocument.Verification involves testing the software to ensure that it is functioning as intendedand producing the associated documentation defining procedures, algorithms, etc. Therisk associated with not performing this testing is that problems will be difficult toidentify and correct without the proper testing and documentation.Validation allows the opportunity to compare actual project costs with RACER costestimates, and to verify the soundness of the generic engineering solutions presented inthe algorithms of the RACER software. The risk associated with not performingvalidation activities is that there is then no benchmark to be used to evaluate theaccuracy of the system.Final V&V ReportContract: W91ZLK-07-D-0002, TO 0008Page 11 of 27

Overall, VV&A represents a best practice in the development of parametric models andwill allow continued enhancement of RACER as a calibration tool.Final V&V ReportContract: W91ZLK-07-D-0002, TO 0008Page 12 of 27

4.V&V TASK ANALYSIS4.1Data V&V Task AnalysisThe documents listed in Section 7.0 of this document are used as internal controls toguide design, development, revisions, verification, and validation of the RACERsoftware. These documents are updated and revised on an ongoing basis; thedocuments listed in Section 7.0 are the versions current at the time of release of RACER2008. The V&V tasks described below are found to be in conformance with thesedocuments.4.1.1 Data VerificationAll underlying costs utilized in the RACER System correlate with costs reported in theGovernment Cost Book, published by the TRACES Committee. Area cost factors(ACFs), a separate type of underlying data, are published by the Office of the Secretaryof Defense (OSD). Values for Markups (including General Conditions, Overhead,Profit, Prime Markup on Sub, Risk, and Owner Cost) are described in the Final TechnicalMemorandum Evaluation of the Markup Template for RACER 2008.144.1.2 Data Validation Task AnalysisAll underlying costs utilized in the RACER System correlate with costs reported in theGovernment Cost Book, published by the TRACES Committee. Area cost factors ACFs,a separate type of underlying data, are published by OSD. Values for Markups(including General Conditions, Overhead, Profit, Prime Markup on Sub, Risk, andOwner Cost) ar

to RACER Verification and Validation (V&V) activities: Air Force Instruction (AFI) 16-1001 Army Regulation (AR) 5-11 DoDI 5000.61. The purpose of this V&V report is to document verification and validation activities for the RACER 2008 system in accordance with DoDI 5000.61, AFI 16-1001, and AR 5-11. 1.1 Intended Use

Related Documents:

new approaches for verification and validation. 1.1. Role of Verification and Validation Verification tests are aimed at "'building the system right," and validation tests are aimed at "building the right system." Thus, verification examines issues such as ensuring that the knowledge in the system is rep-

verification and validation. 1.2 PURPOSE This System Validation and Verification Plan provide a basis for review and evaluation of the effectiveness of the AIV program and its proposed elements. In addition it is an input to the lower level verification. In this document it is proposed a scenario for the full requirements traceability throughout a

Validation of standardized methods (ISO 17468) described the rules for validation or re-validation of standardized (ISO or CEN) methods. Based on principles described in ISO 16140-2. -Single lab validation . describes the validation against a reference method or without a reference method using a classical approach or a factorial design approach.

Cleaning validation Process validation Analytical method validation Computer system validation Similarly, the activity of qualifying systems and . Keywords: Process validation, validation protocol, pharmaceutical process control. Nitish Maini*, Saroj Jain, Satish ABSTRACTABSTRACT Sardana Hindu College of Pharmacy, J. Adv. Pharm. Edu. & Res.

Verification and Validation of an Agent-based Simulation Model Nadine Shillingford Gregory Madey Ryan C. Kennedy Department of Computer Science and Engineering University of Notre Dame Notre Dame, IN 46556 nshillin@nd.edu, gmadey@nd.edu, rkenned1@nd.edu Abstract Verification and validation (V&V) are two important

- Validation (§ 117.160) - Verification that monitoring is being conducted - Verification that corrective action decisions are appropriate - Verification of implementation and effectiveness (§ 117.165) Calibration, product testing, environmental monitoring, review of records - Reanalysis

Independent Verification and Validation (IV&V) Verification by independent authorities necessary for but not limited to requirements that are safety-critical or of high-security nature Independent verification and validation is defined by three parameters: Technical, Managerial und Financial Independence

3 Annual Book of ASTM Standards, Vol 01.01. 4 Annual Book of ASTM Standards, Vol 01.03. 5 Annual Book of ASTM Standards, Vol 01.05. 6 Annual Book of ASTM Standards, Vol 03.01. 7 Annual Book of ASTM Standards, Vol 03.03. 8 Available from Manufacturers’ Standardization Society of the Valve and Fittings Industry, 1815 N. Fort Myer Drive, Arlington, VA 22209. 9 Available from American Society of .