Validation Workshop - Validation Overview

Validation Workshop – Validation OverviewAug. 24, 2005 at NFSTCPresentation OutlineIntroductions: Presenters and ParticipantsValidation WorkshopValidation OverviewJohn M. Butler, PhDNational Institute of Standards and Technology stl.nist.gov/biotech/strbaseNIST and NIJ DisclaimerFunding: Interagency Agreement 2003-IJ-R-029 between theNational Institute of Justice and NIST Office of LawEnforcement StandardsPoints of view are those of the author and do not necessarily represent theofficial position or policies of the US Department of Justice or the NationalInstitute of Standards and Technology.Certain commercial equipment, instruments and materials are identified inorder to specify experimental procedures as completely as possible. In nocase does such identification imply a recommendation or endorsement bythe National Institute of Standards and Technology nor does it imply thatany of the materials, instruments or equipment identified are necessarily thebest available for the purpose.What is validation and why should it be done? Part of overall quality assurance program in a laboratory We want the correct answer when collecting data If we fail to get a result from a sample, we want to haveconfidence that the sample contains no DNA rather thanthere might have been something wrong with thedetection method Prepared by John M. ButlerDay #1 Validation Overview (John) Introduction to DAB Standards (Robyn & John) Developmental Validation (John)Day #2 Inconsistency in Validation between Labs (John) Internal Validation (Robyn) Method Modifications and Performance Checks (Robyn)Day #3 Practical Exercises (Robyn)Overview of This Section Why is validation important? How does validation help with quality assurance within alaboratory? What are the general goals of analytical validation? How is method validation performed in other fields suchas the pharmaceutical industry? Define accuracy, precision, sensitivity, stability,reproducibility, and robustness as applied to generalmeasurementsNRC II Recommendation 3.1 Laboratories should adhere tohigh quality standards (such asthose defined by TWGDAMand the DNA Advisory Board)and make every effort to beaccredited for DNA work (bysuch organizations as ASCLDLAB).1

Validation Workshop – Validation OverviewSome Desirable QC and QA GuidelinesNoted in NRC I pp. 104-105 Reagents and equipment are properlymaintained and monitored. Procedures used are generallyaccepted in the field and supported bypublished, reviewed data that weregathered and recorded in a scientificmanner. Appropriate controls are specified inprocedures and are used. New technical procedures arethoroughly tested to demonstrate theirefficacy and reliability for examiningevidence material before beingimplemented in casework.Elements for Guaranteeing Quality Resultsin Forensic DNA Testing Accepted Standards and Guidelines for OperationLaboratory AccreditationProficiency Testing of AnalystsStandard Operating ProceduresValidated MethodsCalibrated InstrumentationDocumented ResultsLaboratory AuditsTrustworthy IndividualsAug. 24, 2005 at NFSTCEnsuring Accurate Forensic DNA ResultsASCLD-LABAccreditationDABProficiencyTesting inesValidatedMethods(using standards and controls)Costs/Benefits of Quality AssuranceCosts Direct– Test materials– Standards– Quality assuranceequipment– Analysis of QA/QCsamples– Quality assurance official– Committee Work– Interlab Studies– Travel to meetingsBenefits More efficient outputs Fewer replicates for samereliability Fewer do-overs Greater confidence of:– Staff– Laboratory– CustomersTable 26.2 in J.K. Taylor (1987) Quality Assurance of Chemical Measurements. Lewis Publishers: Chelsea, MI.Organizations Involved in InternationalQuality Assurance Issues International Standards Organization (ISO)– http://www.iso.chISO 17025 AOAC International (Association of Official Analytical Chemists)– http://www.aoac.org Eurachem– http://www.eurachem.ul.pt VAM (Valid Analytical Measurement)– http://www.vam.org.uk CCQM (Comité Consultatif pour la Quantité de Matière; Consultative Committee for Amount ofSubstance – Metrology in Chemistry)– http://www.bipm.org/en/committees/cc/ccqm/ CITAC (Co-operation on International Traceability in Analytical Chemistry)– http://www.citac.ccPrepared by John M. ButlerOrganizations Involved in InternationalQuality Assurance Issues ASTM International (American Society for Testing and Materials)– http://www.astm.org CLSI (Clinical and Laboratory Standards Institute)– http://www.clsi.org ANSI (American National Standards Institute)– http://www.ansi.org ILAC (International Laboratory Accreditation Cooperation)– http://www.ilac.org FDA (U.S. Food and Drug Administration)– http://www.fda.gov2

Validation Workshop – Validation OverviewICH Validation DocumentsAug. 24, 2005 at NFSTCICH Method Validation tentd.asp?watersit JDRS-5LT6WZ ICH (International Conference on Harmonization of TechnicalRequirements for Registration of Pharmaceuticals for Human Use)– http://www.ich.org– Q2A: Text on Validation of Analytical Procedures (1994) http://www.fda.gov/cder/guidance/ichq2a.pdf– Q2B: Validation of Analytical Procedures : Methodology (1996) http://www.fda.gov/cder/guidance/1320fnl.pdf From Q2B:– “For the establishment of linearity, a minimum of five concentrations isrecommended”– “Repeatability should be assessed using (1) a minimum of 9 determinationscovering the specified range for the procedure (e.g., 3 concentrations/3replicates each); or (2) a minimum of 6 determinations at 100 percent of the testconcentration.”Method validation provides an assurance of reliability during normal use,and is sometime referred to as "the process of providing documentedevidence that the method does what it is intended to do."Why is Method Validation Necessary? It is an important element of quality control. Validation helps provide assurance that ameasurement will be reliable. In some fields, validation of methods is aregulatory requirement. The validation of methods is good science.Roper, P., et al. (2001) Applications of Reference Materials in Analytical Chemistry. Royal Society ofChemistry, Cambridge, UK, pp. 107-108.Validation DefinitionsISO 170255.4.5.1 Validation is the confirmation by examinationand the provision of objective evidence that theparticular requirements for a specific intended use arefulfilledDAB Quality Assurance Standards for ForensicDNA Testing Laboratories2 (ff) Validation is a process by which a procedure isevaluated to determine its efficacy and reliability forforensic casework analysis and includes:Definition of Validation Validation is confirmation by examination and provisionof objective evidence that the particular requirements fora specified intended use are fulfilled. Method validation is the process of establishing theperformance characteristics and limitations of a methodand the identification of the influences which maychange these characteristics and to what extent. It isalso the process of verifying that a method is fit forpurpose, i.e., for use for solving a particular analyticalproblem.EURACHEM Guide (1998) The Fitness for Purpose of Analytical Methods: A Laboratory Guide to MethodValidation and Related Topics; available at onsJ.M. Butler (2005) Forensic DNA Typing, 2nd Edition, p. 389, 391 Quality assurance (QA) – planned or systematic actionsnecessary to provide adequate confidence that a productor service will satisfy given requirements for quality Quality control (QC) – day-to-day operationaltechniques and activities used to fulfill requirements ofquality Validation – the process of demonstrating that alaboratory procedure is robust, reliable, and reproduciblein the hands of the personnel performing the test in thatlaboratoryTo demonstrate that a method is suitable for its intended purpose Prepared by John M. Butler3

Validation Workshop – Validation OverviewAug. 24, 2005 at NFSTCDefinitionsWhen is validation needed?J.M. Butler (2005) Forensic DNA Typing, 2nd Edition, p. 391 Robust method – successful results are obtained a highpercentage of the time and few, if any, samples need tobe repeated Reliable method – the obtained results are accurateand correctly reflect the sample being tested Reproducible method – the same or very similar resultsare obtained each time a sample is tested Before introduction of a new method into routine use Whenever the conditions change for which a method hasbeen validated, e.g., instrument with differentcharacteristics Whenever the method is changed, and the change isoutside the original scope of the methodL. Huber (2001) Validation of Analytical Methods: Review and Strategy. Supplied by www.labcompliance.comSome Purposes of Validation To accept an individual sample as a member of apopulation under study To admit samples to the measurement process To minimize later questions on sample authenticity To provide an opportunity for resampling when neededSample validation should be based on objective criteria toeliminate subjective decisions J.K. Taylor (1987) Quality Assurance of Chemical Measurements. Lewis Publishers: Chelsea, MI, p. 193Assumptions When Performing Validation The equipment on which the work is being done isbroadly suited to the application. It is clean, wellmaintained and within calibration. The staff carrying out the validation are competent in thetype of work involved. There are no unusual fluctuations in laboratoryconditions and there is no work being carried out in theimmediate vicinity that is likely to cause interferences. The samples being used in the validation study areknown to be sufficiently stable.Roper, P., et al. (2001) Applications of Reference Materials in Analytical Chemistry. Royal Society ofChemistry, Cambridge, UK, pp. 110-111.Steps Surrounding “Validation” in a Forensic LabThe VAM PrinciplesVAM Valid Analytical Measurement1.2.3.4.5.6.Analytical measurements should be made to satisfy an agreedrequirement.Analytical measurements should be made using methods andequipment that have been tested to ensure they are fit for theirpurpose.Staff making analytical measurements should be bothqualified and competent to undertake the task.There should be a regular and independent assessment of thetechnical performance of a laboratory.Analytical measurements made in one location should beconsistent with those made elsewhere.Organizations making analytical measurements should have welldefined quality control and quality assurance procedures.Effort to Bring a Procedure “On-Line”This is what takes the time Installation – purchase of equipment, ordering supplies, setting up in lab Learning – efforts made to understand technique and gain experiencetroubleshooting; can take place through direct experience in the lab or vicariouslythrough the literature or hearing talks at meetings Validation of Analytical Procedure – tests conducted in one’s lab to verifyrange of reliability and reproducibility for procedure SOP Development – creating interpretation guidelines based on lab experience QC of Materials – performance check of newly received reagents Training – passing information on to others in the lab Qualifying Test – demonstrating knowledge of procedure enabling start of casework Proficiency Testing – verifying that trained analysts are performing procedureproperly over timeRoper P et al. (2001) Applications of Reference Materials in Analytical Chemistry. Royal Society of Chemistry:Cambridge UK, p. 2Prepared by John M. Butler4