TIME MONEY - Strbase-b.nist.gov
The Copenhagen Forensic Genetic Summer SchoolJune 27-28, 2012The Copenhagen Forensic Genetic Summer SchoolAdvanced Topics in STR DNA AnalysisUniversity ofCopenhagenJune 27-28, 2012Overview of ValidationMike Coble and Becky HillNIST Applied Genetics GroupNational Institute of Standards and TechnologyGaithersburg, MarylandSome Articles Written on ValidationProfiles in DNA (Promega Corporation), vol. 9(2), pp. NA 902 s/forensic/volume8/PDFs submitted/02A CustomerCorner Val What is it.pdfStages of Technologyfor Forensic DNA Typing IdeaDemonstration of feasibilityResearch and developmentCommercializationValidation by forensic labsRoutine use by the /training/Copenhagen2012-STR-Workshop.htm1
The Copenhagen Forensic Genetic Summer SchoolJune 27-28, 2012Decision to Switch/Upgrade to New TechnologyHard tocalculateCOST toChangeImprovedCapabilitiesValidation time & effortImpact on legacy dataNew multiplex STR kitNew detection technologyNew DNA markersDecisions about ChangingTechnologies Cost to change Comfort and experience levels– court approved methods must be used in forensiclabs Capabilities Enhancements– Are they really needed?– Will legacy data be impacted?Where Is the Future Going for DNATechnology That Can Be Applied toForensic DNA Typing?Constant state of evolution (like computers) Higher levels of multiplexesMore rapid DNA separationsBetter data analysis softwareNew DNA MarkersValidating new technologies will always beimportant in progressive forensic DNA labs en2012-STR-Workshop.htm2
The Copenhagen Forensic Genetic Summer SchoolJune 27-28, 2012Importance of ValidationPurpose of Validation Many forensic laboratories, in an effort to becautious, are taking too long to perform theirvalidation studies and thereby delaying initiationof casework and contributing to backlogs in labsthat are already overburdened Technology will continue to advance and thusvalidation of new methodologies will always beimportant in forensic DNA laboratoriesThere will always be something to “validate” Validation Workshop (Aug 24-26, 2005 at idation/validationworkshop.htmCOURSE CONTENTSDay #1 Validation Overview (John) Introduction to DAB Standards(Robyn & John) Developmental Validation (John)Day #2 Inconsistency in Validationbetween Labs (John) Internal Validation (Robyn) Method Modifications andPerformance Checks (Robyn)Day #3 Practical Exercises openhagen2012-STR-Workshop.htm3
The Copenhagen Forensic Genetic Summer SchoolJune 27-28, 2012Questions to Keep in Mind Why is validation important? How does validation help with quality assurance within alaboratory? What are the general goals of analytical validation? How is method validation performed in other fields suchas the pharmaceutical industry? How do accuracy, precision, sensitivity, stability,reproducibility, and robustness impact measurements?What is Validation and Why Should It Be Done? Part of overall quality assurance program in a laboratory We want the correct answer when collecting data – We want analytical measurements made in one location tobe consistent with those made elsewhere (without thisguarantee there is no way that a national DNA database can besuccessful). If we fail to get a result from a sample, we want to haveconfidence that the sample contains no DNA rather thanthere might have been something wrong with thedetection method Want no false negatives Why is Method Validation Necessary? It is an important element of quality control. Validation helps provide assurance that ameasurement will be reliable. In some fields, validation of methods is aregulatory requirement. The validation of methods is good science.Roper, P., et al. (2001) Applications of Reference Materials in Analytical Chemistry. Royal Society ofChemistry, Cambridge, UK, pp. Copenhagen2012-STR-Workshop.htm4
The Copenhagen Forensic Genetic Summer SchoolJune 27-28, 2012Definition of Validation Validation is confirmation by examination and provisionof objective evidence that the particular requirements fora specified intended use are fulfilled. Method validation is the process of establishing theperformance characteristics and limitations of a methodand the identification of the influences which maychange these characteristics and to what extent. It isalso the process of verifying that a method is fit forpurpose, i.e., for use for solving a particular analyticalproblem.EURACHEM Guide (1998) The Fitness for Purpose of Analytical Methods: A Laboratory Guide to MethodValidation and Related Topics; available at http://www.eurachem.ul.pt/guides/valid.pdfMore Validation DefinitionsISO 170255.4.5.1 Validation is the confirmation by examinationand the provision of objective evidence that theparticular requirements for a specific intended use arefulfilledDAB Quality Assurance Standards for ForensicDNA Testing Laboratories2 (ff) Validation is a process by which a procedure isevaluated to determine its efficacy and reliability forforensic casework analysis and includes:To demonstrate that a method is suitable for its intended purpose DefinitionsJ.M. Butler (2005) Forensic DNA Typing, 2nd Edition, p. 389, 391 Quality assurance (QA) – planned or systematic actionsnecessary to provide adequate confidence that a productor service will satisfy given requirements for quality Quality control (QC) – day-to-day operationaltechniques and activities used to fulfill requirements ofquality Validation – the process of demonstrating that alaboratory procedure is robust, reliable, andreproducible in the hands of the personnel performingthe test in that g/Copenhagen2012-STR-Workshop.htm5
The Copenhagen Forensic Genetic Summer SchoolJune 27-28, 2012DefinitionsJ.M. Butler (2005) Forensic DNA Typing, 2nd Edition, p. 391 Robust method – successful results are obtained a highpercentage of the time and few, if any, samples need tobe repeated Reliable method – the obtained results are accurateand correctly reflect the sample being tested Reproducible method – the same or very similar resultsare obtained each time a sample is testedGeneral Levels of Validation Developmental Validation – commonlyperformed by commercial manufacturer of anovel method or technology (more extensivethan internal validation) Internal Validation – performed by individuallab when new method is introduced Performance Checks – can be performed withevery run (set of samples)The lifecycle of a method of ernalValidationFeinberg et al. (2004) Anal. Bioanal. Chem. 380: openhagen2012-STR-Workshop.htm6
The Copenhagen Forensic Genetic Summer SchoolJune 27-28, 2012Validation Section of the DNA Advisory Board Standardsissued October 1, 1998 and April 1999; published in Forensic Sci. Comm. July 2000STANDARD 8.1 The laboratory shall use validated methods and procedures for forensic casework analyses (DNA analyses).8.1.1 Developmental validation that is conducted shall be appropriately documented.8.1.2 Novel forensic DNA methodologies shall undergo developmental validation to ensure the accuracy, precision andreproducibility of the procedure. The developmental validation shall include the following:8.1.2.1 Documentation exists and is available which defines and characterizes the locus.8.1.2.2 Species specificity, sensitivity, stability and mixture studies are conducted.8.1.2.3 Population distribution data are documented and available.8.1.2.3.1 The population distribution data would include the allele and genotype distributions for the locus or lociobtained from relevant populations. Where appropriate, databases should be tested for independenceexpectations.8.1.3 Internal validation shall be performed and documented by the laboratory.8.1.3.1 The procedure shall be tested using known and non-probative evidence samples (known samples only). Thelaboratory shall monitor and document the reproducibility and precision of the procedure using human DNA control(s).8.1.3.2 The laboratory shall establish and document match criteria based on empirical data.8.1.3.3 Before the introduction of a procedure into forensic casework (database sample analysis), the analyst orexamination team shall successfully complete a qualifying test.8.1.3.4 Material modifications made to analytical procedures shall be documented and subject to validation testing.8.1.4 Where methods are not specified, the laboratory shall, wherever possible, select methods that have been published byreputable technical organizations or in relevant scientific texts or journals, or have been appropriately evaluated for a specific orunique application.FORENSIC SCIENCE COMMUNICATIONSJULY 2000 VOLUME 2 NUMBER 3Revised SWGDAM Validation Guidelines(July 04/standards/2004 03 standards02.htmThe document provides validation guidelines and definitions approved by SWGDAM July 10, 2003.ENFSI Validation Guidelines (November 2010)http://www.enfsi.eu/get doc.php?uid hagen2012-STR-Workshop.htm7
The Copenhagen Forensic Genetic Summer SchoolJune 27-28, 2012Ensuring Accurate Forensic DNA ResultsASCLD-LABAccreditationDABProficiencyTesting inesValidatedMethods(using standards and controls)Checks and Controls on DNA ResultsCommunityFBI DNA Advisory Board’s Quality AssuranceStandards (also interlaboratory studies)LaboratoryASCLD/LAB Accreditation and AuditsAnalystISO17025Proficiency Tests & Continuing EducationMethod/Instrument Validation of Performance(along with traceable standard sample)ProtocolData SetsStandard Operating Procedure is followedAllelic ladders, positive and negative amplificationcontrols, and reagent blanks are usedIndividual SampleInternal size standard present in every sampleInterpretation ofResultSecond review by qualified analyst/supervisorCourt Presentation Defense attorneys and experts with power ofof Evidencediscovery requestsMy perspective en2012-STR-Workshop.htm8
The Copenhagen Forensic Genetic Summer SchoolJune 27-28, 2012Validation PhilosophyWhen is Validation Needed? Before introduction of a new method into routine use Whenever the conditions change for which a method hasbeen validated, e.g., instrument with differentcharacteristics Whenever the method is changed, and the change isoutside the original scope of the methodL. Huber (2001) Validation of Analytical Methods: Review and Strategy. Supplied by www.labcompliance.comSome Purposes of Validation To accept an individual sample as a member of apopulation under study To admit samples to the measurement process To minimize later questions on sample authenticity To provide an opportunity for resampling when neededSample validation should be based on objective criteria toeliminate subjective decisions J.K. Taylor (1987) Quality Assurance of Chemical Measurements. Lewis Publishers: Chelsea, MI, p. hagen2012-STR-Workshop.htm9
The Copenhagen Forensic Genetic Summer SchoolJune 27-28, 2012The VAM PrinciplesVAM Valid Analytical Measurement1.Analytical measurements should be made to satisfy an agreedrequirement.Analytical measurements should be made using methods andequipment that have been tested to ensure they are fit for theirpurpose.Staff making analytical measurements should be bothqualified and competent to undertake the task.There should be a regular and independent assessment of thetechnical performance of a laboratory.Analytical measurements made in one location should beconsistent with those made elsewhere.Organizations making analytical measurements should have welldefined quality control and quality assurance procedures.2.3.4.5.6.Roper P et al. (2001) Applications of Reference Materials in Analytical Chemistry. Royal Society of Chemistry:Cambridge UK, p. 2The Community Benefits from Training To better understand what validation entails and how itshould be performed (why a particular data set issufficient) Many labs already treat DNA as a “black box” andtherefore simply want a “recipe” to follow People are currently driven by fear of auditors and courtsrather than scientific reasoning Many different opinions exist and complete consensus isprobably impossibleHow do you validate a method? Decide on analytical requirements– Sensitivity, resolution, precision, etc. Plan a suite of experimentsCarry out experimentsUse data to assess fitness for purposeProduce a statement of validation– Scope of the methodRoper, P., et al. (2001) Applications of Reference Materials in Analytical Chemistry. Royal Society ofChemistry, Cambridge, UK, pp. Copenhagen2012-STR-Workshop.htm10
The Copenhagen Forensic Genetic Summer SchoolJune 27-28, 2012Assumptions When Performing Validation The equipment on which the work is being done isbroadly suited to the application. It is clean, wellmaintained and within calibration. The staff carrying out the validation are competent in thetype of work involved. There are no unusual fluctuations in laboratoryconditions and there is no work being carried out in theimmediate vicinity that is likely to cause interferences. The samples being used in the validation study areknown to be sufficiently stable.Roper, P., et al. (2001) Applications of Reference Materials in Analytical Chemistry. Royal Society ofChemistry, Cambridge, UK, pp. 110-111.Tools of Method Validation Standard samples– positive controls– NIST SRMs BlanksReference materials prepared in-house and spikesExisting samplesStatisticsCommon senseRoper, P., et al. (2001) Applications of Reference Materials in Analytical Chemistry. Royal Society ofChemistry, Cambridge, UK, p. 110.Urban Legends of Validation Butler, J.M. (2006) Profiles in DNA vol. 9(2), pp. 3-6#1: HUNDREDS OR THOUSANDS OF SAMPLES ARE REQUIRED TO FULLYVALIDATE AN INSTRUMENT OR METHOD#2: VALIDATION IS UNIFORMLY PERFORMED THROUGHOUT THECOMMUNITY#3: EACH COMPONENT OF A DNA TEST OR PROCESS MUST BE VALIDATEDSEPARATELY#4: VALIDATION SHOULD SEEK TO UNDERSTAND EVERYTHING THATCOULD POTENTIALLY GO WRONG WITH AN INSTRUMENT ORTECHNIQUE#5: LEARNING THE TECHNIQUE AND TRAINING OTHER ANALYSTS AREPART OF VALIDATION#6: VALIDATION IS BORING AND SHOULD BE PERFORMED BY SUMMERINTERNS SINCE IT IS BENEATH THE DIGNITY OF A QUALIFIED ANALYST#7: DOCUMENTING VALIDATION IS DIFFICULT AND SHOULD BE EXTENSIVE#8: ONCE A VALIDATION STUDY IS COMPLETED YOU NEVER HAVE TOREVISIT agen2012-STR-Workshop.htm11
The Copenhagen Forensic Genetic Summer SchoolJune 27-28, 2012Validation PhilosophyAsk first: Does the new method improve your capability? Concordance – are the same typing results obtained withthe new technique as with an older one? Constant Monitoring – check multiple allelic ladders in abatch against one another to confirm precision andconsistent lab temperature Common Sense – are replicate tests repeatable?Common Perceptions of ValidationLots ofexperimentsare requiredEffortThe goal is not toexperience everypossible scenarioduring validation “You cannot mimiccasework because everycase is different.”Many labs are examining far too many samplesin validation and thus delaying application ofcasework and contributing to backlogs Significant time is required to perform studiesTimeNumber of Samples NeededRelationship between a sample and a population of dataHow do you relatethese two values?Data collected inyour lab as partof validationstudies“Sample” ofTypical DataAll potential data thatwill be collected inthe future in your labStudent’s t-Testassociates asample to apopulation“Population” of AllData Copenhagen2012-STR-Workshop.htm12
The Copenhagen Forensic Genetic Summer SchoolJune 27-28, 2012Student's t-Tests"Student" (real name: W. S. Gossett [1876-1937]) developedstatistical methods to solve problems stemming from hisemployment in a brewery.Student's t-test deals with the problems associated withinference based on "small" samples: the calculated mean(Xavg) and standard deviation ( ) may by chance deviatefrom the "real" mean and standard deviation (i.e., whatyou'd measure if you had many more data items: a"large" .htmlStudent’s t-Test CurveImpact of Number of Experiments on Capturing Variability in a Population of DataInterval for 95% 31102.26502.011001.985001.96100001.961.96 for aninfinite numberof samplestested20012345678910# Experiments ConductedThe Number “5” in Forensic ValidationNDIS Appendix BExpert SystemValidationRequirements At least 5challenge eventsmust be observedfor each issue(e.g., pullup,shoulders, spikes,tri-allelic patterns,mixtures,contamination,variant Copenhagen2012-STR-Workshop.htm13
The Copenhagen Forensic Genetic Summer SchoolAllele frequencies denoted withan asterisk (*) are below the5/2N minimum allele thresholdrecommended by the NationalResearch Council report (NRCII)The Evaluation of Forensic DNAEvidence published in 1996.Allele Frequency TablesButler et al. (2003)JFS 48(4):908-911Einum et al. (2004)JFS 49(6): 1381-1385CaucasianCaucasianN 302N 7,636110.0017*0.0009120.0017*0.0007D3S1358June 27-28, mum AlleleFrequency 5/2NWant to sample at least5 chromosomes toprovide a somewhatreliable estimate of anallele’s frequency in apopulationValidation in Other Fields(Besides Forensic DNA Testing)Pharmaceutical Industry and FDA FollowsICH Validation Documents ICH (International Conference on Harmonization of TechnicalRequirements for Registration of Pharmaceuticals for Human Use)– http://www.ich.org– Q2A: Text on Validation of Analytical Procedures (1994) http://www.fda.gov/cder/guidance/ichq2a.pdf– Q2B: Validation of Analytical Procedures : Methodology (1996) http://www.fda.gov/cder/guidance/1320fnl.pdf From Q2B:– “For the establishment of linearity, a minimum of five concentrations isrecommended”– “Repeatability should be assessed using (1) a minimum of 9 determinationscovering the specified range for the procedure (e.g., 3 concentrations/3replicates each); or (2) a minimum of 6 determinations at 100 percent of the ase/training/Copenhagen2012-STR-Workshop.htm14
The Copenhagen Forensic Genetic Summer SchoolJune 27-28, 2012ICH Method Validation tentd.asp?watersit JDRS-5LT6WZMethod validation provides an assurance of reliability during normal use,and is sometime referred to as "the process of providing documentedevidence that the method does what it is intended to do."Useful Resources on Validation Taylor JK. (1981) Quality assurance of chemical measurements.Analytical Chemistry 53(14): 1588A-1596A. Taylor JK. (1983) Validation of analytical methods. AnalyticalChemistry 55(6): 600A-608A. Green JM. (1996) A practical guide to analytical method validation.Analytical Chemistry 68: 305A-309A. EURACHEM Guide (1998) The Fitness for Purpose of Analytical Methods:A Laboratory Guide to Method Validation and Related Topics; available athttp://www.eurachem.ul.pt/guides/valid.pdfSee also STRBase Validation alidation.ht
COURSE CONTENTS Day #1 Validation Overview (John) Introduction to DAB Standards (Robyn & John) Developmental Validation (John) Day #2 Inconsistency in Validation between Labs (John) Internal Validation (Robyn) Method Modifications and Performance Ch
2.1 NIST SP 800-18 4 2.2 NIST SP 800-30 4 2.3 NIST SP 800-34 4 2.4 NIST SP 800-37 4 2.5 NIST SP 800-39 5 2.6 NIST SP 800-53 5 2.7 NIST SP 800-53A 5 2.8 NIST SP 800-55 5 2.9 NIST SP 800-60 5 2.10 NIST SP 800-61 6 2.11 NIST SP 800-70 6 2.12 NIST SP 800-137 6 3 CERT-RMM Crosswalk of NIST 800-Series Special Publications 7
NIST SP 800-30 – Risk Assessment NIST SP 800-37 – Risk Management Framework NIST SP 800-39 – Risk Management NIST SP 800-53 – Recommended Security Controls NIST SP 800-53A – Security Control Assessment NIST SP 800-59 – National Security Systems NIST SP 800-60 – Security Category Mapping NIST
NIST Risk Management Framework 1. Categorize information system (NIST SP 800-60) 2. Select security controls (NIST SP 800-53) 3. Implement security controls (NIST SP 800-160) 4. Assess security controls (NIST SP 800-53A) 5. Authorize information system (NIST SP 800-37) 6. Monitor security controls (NIST SP 800-137) Source: NIST CSRC, http .
Source: 9th Annual API Cybersecurity Conference & Expo November 11-12, 2014 - Houston, TX. 11 Industry Standards and Committee Initiatives WIB M2784-X-10 API 1164 ISA 99/IEC 62443 NIST SP 800-82 NIST SP 800-12 NIST SP 800-53 NIST SP 800-53A NIST SP 800-39 NIST SP 800-37 NIST SP 800-30 NIST SP 800-34 ISO 27001,2 ISO 27005 ISO 31000
Mar 01, 2018 · ISO 27799-2008 7.11 ISO/IEC 27002:2005 14.1.2 ISO/IEC 27002:2013 17.1.1 MARS-E v2 PM-8 NIST Cybersecurity Framework ID.BE-2 NIST Cybersecurity Framework ID.BE-4 NIST Cybersecurity Framework ID.RA-3 NIST Cybersecurity Framework ID.RA-4 NIST Cybersecurity Framework ID.RA-5 NIST Cybersecurity Framework ID.RM-3 NIST SP 800-53
Apr 08, 2020 · Email sec-cert@nist.gov Background: NIST Special Publication (SP) 800-53 Feb 2005 NIST SP 800-53, Recommended Security Controls for Federal Information Systems, originally published Nov 2001 NIST SP 800-26, Security Self-Assessment Guide for IT Systems, published Dec 2006 NIST SP 800-53, Rev. 1 published July 2008 NIST SP 800-53A, Guide for
https://nist.gov/rmf NIST RMF Quick Start Guide CATEGORIZE STEP nist.gov/rmf Frequently Asked Questions (FAQs)RISK MANAGEMENT FRAMEWORK RMF NIST NIST Risk Management Framework (RMF) Categorize Step . ecurity categorization standards for information and systems provide a common framework and understanding for expressing security
National Institute of Standards and Technology (NIST) was created in 1901 as the National Bureau of Standards (NBS). The name was changed to NIST in 1988. s iTS NI part of the U.S. Department of Commerce with a mission to develop and promote measurement, standards, and
