Verification Of Precision And Bias - EFLM

EurachemA FOCUS FORANALYTICAL CHEMISTRYIN EUROPEVerification of precision and biasElvar Theodorsson

Validation and verification ofmeasurement methods Procedures aiming at establishing realistic expectations with theanalyst and confidence with the end-user that the methods arefit for the intended purposes

Validation According to VIM 3, verification is“provision of objective evidence that agiven item fulfills specifiedrequirements” and Validation is “verification, where thespecified requirements are adequate forthe intended use”

Method validation Method validation is a specific kind of validation “the process ofdefining an analytical requirement, and confirming that themethod under consideration has performance capabilitiesconsistent with what the application requires” Method validation includes procedures that both 1) establish theperformance characteristics and limitations of a measurementmethod (e.g., trueness, precision, recovery, linearity,robustness) and 2) establish whether the performancecharacteristics of the measurement method being investigatedare fit for the intended purpose

Method verification Procedures to test to what extent the performance dataobtained by manufacturers during method validation can bereproduced in the environments of end-users Possible if the method (reagents, procedure and themeasurement instrument) is manufactured by a company orother reliable source which has performed proper methodvalidation and who is providing you with the detailed results, asingle laboratory method validation is not needed.

Method validation is performed to a varyingextent depending on its intended use Single laboratory method validation is appropriate where themethod is used for a specific purpose in a specific laboratory bypersonnel with the appropriate training. Full method validation includes, in addition to the proceduresemployed in single laboratory validation an interlaboratory study(collaborative study/ collaborative trial) with many measurementinstruments several operators etc. The performance characteristicsof the measurement method over extended periods of time are alsostudied in full method validation, including the effects of lot-to-lotvariation etc. Full diagnostic method validation is establishing the diagnosticproperties of the method e.g. in health and disease

Verification of measurement methods In Vitro Diagnostic (IVD) medical devices are in Europeregulated by a third EC Medical Device Directive, the IVDmedical device Directive 98/79/EC which has been mandatoryin Europa since December 2003 Local verification practices have commonly been establishedover time and are frequently influenced by accreditation andcertification authorities. Published practices for end-userverification officially endorsed both by the end-users and by thecompanies have appeared only recently7

Verification The EP15-A2 protocol from CLSI Uses control material with assigned concentration (e g fromexternal quality control) or certified reference materials Does not test for matrix effects which may occur in patientmaterials Practical and pragmatic method using patient samplesand common samples for internal quality control Bias is tested by comparison with a well-established methodsusing at least 20 patient samples Variation within- and between series is measured using thenormally used stable materials for internal quality control at leasttwice daily during two weeks8

99

Verification of measurement methods Clinical laboratories commonly measure in the order of 20 to 200natural patient samples having as wide concentration range aspossible, using both the method being replaced and the newmethod. At least two pooled patient samples may also be used, andthis may actually be an advantage when the medical decision pointis close to the detection limit of the method. Suitable stable materials for internal quality control are measured attwo levels in at least three replicates for at least five consecutivedays but preferably at least two weeks for estimating imprecision andfor establishing initial control limits for the internal quality controlprocedures. Linear regression, bias-plot and analysis of variancetechniques are used to determine bias, imprecision, matrix effects10

Measurement precision/imprecision Closeness between indications or measured quantity valuesobtained by replicate measurements on the same or similarobjects under the specified conditions of measurement The quantitative expression of precision is the standarddeviation (SD) or relative standard deviation (CV/CV %) The standard deviation of the estimate of the standarddeviation is inversely proportional to the square root of thenumber of replicates11

Harmonised terminology – VIM 3

Improved truenessxImproved precision

Repeatability imprecision When the same measurement procedure, same operators,same measuring system, same operating conditions and samelocation, and replicate measurements on the same or similarobjects over a short period of time A short period of time is usually less than a working day of 8hours Example of repeatability condition is when a stable controlmaterial or the same unknown sample is measured repeatedlyon the same day A prudent and cost effective number of replicate measurementsfor estimating repeatability imprecision are in the order of 1514

Intermediate measurement precision When a set of conditions that includes the same measurementprocedure, same location, and replicate measurements on thesame or similar objects over an extended period of time, butmay include other conditions involving changes Intermediate measurement imprecision includes variation due tonew calibrations, new reagent lots, new operators etc. The concept of between-days, between series, inter-seriesimprecision has earlier been used to describe this type ofimprecision15

Intermediate measurement precision Intermediate imprecision is usually measured using stablecontrol materials in two different concentrations which aremeasured routinely/daily over extended periods of time for atleast 1 year, but preferably during 2-3 years It is crucial that all sources of variation included in intermediateimprecision including e.g. lot-number changes are included insufficient/appropriate number of occurrences16

Intermediate and repeatability precision If the numbers of results obtained in each series/day are thesame, common two-way analysis of variance (ANOVA) can beused to calculate the total SD and its components of SD withinand between series. However, as is commonly the case inclinical laboratories, the number of replicate observations in theseries is unequal, more advanced ANOVA and variancecomponent analysis models catering for unequal number ofobservations each day/series should be used17

Fishbone/cause and effect/Ischikawadiagram18

Measurement bias Bias in the preparation of the calibrator, including erroneousvolume measurements or weighing of calibrators Using sample matrix for the calibrators which differs from thematrix in the samples Interferences/matrix effects in the samples, e.g. the colour ofbilirubin and haemoglobin in icteric and haemolytic samples inlaboratory medicine or the presence of high concentrations oflipids or proteins in the sample (hyperlipidaemia or myeloma).Manufacturers commonly use samples from healthy subjects fortheir validation studies, and the real influence of matrix effectson the methods may be fully evident only when the methods arefully introduced in diagnosing and monitoring seriously illpatients.19

Measurement bias The presence of molecules in the sample specifically interferingwith the reagents used in the measurement process, e.g.heterophilic antibodies (e.g. human antibodies against mouseIgG frequently used in immunoassays). Uncorrected loss of measurand at extraction Instability of the sample during transport or storage20

Determining measurement bias Purchasing certified reference materials from companies ororganizations of high metrological competence and comparing thestated concentration with the concentration your own methodsshows Comparing the concentrations your method measured in naturalsamples with the concentrations a reference method measured inthe same sample Participating in programs for external quality control. Most of theseprograms are based on consensus concentrations in modifiedcontrol samples, but some few are based on comparison toreference methods. The latter are frequently preferable.21

Determining measurement bias Measuring the recovery of the measurand in spiked naturalsamples Comparing the serial dilution of a natural sample or that of aspiked natural sample with the serial dilution of thecalibrator in the calibration curve22

Determining measurement bias Making studies of possible interferences/selectivity. This is evidently very different amongst different measurementmethods and fields of study. In laboratory medicine the studiesof interferences by bilirubin, haemoglobin, lipids, proteins anddrugs are amongst the most important. VIM 3 defines selectivityas “property of a measuring system, used with a specifiedmeasurement procedure, whereby it provides measuredquantity values for one or more measurands such that thevalues of each measurand are independent of othermeasurands or other quantities in the phenomenon, body, orsubstance being investigated”23

Common linear regressionOrthogonal linear regressionAdcock (1878) Deming (1943)Passing Babloc (1988)Bartlett (1949)

Method 1 - Method 2Bias plotMean difference plot (Tukey, 1977)Eksborg (1981)Bland-Altman (1983)(Method 1 Method 2)/2

Beware of the use of the correlationcoefficient Any correlation coefficient between two methods can be improved(made closer to 1) by increasing the range of concentrationsmeasured.

A conglomerate of laboratories In healthcare, the samples from a patient are over time likely tobe measured in different laboratories using different methods asthe patient visits primary care and different levels of hospitalcare.27

Laboratory iasXXMeasurement uncertaintyMeasured concentrationTrueconcentration

29

30

31

32

Full diagnostic method validationParticipantsWith diseaseWithout diseasePositive test True positivesFalse positivesTotal positive [PPV](type I error)Negative test False negatives True negatives Total negative [NPV](type II error)Total with diease Total withoutdiease[Sensitivity][Specificity]33

11.31135

What level on the measurement scaleis used when quantifying precision?A. Nominal0%B. Ordinal0%C. Interval0%D. Ratio0%

Validation should be used instead ofverification when.A. Applying the method on a new measurement system0%B. When a new technician is engaged0%C. When the method is modified0%D. When the control material shows out of control condition0%

Take a home message

Method validation Method validation is a specific kind of validation "the process of defining an analytical requirement, and confirming that the method under consideration has performance capabilities consistent with what the application requires" Method validation includes procedures that both 1) establish the