Section 6: Examples Of Evaluation Measures

Table 6-3. Data exchange between providers and laboratoriesMeasureData Source(s)Practical NotesConsiderations Waselectronicordering oflaboratorytests betweenoutpatientproviders andlaboratoriesachieved? Implementation Exchange requires anteaminterface between theambulatory EHR systemand the laboratorydata system. This isan exchange capabilityquestion as to whether thishas been demonstratedanywhere within theexchange.Is this a standardsbased bidirectionalinterface? Are providersusing dataexchangecapabilitywithlaboratories? Usage statisticsfrom system’saudit logs (e.g.,order logs,result viewlogs, systemlog-on tracking)Finding baselineprovider ratesmight be difficult.For example, whatis your sample ofphysicians whocould be using thesystem? You couldconsider gettingthis informationfrom local medicalsocieties or boardsof medicine.You could measure this inseveral ways. One wouldbe to divide the numberof providers using thesystem (numerator) by thenumber of total providers(denominator). A secondapproach might measurehow often individualproviders are accessingthe system, with accesshit rates as the numeratorand the number ofindividual providers asthe denominator. A thirdapproach might be to getan overall average rate bydividing the number ofaccess hits by the totalnumber of providers.Providers might bedefined as nurses and/or physicians. Trackingthis information over timeand presenting it visuallywould give stakeholdersan understanding ofadoption trends for yourproject. You could alsotrack the number of papertransactions still beingused (i.e., clinical staffputting laboratory resultsinto records).6-8ResourcesSee sTechnicalReport, p. 133,for a detaileddefinition andevaluationmethod for this35measure. Thisresource isfreely available.

Table 6-3. Data exchange between providers and laboratories (continued)MeasureData Source(s) Whatpercentageof laboratoryorders is sentelectronically? Was therea reductionin calls toprovidersto clarify anorder?Practical NotesConsiderationsResources Usage statistics The denominator is allfrom system’s orders (electronic andaudit logspaper). The numerator iselectronic orders only. Thiscan be done on both thelaboratory and providerside.This measurecan be costly if itrequires countingpaper orders.See sTechnicalReport, p. 136for a detaileddefinition andevaluationmethod for this35measure. Call logsThis measure requirestracking call volume beforeand after the intervention.Calls may not be fororder clarificationbut to reportother issues (e.g.,improper specimencollection,unavailability oftest, or new testversion).See sTechnicalReport, p. 64for a detaileddefinition andevaluationmethod for this35measure. Pre- and postimplementationreview offinancial logs,time andmotion studies,and workflowanalysis ina sample ofvarious settingsFirst, estimate what thesecosts are per order (laborcosts to prepare forms,costs to send forms)and then multiply by thenumber of orders sent out.Using time and motionstudies compare paperand electronic methods onhow much time individualsspend searching forresults, writing orders, andtranscribing; multiply timeby mean staff hourly wage.Make sure totrack orderselectronically.The cost ofan “electronictransfer” is notzero; it includes thecost of developingand maintainingthe infrastructureto send theinformationelectronically.See AHRQ’sTime andMotion StudiesDatabase fora detaileddefinition ofthis measureand additionalresources.This resourceis freely34available. How much ofa reduction? What was thereduction incosts to sendorders tolaboratory?6-9

Table 6-3. Data exchange between providers and laboratories (continued)MeasureData Source(s)Practical NotesConsiderations Impact onduplicatelaboratorytests Pre- and postimplementationreview ofclaims dataIf you are rolling out yourproject in stages, you coulduse those organizationsor providers who havenot gone live yet as yourcontrol group, therebyavoiding the need for aretrospective medicalrecord review. You mayalso be able to use billingdata to help focus thesearch for redundant tests.Need to define“duplicate” foreach type of test.For example,the definition ofduplicate woulddiffer by typeof blood test,and would differbased on whetherthe initial testwere normal vs.abnormal. Thismeasure might becostly if you have todo a medical recordreview. Waselectronicexchange oflaboratoryresultsbetweenoutpatientproviders andlaboratoriesachieved? Implementation This exchange requiresteaman interface between theambulatory EHR systemand the laboratory datasystem. The measureis whether exchangecapability is in place.Is this a standardsbased bidirectionalinterface? Impact onthe numberof calls to thelaboratory forresults Laboratory calllogsMeasurementsneed to be adjustedfor the volume oftests conductedby each of theparticipatinglaboratories.Also, changes inmarket share bylaboratories need tobe considered.A reduction in the numberof calls to the laboratoryfor results suggests thatproviders can find resultsin a timelier fashion.6-10Resources

Table 6-3. Data exchange between providers and laboratories (continued)MeasureData Source(s) Decreasein time toreport criticalresults by thelaboratoryPractical NotesConsiderationsResources Call logsThis is a great measure topre- and postconsider, given the Jointimplementation Commission’s interest inthis topic.If call loginformation isnot already beingcollected, it will behard to collect.See sTechnicalReport, p. 57,for a detaileddefinition andevaluationmethod for this35measure. Costs savedfor sendingand receivingresults Financial logsEstimate the costsassociated with receivinga single result (labor toopen mail, sort, distributeto clinicians, and post onpatient medical record) andmultiply by the number oflaboratory results received.If users are stillprinting outelectronic resultsto put in papermedical records,this cost must beconsidered as well. Impact on thesatisfactionof clinicians Surveys orfocus groupsexamining theperception ofusability, theease of learningto use thesystem, andefficiency asa result of thedata exchangeYou might considersampling both your usersas well as clinicians whocould be involved in theproject but who havechosen not to participate.Going to State- or regionwide provider databasesfrom local medicalsocieties or boards ofregistrations may be waysto determine your targetsurvey group. Considerquestions such as askingclinicians how often theywere able to find the resultthey were looking for in atimely manner. You couldcompare responses beforeand after implementation.It may be helpful toconduct satisfactionsurveys multiple timesat different stages of theproject to monitor trendsand potential unintendedconsequences (positiveand negative).6-11

Table 6-3. Data exchange between providers and laboratories (continued)MeasureData Source(s)Practical Notes Satisfactionof laboratorypersonnel Survey or focus Your survey could samplegroupsthe laboratory technicians,or the administrativepersonnel, including thosewho are responsible fortaking phone calls. Thesurvey would need to bedesigned to be distributedto all involved laboratories.It could be helpful toconduct the survey multipletimes at different stagesof the project to monitortrends and potentialunintended consequences(positive and negative). How muchdata wereable to beexchanged? Implementation Look at the number ofNote that justteamdiscrete elements that were because a messagewas sent properly, Data exchange exchanged.it does not meanlogsthat it was received Number ofand processedmessages sentproperly. Foror receivedexample, if anabnormal resultis placed in anexception queue, itmight stay in thatstate for monthsbefore the “correct”individual hasaccess to thoseresults.6-12ConsiderationsResourcesBe careful to surveyonly the personnelaffected by dataexchange, whichmay be invisible tosome staff. That is,they may not knowto whom the dataare being sent orwho is accessingit. For example, ifa laboratory resultis viewed by aprovider outsidethe laboratory’straditional servicebase, the laboratorytechnician maynot know that, andthus may not beaware of the dataexchange.Consider usingor amendingan existingsatisfactionsurvey. Reviewexisting surveysusing the HealthIT SurveyCompendiumon the AHRQHealth IT Web32site.

Table 6-4. Data exchange between providers and pharmaciesMeasure Is e-prescribingavailable in yourHIE region? What percentageof prescribersuse EHRtechnology toe-prescribe? What percentageof prescribersuse a standalonesystem fore-prescribing?Data Source(s)Practical NotesConsiderationsThis could beaccomplished throughan e-prescribing system(i.e., via RxHub orSureScripts) or throughan existing HIE system.This measures whetherthis type of exchangecapability is availablethrough the HIE system.Is this astandards-basedbidirectionalinterface?What doesthe pharmacycommunicateto the provider?Is thatcommunicationdone usingelectronicexchange ofinformation?6-13Resources

Table 6-4. Data exchange between providers and pharmacies (continued)MeasureData Source(s) Are providersusing dataexchangecapability withpharmacies? Usage statisticsfrom system’saudit logs How many newprescriptionsvs. renewalswere orderedelectronically? How areprovidersperformingon meaningfuluse measure 4,which (for stage1) requires 40percent of allprescriptionsto be sentelectronically?Practical NotesElectronic informationcollection is possiblein several ways. First,you could look at the Implementationnumber of electronicteamprescriptions received Regionalas the numerator andextensionthe total number ofcenters, whichprescriptions receivedtrack the number (both electronicof providers who and printed) as thehave reacheddenominator. A secondmilestone 3,approach would be to(attesting todivide the number ofmeaningful use) physicians submittingprescriptionselectronically(numerator) by thetotal number ofusers of the system(denominator). A thirdapproach would be todivide the number ofphysicians submittingprescriptionselectronically(numerator) by the totalnumber of physiciansin the service area(denominator).In addition to providerswho have reachedmilestone 3, providerswho have reachedmilestone 2 haveimplemented an EHRsystem and mayhave operationalizede-prescribing. Anyauthorized testing andcertification bodies(ATCB)-certifiedcomplete EHR systemmust be able toe-prescribe.Some States havefound that initial ordersfor prescriptionsare being orderedelectronically, whilerenewals are notordered electronically.6-14ConsiderationsResources

Table 6-4. Data exchange between providers and pharmacies (continued)MeasurePractical NotesConsiderations Usage statisticsfrom system’saudit logs fore-prescribingordersUse the number ofe-prescribing orderssent as the numeratorand the total numberof prescriptions filled(both electronicand printed) as thedenominator. The totalnumber of prescriptionsmay need to beestimated by surveyinga sample of providerpractices, or byreviewing e-prescribingsystem audit logs.Be sure thatthe messageswere correctlyreceived andprocessed on thereceiving end.Evaluators mayneed to contactthe pharmacyto verify thenumerator. Impact on callsto pharmacies Provider calllogs withprotected healthinformationremovedThe logs should alsocapture the nature ofthe call.This is primarydata collectionfrom theprovider office. Impact on callsto providersto clarify aprescription Pharmacycall logs withprotected healthinformationremovedMake sure thepharmacy call log hasthe requisite level ofdetail to capture thenature of the call.This is primarydata collection. Impact on costsdue to improvedformularycompliance oruse of genericdrugs IT teamIf the new system hasdecision support, thesystem may have thedata to show how oftena switch is made from anonformulary choice toa formulary alternative.Evaluating formularypatterns may be morefeasible if you focuson a single drug classor narrow down to asubset of patients.It could bedifficult tofind the preimplementationcompliance rate.The measuremay be costly ifmedical recordreviews arerequired. How much datawere able to beexchanged? What typeof data ory), and bywhom?Data Source(s) How manyelectronic drugorders weretransmitted asa percentageof total drugsordered? Medical recordreviews Health planutilization reviewdatabases6-15ResourcesSee CanadaHealth Infoway’sBenefitsEvaluationIndicatorsTechnical Report,p. 54, for adetailed definitionand evaluationmethod for this35measure.

Table 6-4. Data exchange between providers and pharmacies (continued)MeasureData Source(s)Practical NotesConsiderations Impact on costsby switching togenerics Health planutilization reviewdatabasesIf the new system hasdecision support, thesystem may have thedata to show how oftena switch is made froma brand name choiceto a generic alternative.Evaluating brand togeneric patterns maybe more feasible if youfocus on a single drugclass or narrow down toa subset of patients.Measuring costsimpact may becostly if medicalrecord reviewsare required,or if the EHRsystem cannotreport it.You will need to havelongitudinal data inorder to measure this.You could do activesurveillance and buildprompts into thesystem for cliniciansto report adverse drugevents under certaincircumstances (e.g.,when discontinuing adrug).This can be verydifficult to defineand measure.The teams mustcome togetherto decide whatconstitutesan adversedrug eventand how it willbe measured.Adverse drugevents arerelatively rareand it takesmany medicalrecord reviewsto be confidentabout theresults. IT team Medical recordreviews Impact onadverse drugevents Medical recordreviews6-16ResourcesSee CanadaHealth Infoway’sBenefitsEvaluationIndicatorsTechnical Report,p. 43 for adetailed definitionand evaluationmethod for this35measure.

Table 6-4. Data exchange between providers and pharmacies (continued)MeasureData Source(s)Practical NotesConsiderationsResources Cliniciansatisfaction SurveysYou might considersampling both yourusers as well asclinicians who could beinvolved in the projectbut who have chosennot to participate. Goingto State- or region-wideprovider databasesfrom local medicalsocieties or boards ofregistrations may beways to determine yourtarget survey group.It may be helpful toconduct the satisfactionsurvey multiple timesduring differentstages of project tomonitor trends andpotential unintendedconsequences (positiveand negative).Costs may beprohibitive forconducting asurvey.See CanadaHealth Infoway’sBenefitsEvaluationIndicatorsTechnical Report,p. 121 for adetailed definitionand evaluationmethod for35this measure.Consider usingor amendingan existingsatisfactionsurvey. Reviewexisting surveysusing the HealthIT SurveyCompendium onthe AHRQ Health32IT Web site. Pharmacistsatisfaction Surveys Focus groups Focus groupsYour surveycould sample thepharmacists, thetechnicians, orthe administrativepersonnel, includingthose who areresponsible for takingphone calls. Thesurvey would needto be designed tobe distributed to allinvolved pharmacies.It may be helpful toconduct the satisfactionsurvey multiple timesduring differentstages of the projectto monitor trends andpotential unintendedconsequences (positiveand negative).6-17Consider usingor amendingan existingsatisfactionsurvey. Reviewexisting surveysusing the HealthIT SurveyCompendium onthe AHRQ Health32IT Web site.

Table 6-4. Data exchange between providers and pharmacies (continued)MeasureData Source(s)Practical Notes Patientsatisfaction SurveysOne approach is to givepatients a survey alongwith the prescription. Focus groups6-18ConsiderationsResourcesConsider usingor amendingan existingsatisfactionsurvey. Reviewexisting surveysusing the HealthIT SurveyCompendium onthe AHRQ Health32IT Web site.

Table 6-5. Data exchange between providersMeasureData Source(s) What percentageof participatingpracticeswere able todemonstratemeaningfuluse measure14 (exchangeof key clinicalinformation)? Implementationteam What percentageof practices usedthe HIE systemto demonstrateexchange ofkey clinicalinformation?Practical NotesConsiderationsIn addition toIs this a standardsproviders who have based bidirectionalreached milestone exchange? Data exchange3, providers whologshave reached Regionalmilestone 2 haveextensionimplementedcenters, whichan EHR systemtrack the number and may haveof providers who operationalizedhave reachedelectronicmilestone 3,exchange(attesting tomeaningful use) What percentageused Direct(securemessagingprotocol)? Did providersuse other meansto achieveelectronicexchange ofinformation?6-19Resources

Table 6-5. Data exchange between providers (continued)MeasureData Source(s) Are providersusing HIEdata exchangecapability withother providers? Usage statisticsfrom system’saudit logs How much datawere able to beexchanged? Implementationteam How much of thetotal health datawas exchangedelectronicallyvs. using othermethods (e.g.,fax, mail, andcourier)? ImplementationteamPractical NotesIf the exchangeis sending onlyadministrative data,this clearinghouse Surveysfunction is not Implementationconsidered clinicalteamexchange. It isimportant to Number ofconsider how youprovidersdefine providersaccessing dataexchangingin or throughHIE, and average information withother providers.number ofrecords accessed Would you defineit as e-mailper month percommunication, orproviderdoes it need to besomething more,such as the abilityto send referralselectronically,or the ability toelectronicallysend a patient’smedical record fora referral?ConsiderationsHospital dischargesummaries aresometimes madeavailable throughthe exchange.Message countmight be usedIt will be difficultto determine theamount of databeing exchangedby nonelectronicmethods means. Logs6-20Resources

Table 6-5. Data exchange between providers (continued)MeasureData Source(s)Practical NotesConsiderationsResources Impact on costsof medicalrecord pulls LogsEstimate thelabor cost of amedical recordpull and multiplyby the numberof referrals in agiven time period.You could alsoreview a sampleof medical recordsto determine thepercentage ofconsultant notesthat are capturedelectronically for asample of patients.This assumes thatthe requisite datafor a referral orother request isbeing exchangedelectronically. Inmany cases, datasuch as notesare not availableelectronicallybecause th

(e.g., meeting minutes, Gantt charts, and organizational charts). The tables list measures in the following categories: z. Table 6-1. Measures of the infrastructure development effort; z. Table 6-2. Measures of process.