Usability Veracity Attestation (170.315.g.3) - Drummond Group, LLC
Usability Veracity Attestation (170.315.g.3)Debbie McKay, RN, BHS, JDSenior Solutions Manager – RegulatorySunrise Business UnitAllscripts Healthcare Solutions222 Merchandise Mart Plaza, Suite 2024Chicago, IL 606541.800.334.8534For public release:Allscripts attests that the usability standard/process and usability report submitted for thecertification of Sunrise Clinical Manager (Allscripts 'Sunrise Acute Care', and Allscripts 'SunriseAmbulatory Care' EHRs) is accurate and complete per the requirements of the ONC criterion170.315(g)(3).I hereby attest that all above statements are true, as an authorized signing authority onbehalf of my organization.Debbie McKay, RN, BHS, JDSr Solutions Manager-Regulatory11/17/2016
User-Centered Design Process Template (170.315.g.3)Debbie McKay, RN, BHS, JDSenior Solutions Manager – RegulatorySunrise Business UnitAllscripts Healthcare Solutions222 Merchandise Mart Plaza, Suite 2024Chicago, IL 606541.800.334.8534For public release:Allscripts used the following usability design in developing and designing their health ITmodule, Sunrise Clinical Manager (Allscripts 'Sunrise Acute Care', and Allscripts 'SunriseAmbulatory Care' EHRs): NISTIR 7741.National Institute of Standards and Technology. (2010). NIST Guide to the Processes Approachfor Improving the Usability of Electronic Health Records (NISTIR 7741). Gaithersburg, pub id 907313.According to the National Institute of Standards and Technology (NIST) Guide to the ProcessesApproach for Improving the Usability of Electronic Health Records, EHRs should support aprocess that provides a high level of usability for all users. The goals are for users to interactwith the system safely, effectively, efficiently, and with an acceptable level of satisfaction. Tothis end, design to optimize for safety, effectiveness, efficiency and user satisfaction was utilizedthroughout the design and development cycle.I hereby attest that all above statements are true, as an authorized signing authority on behalf ofmy organization.Debbie McKay, RN, BHS, JDSr Solutions Manager-Regulatory11/17/2016
Allscripts Sunrise Acute Care 16.3Customized Common IndustryFormat Template for ElectronicHealth Record Usability TestingA User-Centred Design (UCD) ActivityCopyright 2016 Allscripts Healthcare Solutions, Inc.www.allscripts.com
Copyright NoticeCopyright 2016 Allscripts Healthcare Solutions, Inc.Confidentiality and Proprietary RightsThis document is the confidential property of Allscripts Healthcare Solutions, Inc. Itis furnished under an agreement with Allscripts Healthcare Solutions, Inc. and mayonly be used in accordance with the terms of that agreement. The use of thisdocument is restricted to customers of Allscripts Healthcare Solutions, Inc. and theiremployees. The user of this document agrees to protect the Allscripts HealthcareSolutions, Inc. proprietary rights as expressed herein. The user further agrees not topermit access to this document by any person for any purpose other than as an aidin the use of the associated system. In no case will this document be examined forthe purpose of copying any portion of the system described herein or to designanother system to accomplish similar results. This document or portions of it maynot be copied without written permission from Allscripts Healthcare Solutions, Inc.The information in this document is subject to change without notice.Solution Design Document22-Dec-16Page 2 of 208
ContentsExecutive summary . 51.1Data Summary . 61.2Major Findings and Areas for Improvement . 162 Introduction . 253 Method. 263.1Participants . 263.2Study Design . 333.3Tasks . 343.4Procedures. 353.5Test Location . 363.6Test Environment . 363.7Test Forms and Tools . 373.8Participant Instructions . 373.8.1Participant Instructions . 373.9Usability Metrics . 403.10Data Scoring . 404 Results: §170.315(a)(1) Computerized Provider Order Entry (CPOE) – Medications 444.1Task Mapping . 444.2Task Participant and Instructions . 454.3Data Analysis and Reporting. 454.4Discussion of the Findings . 475 Results: §170.315(a)(2) Computerized Provider Order Entry (CPOE) – Laboratory . 505.1Task Mapping . 505.2Task Participant and Instructions . 515.3Data Analysis and Reporting. 515.4Discussion of the Findings . 536 Results: §170.315(a)(3) Computerized Provider Order Entry (CPOE) – DiagnosticImaging . 556.1Task Mapping . 556.2Task Participant and Instructions . 566.3Data Analysis and Reporting. 566.4Discussion of the Findings . 587 Results: §170.315(a)(4) Drug-Drug, Drug-Allergy Interaction Checks - Clinical . 607.1Task Mapping . 607.2Task Participant and Instructions . 617.3Data Analysis and Reporting - Clinical . 627.4Discussion of the Findings . 638 Results: §170.315(a)(4) Drug-Drug, Drug-Allergy Interaction Checks - Configuration668.1Task Mapping . 668.2Task Participant and Instructions . 678.3Data Analysis and Reporting - Configuration. 678.4Discussion of the Findings . 681Solution Design Document22-Dec-16Page 3 of 208
91011121314151617Results: 170.315(a)(5) Demographics - Clinical . 719.1Task Mapping . 719.2Task Participant and Instructions . 729.3Data Analysis and Reporting - Clinical . 729.4Discussion of the Findings . 73Results: 170.315(a)(5) Demographics - Registration . 7610.1Task Mapping . 7610.2Task Participant and Instructions . 7710.3Data Analysis and Reporting . 7810.4Discussion of the Findings . 80Results: 170.315(a)(6) Problem List . 8411.1Task Mapping . 8411.2Task Participant and Instructions . 8511.3Data Analysis and Reporting . 8511.4Discussion of the Findings . 86Results: 170.315(a)(7) Medication List . 8912.1Task Mapping . 8912.2Task Participant and Instructions . 8912.3Data Analysis and Reporting . 9012.4Discussion of the Findings . 91Results: 170.315(a)(8) Medication Allergy List . 9313.1Task Mapping . 9313.2Task Participant and Instructions . 9313.3Data Analysis and Reporting . 9313.4Discussion of the Findings . 94Results: §170.315(a)(9) Clinical Decision Support - Clinical. 9814.1Task Mapping . 9814.2Task Participant and Instructions . 9914.3Data Analysis and Reporting . 10014.4Discussion of the Findings . 104Results: §170.315(a)(9) Clinical Decision Support - Configuration . 10915.1Task Mapping . 10915.2Task Participant and Instructions . 11015.3Data Analysis and Reporting . 11115.4Discussion of the Findings . 112Results: 170.315(a)(14) Implantable Device List . 11516.1Task Mapping . 11516.2Task Participant and Instructions . 11616.3Data Analysis and Reporting - Clinical . 11716.4Discussion of the Findings . 119Results: §170.315(b)(2) Clinical Information Reconciliation and Incorporation (CIRI)12317.1Task Mapping . 12317.2Task Participant and Instructions . 12417.3Data Analysis and Reporting - Clinical . 125Solution Design Document22-Dec-16Page 4 of 208
1819202117.4Discussion of the Findings . 128Results: 170.315(b)(3) Electronic Prescribing . 13118.1Task Mapping . 13118.2Task Participant and Instructions . 13218.3Data Analysis and Reporting . 13318.4Discussion of the Findings . 136Results: Critical Use Risks for Patient Safety and NISTIR 7804-1. 14319.1Task Mapping . 14319.2Task Participant and Instructions . 14319.3Data Analysis and Reporting . 14419.4Discussion of the Findings . 144System Satisfaction . 14820.1About System Usability Scale (SUS) Scores . 14820.2Clinical System Satisfaction Results . 14820.3CIRI and Electronic Prescribing System Satisfaction Results . 14820.4Configuration System Satisfaction Results . 14820.5Demographic System Satisfaction Results . 148Appendices . 14921.1Appendix 1: Sample Recruiting Screener . 15021.2Appendix 2: Participant Demographics Update for Sunrise . 15121.3Appendix 3: NDA and Informed Consent Form . 15321.4Appendix 4: Example Moderator’s Guide and Task Sheet . 15421.4.1Moderator’s Guide Common Instruction for all Sessions . 15421.4.2Moderator Guide Specific to Clinical Wave 1 . 15721.4.3Moderator’s Guide Specific to Configuration Test. 16321.4.4Moderator’s Guide Specific to Registration Test . 16721.4.5Moderator’s Guide Specific to Clinical (Wave 2) . 16921.4.6Task Data Sheets for Clinical Test Wave 1 . 17621.4.7Task Data Sheets for Configuration . 18221.4.8Task Data Sheets for Registration . 18421.4.9Task Data Sheets for Clinical Wave 2 . 18521.5Appendix 5: System Usability Scale Questionnaire . 19721.6Appendix 6: Incentive Receipt and Acknowledgment Form . 198Solution Design Document22-Dec-16Page 5 of 208
EHR Usability Test Report of Sunrise Acute Care 16.3Report based on ISO/IEC 25062:2006 Common Industry Format for Usability TestReportsSunrise Acute Care 16.3Date of Usability Test:Date of Report:Report Prepared By:Solution Design DocumentJuly 14, 2016 – August 4, 2016 (Clinical, Configuration,and Registration Tests)August 10 - August 30, 2016 (Clinical Test)October 4, 2016User-View, Inc.David Clarke, PhD and Janey Barnes, PhDHuman Factors Specialist919.697.5329jbarnes@user-view.com1109 Holland Ridge DriveRaleigh, NC 2760322-Dec-16Page 4 of 208
1 Executive summaryA usability test of Allscripts Sunrise Acute Care 16.3 was conducted in two waves, inperson from July 14 through August 4, 2016, and remotely from August 10 throughAugust 30, by User-View, Inc. The purpose of this test was to test and validate theusability of the current user interface, and provide evidence of usability in the EHRunder Test (EHRUT). During the usability test, 96 healthcare providers and 17configuration specialists and 17 registration specialists matching the targetdemographic criteria served as participants and used the EHRUT in simulated, butrepresentative tasks.This study collected performance data on 12 Scenarios typically conducted on anEHR:1. Document problems, order medications, review and act on CDS, accesslinked referential decision support, and review and act upon drug-allergyinteraction alert.2. Add new patient problem, order specific labs and radiology based on theinformation provided, change a lab order, and radiology order, add newmedication, review and act on CDS, review and act on drug-druginteraction alert, access source attributes for alerts.3. Reconcile specific clinical information (medications, problems, allergies)based on the information provided.4. Work with Electronic Prescribe: order, cancel, approve pharmacysubstitution recommendation, change to generic, review medicationhistory, and review fill status.5. Choose a patient with a similar name as another patient.6. Document date and cause of death. Change date and cause of death.7. Add, edit and remove implantable device data. Access and obtain detailedinformation about devices.8. Add new and review allergies and make updates based on the informationprovided.9. Add new and review medications and make updates based on theinformation provided.10. Add and edit demographic data (registration).11. Configure Drug-Drug / Drug-Allergy alerts.12. Configure CDS recommendations.During the 60-minute clinical (wave 1) in-person one-on-one usability test, the 30minute clinical (wave 2) one-on-one remote usability test, the 30-minute registrationone-on-one remote usability test, and the 30-minute configuration one-on-oneremote usability tests, each participant was greeted by the administrator and askedto review and sign an informed consent/release form (included in Appendix 3: NDAand Informed Consent Form); they were instructed that they could withdraw at anySolution Design Document22-Dec-16Page 5 of 208
time. Participants had prior experience with the EHR. The administrator introducedthe test, and instructed participants to complete a series of tasks (given one at atime) using the EHRUT. During the testing, the administrator timed each task and,along with the data logger, recorded user performance data on paper andelectronically. The administrator did not give the participant assistance in how tocomplete the task.Participant screens and audio were recorded for subsequent analysis.The following types of data were collected for each participant: Number of tasks successfully completed within the allotted time withoutassistanceTime to complete the tasksNumber and types of errorsPath deviationsParticipant’s verbalizationsParticipant’s satisfaction ratings of the systemAll participant data was de-identified – no correspondence could be made from theidentity of the participant to the data collected. Following the conclusion of thetesting, participants were asked to complete a post-test questionnaire and werecompensated for their time based on agreements made between Allscripts and theparticipating institution. Compensation was based on 100 per participant.1.1Data SummaryVarious recommended metrics, in accordance with the examples set forth in the NISTGuide to the Processes Approach for Improving the Usability of Electronic HealthRecords, were used to evaluate the usability of the EHRUT. Following is a summary ofthe performance and rating data collected on the EHRUT.Solution Design Document22-Dec-16Page 6 of 208
Table 1. Summary of the performance and rating (Observed/ Optimal)33100%NTask TimeMean(SD)SecDeviations(Observed/ Optimal)ErrorsScenarioRatings5 Easy(%)Mean(SD)TaskCPOE MedicationsScenario 2a.1: Accessorders to place amedication orderScenario 2a.2: OrderPromethazineInjectable, 25 mg, IM,OnceScenario 2a.3:Enter/Ensure reason fororder: VomitingCPOE MedicationsScenario 2b.1: Accessmedication orders tochange an orderScenario 2b.2: ChangePromethazineInjectable, 12.5 mg,IM, OnceCPOE LaboratoryScenario 2a.4: Accessorders to place alaboratory orderScenario 2a.5: OrdersLaboratory Test: StrepScreenScenario 2a.6:Enter/Ensure reason fororder: sore throatCPOE Laboratory10/1033100%3382%32100%Scenario 2a.9:Enter/Ensure reason fororder: CongestionSolution Design 0.8)18%**5/225/13Scenario 2b.3: Accessorders to change a32Laboratory Test orderScenario 2b.4: Change32Laboratory Test: StrepScreen to STATCPOE Diagnostic ImagingScenario 2a.7: Accessorders to place adiagnostic imagingorderScenario 2a.8: OrdersXR Chest 2 -Dec-16Page 7 of 208
erved/ Optimal)Mean(SD)Sec14/11201*(86)Task TimeDeviations(Observed/ Optimal)ErrorsScenarioRatings5 Easy(%)Mean(SD)TaskScenario 2b CPOE Diagnostic ImagingScenario 2b.5: Accessorder to change adiagnostic imagingorderScenario 2b.6: ChangeXR Chest 2 Views toSTATDrug-Drug Drug AllergyScenario 1 Subtask B1:Review and act uponDrug-Allergy checkdisplayed for Codeineand MorphineInjectableScenario 1 Subtask C1:Review and act uponDrug-Drug checkdisplayed for Aspirinand WarfarinTask C2a: RemoveDrug-Drug interactionseverity level securityrightTask C3: InitialExposure: Adjust drugdrug interactionseverity level to a MinorInteractionTask C4: Adjust drugdrug interactionseverity level to aminor levelDemographicsScenario 3a.1: CreateDocument (Access)Scenario 3a.2: Recordthe preliminary causeof death and the dateof deathSolution Design %**4.9 (0.4)1527%13/11123*(72)123*/4573%**3.5 (1.1)1587%11/1183*(27)83*/4513%**3.7 **4.7*(0.6)Page 8 of 208
rved/ Optimal)36100%NTaskScenario 3b.1: AccessdocumentsScenario 3b.2: Changethe preliminary causeof death the date ofdeathTask R1.1: Access toenter patientdemographic dataTask R1.2: Enterpatient’s date of birthTask R1.3: Enterpatient’s raceTask R1.4: Enterpatient’s ethnicityTask R1.5: Enterpatient’s languageTask R1.6: Enterpatient’s language aspreferred languageTask R1.7: Enterpatient’s sexTask R1.8: Enterpatient’s sexualorientationTask R1.9: Enterpatient’s genderidentitySolution Design DocumentTask TimeMean(SD)SecDeviations(Observed/ Optimal)ErrorsScenarioRatings5 **22-Dec-1656%**4.9*(0.4)3.2*(1.0)Page 9 of 208
rved/ sk R2.1: Access tochange patientdemographic dataTask R2.2: Changepatient’s date of birthTask R2.3: Changepatient’s raceTask R2.4: Changepatient’s ethnicityTask R2.5: Changepatient’s languageTask R2.6: Changepatient’s sexTask R2.7: Changepatient’s sexualorientationTask R2.8: Changepatient’s genderidentityTask TimeMean(SD)SecDeviations(Observed/ Optimal)ErrorsScenarioRatings5 1587%13%**1587%13%**1587%13%**2.9*(0.9)Scenario 1, Task W1c Problem List (SED a.6)Scenario 1 Task A.01:Access Health Issues toAdd New Problem (afterinitial exposure)Scenario 1 Task A.02:Add New Problem RenalDysfunctionTask W1c.1: Accesspatient’s existing andhistorical problems tochange a problemTask W1c.2: UpdateSevereHypothyroidism: statusis now resolvedMedication ListScenario 2a.10: AddNew Medication FluVaccineSolution Design 4.2*(0.8)Page 10 of 208
MeasureN#TaskTask W1b.1: Access16and review patient’sexisting medicationsTask W1b.2: Changeexisting medicationCeftriaxone - update16that the last dose wastaken yesterdayeveningMedication Allergy ListScenario 5.1: Accessand review patient’s22active medicationallergy listScenario 5.2: Add newmedication allergy and21detailsScenario 5.3: Changeexisting Medication22AllergyClinical Decision SupportScenario 1 Subtask B4:Review and act uponCDS – Problem Listdisplayed for Renal36Dysfunction and/orAtrial Fibrillation andMorphineScenario 1 Subtask A1:Review and act uponCDS - Medication List26displayed for Aspirinand Renal DysfunctionScenario 2 Subtask D1:Review and act uponCDS – Laboratory36Test results displayedfor High Potassium andPotassium ChlorideReview and act uponCDS - Medication List *Problem List36(Combination)displayed for RenalDysfunction and/orSolution Design rved/ Optimal)Task TimeMean(SD)SecDeviations(Observed/ Optimal)100%ErrorsScenarioRatings5 80100%1.4/10%**100%1.6/10%**22-Dec-164.4*(0.6)Page 11 of 208
rved/ Optimal)30100%1.3/10%**30100%1.3/10%**NTaskAtrial Fibrillation andWarfarinScenario 2 SubtaskB3a: Review and actupon CDS - Vital SignsScenario 2a SubtaskB3b: Review and actupon CDS –DemographicsScenario 2a SubtaskB6: Review and actupon CDS –Medication AllergyList displayed forallergy to egg in FluVaccineScenario 1 Subtask E1:CDS - Access linkedreferential decisionsupport for medication(morphine/potassium)Scenario 1 Subtask E2:CDS - Access linkedreferential decisionsupport for problem(renal dysfunction)Scenario 2a.11: AccessSource Attributes foralerts: BibliographicCitation of theInterventionTask C1: Limit accessto alert history to a setof identified usersTask C2b: Limit CDSConfiguration to a setof identified usersTask C5: Turn onproduction alerts forthe clinical decisionsupport - ProblemsSolution Design DocumentTask TimeMean(SD)SecDeviations(Observed/ Optimal)ErrorsScenarioRatings5 302*(91)302*/1804%**4.2*(0.8)4.2*(0.8)Initial Exposure for some participants who had not performed thisadjustment during their 28)41*/207%**4.9*(0.4)Initial Exposure for some participants who had not performed thisadjustment during their Page 12 of 208
MeasureN#TaskTask C6: Turn onproduction alerts for13multiple clinicaldecision supportsImplantable Device ListScenario 4a.1: AccessImplantable Device List37after first exposureScenario 4a.2: ChangeStatus of the Bone Pin37DeviceScenario 4b: Add newimplantScenario 4c: Accessand obtain informationregarding erved/ ions(Observed/ Optimal)46*/15ErrorsScenarioRatings5 122*/7511/959%**17%**83%145*(13)9/833Task Time89%145*/12311%**3.1*(0.9)Clinical Information Reconciliation Integration (CIRI)Reconcile Medication AllergiesTask W1a.1: Detectmedications allergiesfrom outside sourceTask W1a.2: Detectmedications allergiesfrom EHRTask W1a.3: Reconcilesmedication allergiesTask W1a.4: Reviewand validate reconciledlist of medicationallergiesReconcile MedicationsTask W1b.3: Detectmedications fromoutside sourceTask W1b.4: Detectmedications from EHRTask W1b.5: ReconcilesmedicationsTask W1b.6: Reviewand validate reconciledlist of medicationsSolution Design 16100% 0%**1685%15%**1681%19%**1688%1693% 22-Dec-16Page 13 of 208
erved/ Optimal)Task TimeMean(SD)SecDeviations(Observed/ Optimal)ErrorsScenarioRatings5 Easy(%)Mean(SD)TaskCIRI Scenario 6C: Reconcile ProblemsTask W1c.3: Detectproblems from outsidesourceTask W1c.4: Detectproblems from EHRTask W1c.5: ReconcilesproblemsTask W1c.6: Reviewand validate reconciledlist of problems1587%13%**1588%12%**1587%1593% 7%**100%0%**Electronic PrescriptionTask W2.1: Create andsend new electronicprescription15Azithromycin 250 mgoral tabletTask W2.2: Assign anexisting health issue asreason for the15prescription, AcuteSinusitisTask W4.1: Cancelpharmacy prescriptionfor Woman’s Pack15multivitamin for lack offund reasonsTask W4.2: Confirmprevious prescriptionorder was cancelled for15One Daily MultiEssential Oral TabletTask W567:Prescription change via21 pharmacy requestTask W8: Determine fill15status of prescriptionTask W9: Process refill15requestTask W10: Review andadd medication from15patient medicationhistory informationSolution Design 3%87%**80% 760%13/773% -Dec-1673*/2748%**70*/3440%**91*/3227%**3.1 (0.9)137/2567%**Page 14 of 208
erved/ Optimal)Task TimeMean(SD)SecDeviations(Observed/ Optimal)ErrorsScenarioRatings5 Easy(%)Mean(SD)TaskTask W3: Limit user’sability to enter oral12100%NANANA0%NAliquid meds in ml andinsert/trailing zeros* Mean (SD) Task Time is the time to complete the entire scenario. Deviations Observed/Optimal is thetime to complete the entire scenario. Scenario Ratings are Mean (SD) ease of u
Sunrise Business Unit . Allscripts Healthcare Solutions . 222 Merchandise Mart Plaza, Suite 2024 . Chicago, IL 60654 . 1.800.334.8534 . For public release: Allscripts attests that the usability standard/process and usability report submitted for the certification of Sunrise Clinical Manager (Allscripts 'Sunrise Acute Care', and Allscripts 'Sunrise
Medgen EHR Safety-Enhanced Design Usability Report. This report is being submitted as part of the EHR certification requirements outlined in 170.315(g)(3)- Safety Enhanced Design. . Review/Configure Clinical Decision Support and Therapeutic References 12 §170.315(a)(1) CPOE §170.315(a)(2) Drug-drug, Drug- allergy interaction checks Record
usability testing is a very frequently used method, second only to the use of iterative design. One goal of this chapter is to provide an introduction to the practice of usability testing. This includes some discussion of the concept of usability and the history of usability testing, various goals of usability testing, and running usability tests.
MODELNO. 315.175040 , EAR OWNER'S MANUAL MODEL NOS. 315.175040 315.175050 315.175060 CAUTION: Read and follow ALL safety rules and instructions. before operating th_ equipment. MODELNO.315.175060 MODELNO.315.175050 CRAFTSMI:iN" Router Thank You for Buying A Craftsman Router DoUble Insulated Warranty Rules For Safe Operation Table Of .File Size: 1MBPage Count: 28
Usability Testing Formal usability testing involves real users and more resources and budget than do the discount usability techniques. Usability testing is the observation and analysis of user behavior while users use a prod- uct or product prototype to achieve a goal. It has five key components: 1. The goal is to improve the usability of a .
Usability is the quality attribute that measures the easiness of an interface [4]. Battleson et al. (2001) asserted that usability testing is the best approach to asses a website's usability [5]. The meaning of usability, overall, proposes that there are four common factors that influence usability of the interactive
9.1 Login to the New Jersey EHR Incentive Program Attestation Application . 27 9.1.1 Starting the New Jersey EHR Incentive Program Attestation Application . 27 9.2 Registering a Provider within the New Jersey EHR Incentive Program Attestation Application 30
Comment remplir l’attestation de salaire ? 2. 3 Sommaire Pages . sur l’attestation de salaire intervient le dernier jour du mois, le salaire se rapportant à ce mois est pris en compte dans le calcul de l’indemnité journalière et doit donc figurer sur l’attestation de salaire.
when using chords in your riffs and solos. mattwarnockguitar.com 14 After working this line in a few keys, experiment by adding bends to other chords in your vocabulary. Audio Example 14 The last line uses A7#9 chord shapes to create tension in this Stevie Ray Vaughan style line. Remember, you can use 7#9 chords anywhere in a blues, though they fit most naturally on the V7 chord. If you use .
