Formative Usability Testing Reduces Severe Blood Product Ordering Errors.

2019

981-990

2019 Oct

1869-0327

<p><strong>BACKGROUND: </strong> Medical errors in blood product orders and administration are common, especially for pediatric patients. A failure modes and effects analysis in our health care system indicated high risk from the electronic blood ordering process.</p>

<p><strong>OBJECTIVES: </strong> There are two objectives of this study as follows:(1) To describe differences in the design of the original blood product orders and order sets in the system (original design), new orders and order sets designed by expert committee (DEC), and a third-version developed through user-centered design (UCD).(2) To compare the number and type of ordering errors, task completion rates, time on task, and user preferences between the original design and that developed via UCD.</p>

<p><strong>METHODS: </strong> A multidisciplinary expert committee proposed adjustments to existing blood product order sets resulting in the DEC order set. When that order set was tested with front-line users, persistent failure modes were detected, so orders and order sets were redesigned again via formative usability testing. Front-line users in their native clinical workspaces were observed ordering blood in realistic simulated scenarios using a think-aloud protocol. Iterative adjustments were made between participants. In summative testing, participants were randomized to use the original design or UCD for five simulated scenarios. We evaluated differences in ordering errors, time on task, and users' design preference with two-sample -tests.</p>

<p><strong>RESULTS: </strong> Formative usability testing with 27 providers from seven specialties led to 18 changes made to the DEC to produce the UCD. In summative testing, error-free task completion for the original design was 36%, which increased to 66% in UCD (30%, 95% confidence interval [CI]: 3.9-57%;  = 0.03). Time on task did not vary significantly.</p>

<p><strong>CONCLUSION: </strong> UCD led to substantially different blood product orders and order sets than DEC. Users made fewer errors when ordering blood products for pediatric patients in simulated scenarios when using the UCD orders and order sets compared with the original design.</p>

10.1055/s-0039-3402714

Appl Clin Inform

31875648

A Heuristic Evaluation to Assess Use of After Visit Summaries for Supporting Continuity of Care.

2018

714-724

2018 07

1869-0327

<p><strong>BACKGROUND: </strong>Outpatient providers often do not receive discharge summaries from acute care providers prior to follow-up visits. These outpatient providers may use the after-visit summaries (AVS) that are given to patients to obtain clinical information. It is unclear how effectively AVS support care coordination between clinicians.</p>

<p><strong>OBJECTIVES: </strong>Goals for this effort include: (1) developing usability heuristics that may be applied both for assessment and to guide generation of medical documents in general, (2) conducting a heuristic evaluation to assess the use of AVS for communication between clinicians, and (3) providing recommendations for generating AVS that effectively support both patient/caregiver use and care coordination.</p>

<p><strong>METHODS: </strong>We created a 17-item heuristic evaluation instrument for assessing usability of medical documents. Eight experts used the instrument to assess each of four simulated AVS. The simulations were created using examples from two hospitals and two pediatric patient cases developed by the National Institute of Standards and Technology.</p>

<p><strong>RESULTS: </strong>Experts identified 224 unique usability problems ranging in severity from mild to catastrophic. Content issues (e.g., missing medical history, marital status of a 2-year-old) were rated as most severe, but widespread formatting and structural problems (e.g., inconsistent indentation, fonts, and headings; confusing ordering of information) were so distracting that they significantly reduced readers' ability to efficiently use the documents. Overall, issues in the AVS from Hospital 2 were more severe than those in the AVS from Hospital 1.</p>

<p><strong>CONCLUSION: </strong>The new instrument allowed for quick, inexpensive evaluations of AVS. Usability issues such as unnecessary information, poor organization, missing information, and inconsistent formatting make it hard for patients, caregivers, and clinicians to use the AVS. The heuristics in the new instrument may be used as guidance to adapt electronic health record systems so that they generate more useful and usable medical documents.</p>

10.1055/s-0038-1668093

Appl Clin Inform

30208496

Development and dissemination of clinical decision support across institutions: standardization and sharing of refugee health screening modules.

2019

2019 Aug 02

1527-974X

<p><strong>OBJECTIVES: </strong>We developed and piloted a process for sharing guideline-based clinical decision support (CDS) across institutions, using health screening of newly arrived refugees as a case example.</p>

<p><strong>MATERIALS AND METHODS: </strong>We developed CDS to support care of newly arrived refugees through a systematic process including a needs assessment, a 2-phase cognitive task analysis, structured preimplementation testing, local implementation, and staged dissemination. We sought consensus from prospective users on CDS scope, applicable content, basic supported workflows, and final structure. We documented processes and developed sharable artifacts from each phase of development. We publically shared CDS artifacts through online dissemination platforms. We collected feedback and implementation data from implementation sites.</p>

<p><strong>RESULTS: </strong>Responses from 19 organizations demonstrated a need for improved CDS for newly arrived refugee patients. A guided multicenter workflow analysis identified 2 main workflows used by organizations that would need to be supported by shared CDS. We developed CDS through an iterative design process, which was successfully disseminated to other sites using online dissemination repositories. Implementation sites had a small-to-modest analyst time commitment but reported a good match between CDS and workflow.</p>

<p><strong>CONCLUSION: </strong>Sharing of CDS requires overcoming technical and workflow barriers. We used a guided multicenter workflow analysis and online dissemination repositories to create flexible CDS that has been adapted at 3 sites. Organizations looking to develop sharable CDS should consider evaluating the workflows of multiple institutions and collecting feedback on scope, design, and content in order to make a more generalizable product.</p>

10.1093/jamia/ocz124

J Am Med Inform Assoc

31373356

Identifying Electronic Health Record Usability And Safety Challenges In Pediatric Settings.

2018

1752-1759

2018 11

1544-5208

<p>Pediatric populations are uniquely vulnerable to the usability and safety challenges of electronic health records (EHRs), particularly those related to medication, yet little is known about the specific issues contributing to hazards. To understand specific usability issues and medication errors in the care of children, we analyzed 9,000 patient safety reports, made in the period 2012-17, from three different health care institutions that were likely related to EHR use. Of the 9,000 reports, 3,243 (36&nbsp;percent) had a usability issue that contributed to the medication event, and 609 (18.8&nbsp;percent) of the 3,243 might have resulted in patient harm. The general pattern of usability challenges and medication errors were the same across the three sites. The most common usability challenges were associated with system feedback and the visual display. The most common medication error was improper dosing.</p>

10.1377/hlthaff.2018.0699

Health Aff (Millwood)

30395517

User Testing an Information Foraging Tool for Ambulatory Surgical Site Infection Surveillance.

2018

791-802

2018 Oct

1869-0327

<p><strong>BACKGROUND: </strong> Surveillance for surgical site infections (SSIs) after ambulatory surgery in children requires a detailed manual chart review to assess criteria defined by the National Health and Safety Network (NHSN). Electronic health records (EHRs) impose an inefficient search process where infection preventionists must manually review every postsurgical encounter (&lt; 30 days). Using text mining and business intelligence software, we developed an information foraging application, the SSI Workbench, to visually present which postsurgical encounters included SSI-related terms and synonyms, antibiotic, and culture orders.</p>

<p><strong>OBJECTIVE: </strong> This article compares the Workbench and EHR on four dimensions: (1) effectiveness, (2) efficiency, (3) workload, and (4) usability.</p>

<p><strong>METHODS: </strong> Comparative usability test of Workbench and EHR. Objective test metrics are time per case, encounters reviewed per case, time per encounter, and retrieval of information meeting NHSN definitions. Subjective measures are cognitive load using the National Aeronautics and Space Administration (NASA) Task Load Index (NASA TLX), and a questionnaire on system usability and utility.</p>

<p><strong>RESULTS: </strong> Eight infection preventionists participated in the test. There was no difference in effectiveness as subjects retrieved information from all cases, using both systems, to meet the NHSN criteria. There was no difference in efficiency in time per case between the Workbench and EHR (8.58 vs. 7.39 minutes,  = 0.36). However, with the Workbench subjects opened fewer encounters per case (3.0 vs. 7.5,  = 0.002), spent more time per encounter (2.23 vs. 0.92 minutes,  = 0.002), rated the Workbench lower in cognitive load (NASA TLX, 24 vs. 33,  = 0.02), and significantly higher in measures of usability.</p>

<p><strong>CONCLUSION: </strong> Compared with the EHR, the Workbench was more usable, short, and reduced cognitive load. In overall efficiency, the Workbench did not save time, but demonstrated a shift from between-encounter foraging to within-encounter foraging and was rated as significantly more efficient. Our results suggest that infection surveillance can be better supported by systems applying information foraging theory.</p>

10.1055/s-0038-1675179

Appl Clin Inform

30357777

Identifying surgical site infections in electronic health data using predictive models.

2018

2018 Jun 29

1527-974X

<p><strong>Objective: </strong>The objective was to prospectively derive and validate a prediction rule for detecting cases warranting investigation for surgical site infections (SSI) after ambulatory surgery.</p>

<p><strong>Methods: </strong>We analysed electronic health record (EHR) data for children who underwent ambulatory surgery at one of 4 ambulatory surgical facilities. Using regularized logistic regression and random forests, we derived SSI prediction rules using 30 months of data (derivation set) and evaluated performance with data from the subsequent 10 months (validation set). Models were developed both with and without data extracted from free text. We also evaluated the presence of an antibiotic prescription within 60 days after surgery as an independent indicator of SSI evidence. Our goal was to exceed 80% sensitivity and 10% positive predictive value (PPV).</p>

<p><strong>Results: </strong>We identified 234 surgeries with evidence of SSI among the 7910 surgeries available for analysis. We derived and validated an optimal prediction rule that included free text data using a random forest model (sensitivity = 0.9, PPV = 0.28). Presence of an antibiotic prescription had poor sensitivity (0.65) when applied to the derivation data but performed better when applied to the validation data (sensitivity = 0.84, PPV = 0.28).</p>

<p><strong>Conclusions: </strong>EHR data can facilitate SSI surveillance with adequate sensitivity and PPV.</p>

10.1093/jamia/ocy075

J Am Med Inform Assoc

29982511

Genomic decision support needs in pediatric primary care.

2017

2017 Feb 19

1527-974X

<p>Clinical genome and exome sequencing can diagnose pediatric patients with complex conditions that often require follow-up care with multiple specialties. The American Academy of Pediatrics emphasizes the role of the medical home and the primary care pediatrician in coordinating care for patients who need multidisciplinary support. In addition, the electronic health record (EHR) with embedded clinical decision support is recognized as an important component in providing care in this setting. We interviewed 6 clinicians to assess their experience caring for patients with complex and rare genetic findings and hear their opinions about how the EHR currently supports this role. Using these results, we designed a candidate EHR clinical decision support application mock-up and conducted formative exploratory user testing with 26 pediatric primary care providers to capture opinions on its utility in practice with respect to a specific clinical scenario. Our results indicate agreement that the functionality represented by the mock-up would effectively assist with care and warrants further development.</p>

10.1093/jamia/ocw184

J Am Med Inform Assoc

28339689

Patient-generated Digital Images after Pediatric Ambulatory Surgery.

2016

646-52

2016

1869-0327

<p><strong>OBJECTIVE: </strong>To describe the use of digital images captured by parents or guardians and sent to clinicians for assessment of wounds after pediatric ambulatory surgery.</p>

<p><strong>METHODS: </strong>Subjects with digital images of post-operative wounds were identified as part of an on-going cohort study of infections after ambulatory surgery within a large pediatric healthcare system. We performed a structured review of the electronic health record (EHR) to determine how digital images were documented in the EHR and used in clinical care.</p>

<p><strong>RESULTS: </strong>We identified 166 patients whose parent or guardian reported sending a digital image of the wound to the clinician after surgery. A corresponding digital image was located in the EHR in only 121 of these encounters. A change in clinical management was documented in 20% of these encounters, including referral for in-person evaluation of the wound and antibiotic prescription.</p>

<p><strong>CONCLUSION: </strong>Clinical teams have developed ad hoc workflows to use digital images to evaluate post-operative pediatric surgical patients. Because the use of digital images to support follow-up care after ambulatory surgery is likely to increase, it is important that high-quality images are captured and documented appropriately in the EHR to ensure privacy, security, and a high-level of care.</p>

10.4338/ACI-2015-12-CR-0168

Appl Clin Inform

27452477

Adoption of a Portal for the Primary Care Management of Pediatric Asthma: A Mixed-Methods Implementation Study.

2016

e172

2016

1438-8871

<p><strong>BACKGROUND: </strong>Patient portals may improve communication between families of children with asthma and their primary care providers and improve outcomes. However, the feasibility of using portals to collect patient-reported outcomes from families and the barriers and facilitators of portal implementation across diverse pediatric primary care settings have not been established.</p>

<p><strong>OBJECTIVE: </strong>We evaluated the feasibility of using a patient portal for pediatric asthma in primary care, its impact on management, and barriers and facilitators of implementation success.</p>

<p><strong>METHODS: </strong>We conducted a mixed-methods implementation study in 20 practices (11 states). Using the portal, parents of children with asthma aged 6-12 years completed monthly surveys to communicate treatment concerns, treatment goals, symptom control, medication use, and side effects. We used logistic regression to evaluate the association of portal use with child characteristics and changes to asthma management. Ten clinician focus groups and 22 semistructured parent interviews explored barriers and facilitators of use in the context of an evidence-based implementation framework.</p>

<p><strong>RESULTS: </strong>We invited 9133 families to enroll and 237 (2.59%) used the portal (range by practice, 0.6%-13.6%). Children of parents or guardians who used the portal were significantly more likely than nonusers to be aged 6-9 years (vs 10-12, P=.02), have mild or moderate/severe persistent asthma (P=.009 and P=.04), have a prescription of a controller medication (P&lt;.001), and have private insurance (P=.002). Portal users with uncontrolled asthma had significantly more medication changes and primary care asthma visits after using the portal relative to the year earlier (increases of 14% and 16%, respectively). Qualitative results revealed the importance of practice organization (coordinated workflows) as well as family (asthma severity) and innovation (facilitated communication and ease of use) characteristics for implementation success.</p>

<p><strong>CONCLUSIONS: </strong>Although use was associated with higher treatment engagement, our results suggest that achieving widespread portal adoption is unlikely in the short term. Implementation efforts should include workflow redesign and prioritize enrollment of symptomatic children.</p>

<p><strong>CLINICALTRIAL: </strong>Clinicaltrials.gov NCT01966068; https://clinicaltrials.gov/ct2/show/NCT01966068 (Archived by WebCite at http://www.webcitation.org/6i9iSQkm3).</p>

10.2196/jmir.5610

J. Med. Internet Res.

27357835

Clinical Decision Support and Palivizumab: A Means to Protect from Respiratory Syncytial Virus.

2015

769-84

2015

1869-0327

<p><strong>BACKGROUND AND OBJECTIVES: </strong>Palivizumab can reduce hospitalizations due to respiratory syncytial virus (RSV), but many eligible infants fail to receive the full 5-dose series. The efficacy of clinical decision support (CDS) in fostering palivizumab receipt has not been studied. We sought a comprehensive solution for identifying eligible patients and addressing barriers to palivizumab administration.</p>

<p><strong>METHODS: </strong>We developed workflow and CDS tools targeting patient identification and palivizumab administration. We randomized 10 practices to receive palivizumab-focused CDS and 10 to receive comprehensive CDS for premature infants in a 3-year longitudinal cluster-randomized trial with 2 baseline and 1 intervention RSV seasons.</p>

<p><strong>RESULTS: </strong>There were 356 children eligible to receive palivizumab, with 194 in the palivizumab-focused group and 162 in the comprehensive CDS group. The proportion of doses administered to children in the palivizumab-focused intervention group increased from 68.4% and 65.5% in the two baseline seasons to 84.7% in the intervention season. In the comprehensive intervention group, proportions of doses administered declined during the baseline seasons (from 71.9% to 62.4%) with partial recovery to 67.9% during the intervention season. The palivizumab-focused group improved by 19.2 percentage points in the intervention season compared to the prior baseline season (p &lt; 0.001), while the comprehensive intervention group only improved 5.5 percentage points (p = 0.288). The difference in change between study groups was significant (p = 0.05).</p>

<p><strong>CONCLUSIONS: </strong>Workflow and CDS tools integrated in an EHR may increase the administration of palivizumab. The support focused on palivizumab, rather than comprehensive intervention, was more effective at improving palivizumab administration.</p>

10.4338/ACI-2015-08-RA-0096

Appl Clin Inform

26767069