A Protocol to Measure Nursing Electronic Health Record Usability

Abstract

Purpose: The American Recovery and Reinvestment Act (Centers for Medicare and Medicaid Services, 2010) incentivized the adoption of electronic health records (EHR) in healthcare settings. The adoption rate of these systems is reported to be higher than ever before (Dustin et al, 2013). Many organizations are facing the challenge of modifying their existing system or selecting a new system that best meets their needs. A key contributor to the safe and effective use of technology is usability. Usability is an important concept that should be considered when making major EHR decisions. The usability of an EHR is defined by the National Institute of Standards and Technology (Lowry et al, 2012) as the extent to which a product can be used to achieve the goals of efficiency, effectiveness, and user satisfaction. While government mandates for meaningful use of the EHR have resulted in wide-spread purchase of hospital information systems, the literature regarding nursing EHR usability is minimal. Our research team implemented a two-phase study. The first phase focused on uncovering the variables associated with EHR usability, and selecting a satisfaction scale. The second phase presented in this paper aimed to develop and implement a protocol to measure nursing EHR usability. The purpose of this presentation is to report the results of the second phase of this study. Methods: The theoretical framework used was the Technology Acceptance Model (Venkatesh & Davis, 1996), which attempts to predict and explain the use of information systems. This framework provides a parsimonious depiction of the process seen in EHR implementation. Thirty registered staff nurses with at least two years' experience in a Midwestern hospital were recruited to participate in this study. Eight nursing tasks within three medical-surgical scenarios were constructed and then uploaded into the training component of the hospital's EHR. Nurse participants were tested in a laboratory in the hospital's computer lab set aside specifically for this study. Each scenario included the tasks of: results look up, care organization, assessment, care plan, problem list, medication administration, order entry, and discharge. The three scenarios included patients with congestive heart failure (CHF), cerebrovascular accident (CVA), and pneumonia (PN). Nurse participants were asked to record tasks using MORAE usability software by TechSmith (2012). Efficiency was measured by recording time, number of keystrokes, number of mouse clicks, and amount of mouse movement. Effectiveness was measured by the error rate. User satisfaction was measured with Brooke's System Usability Scale (1986), a ten-item Likert-scale questionnaire. Baseline scores for each of the three components of usability (efficiency, effectiveness, and satisfaction) were calculated. Efficiency scores were determined by comparing participant time, number of keystrokes, mouse clicks, and mouse movement. Nurse participants were grouped into one of three categories labeled as very efficient, efficient, and not efficient. Outliers greater than two standard deviations from the mean were removed from the analysis. Effectiveness was measured by counting the total number of errors made during each scenario. Nurse participants with zero errors were labeled as effective, those with one error were somewhat effective, and those with two or more errors were labeled not effective. The System Usability Scale conversion scores published by Sauro (2011) were used to calculate the satisfaction score. The three separate scores for efficiency, effectiveness, and user satisfaction were then converted to individual grades using a 4.0 grading scale. Each of the three grades was combined to calculate the overall usability grade/score. Results: Scores for efficiency, effectiveness and satisfaction were obtained and then converted into a grade. The system tested received a grade of C for each component of usability. In addition, we found that older nurse participants took more time to use the system compared to younger nurse participants. We found a statistically significant correlation for the pneumonia (PN) scenario (p = .043) and congestive heart failure (CHF) (p = .016), but not for cerebral vascular accident (CVA) (p = .231). However, this is likely due to a tighter age range of participants tested in the CVA scenario. We also found in the CHF scenario that, while the younger nurse participants completed their tasks in less time (more efficient), they committed more errors (were less effective) compared to the older nurse participants (p = .039). While statistical significance was not found for PN or CVA, it might have been due to our small sample size. Conclusion: This study provides information regarding the development and implementation of a protocol to measure nursing EHR usability. This protocol including the scenarios will be made available for others wishing to conduct similar usability studies. Obtaining a baseline nursing EHR usability score is important for facilities wanting to know the effect on usability following EHR modifications. Because this study is the first to be conducted on nursing EHR usability, these findings are limited to the EHR system studied and the hospital where the study occurred. More research using this nursing usability protocol will need to be completed for further comparisons across different EHR systems.