Background: The introduction of electronic workflows has allowed for the flow of raw uncontextualized clinical data into medical documentation. As a result, many electronic notes have become replete of “noise” and deplete clinically significant “signals.” There is an urgent need to develop and implement innovative approaches in electronic clinical documentation that improve note quality and reduce unnecessary bloating.
Objective: This study aims to describe the development and impact of a novel set of templates designed to change the flow of information in medical documentation.
Methods: This is a multihospital nonrandomized prospective improvement study conducted on the inpatient general internal medicine service across 3 hospital campuses at the New York University Langone Health System. A group of physician leaders representing each campus met biweekly for 6 months. The output of these meetings included (1) a conceptualization of the note bloat problem as a dysfunction in information flow, (2) a set of guiding principles for organizational documentation improvement, (3) the design and build of novel electronic templates that reduced the flow of extraneous information into provider notes by providing link outs to best practice data visualizations, and (4) a documentation improvement curriculum for inpatient medicine providers. Prior to go-live, pragmatic usability testing was performed with the new progress note template, and the overall user experience was measured using the System Usability Scale (SUS). Primary outcome measures after go-live include template utilization rate and note length in characters.
Results: In usability testing among 22 medicine providers, the new progress note template averaged a usability score of 90.6 out of 100 on the SUS. A total of 77% (17/22) of providers strongly agreed that the new template was easy to use, and 64% (14/22) strongly agreed that they would like to use the template frequently. In the 3 months after template implementation, general internal medicine providers wrote 67% (51,431/76,647) of all inpatient notes with the new templates. During this period, the organization saw a 46% (2768/6191), 47% (3505/7819), and 32% (3427/11,226) reduction in note length for general medicine progress notes, consults, and history and physical notes, respectively, when compared to a baseline measurement period prior to interventions.
Conclusions: A bundled intervention that included the deployment of novel templates for inpatient general medicine providers significantly reduced average note length on the clinical service. Templates designed to reduce the flow of extraneous information into provider notes performed well during usability testing, and these templates were rapidly adopted across all hospital campuses. Further research is needed to assess the impact of novel templates on note quality, provider efficiency, and patient outcomes.
Background and Significance
Since the Health Information Technology for Economic and Clinical Health (HITECH) Act was enacted, provider notes have lengthened with an increase in note redundancy . Across the same period, US notes have been reported as 4 times longer than those in other countries [ ]. Some claim that poor note quality is one of the unintended consequences of nationwide electronic health record (EHR) adoption [ - ]. The introduction of electronic workflows has allowed the flow of raw uncontextualized clinical data into provider notes [ , - ]. Over time, and with the change in the culture of clinical practice, many electronic notes have become full of noise, and clinically significant signals are now becoming harder to find [ , , , , ]. There is an urgent need to develop and implement innovative approaches in electronic clinical documentation that improve note quality and reduce unnecessary bloating [ , , ].
Sequence and Flow in the EHR
Significant unintended consequences to care associated with the adoption of EHR are rooted in unplanned changes to the sequence of complex clinical events. EHR interventions can have unintended changes to workflow, which is defined as the sequence of activities necessary to complete a task, and patient flow, which is described as the ordered movement of patients in a health care setting. Examples of disordered workflow after EHR adoption include changes to sequence of complex physician ordering processes after the implementation of computerized physician order entry  and changes to temporal trends in physician task workflow as well as increased time spent in clinical review and documentation [ ]. Some examples of changes in patient flow included a lack of understanding of the sequence of patient event after transition from paper to electronic flowsheets [ ] and disordered patient flow with the requirement that staff need physician orders prior to initiating “routine” clinical testing [ ].
Sequence and Flow in Clinical Thinking and Communication
Sequence also plays a role in clinical thinking and communication processes. SBAR (situation, background, assessment, and recommendation) , IPASS (illness severity, patient summary, action list, situation awareness, and contingency planning and synthesis by receiver) [ ], and SOAP (subjective, objective, assessment, and plan) [ ] are frameworks, or mental models, that advance ordered standards for thought and communication. SBAR and IPASS are used at the time of clinical handoff, and SOAP is a construct for clinical notes. These standards promote a sequential structure where adherence to sequence is a key element to improve patient safety and impact other clinical outcomes [ - ]. In this context, the Data, Information Knowledge, Wisdom (DIKW) framework is a valuable tool that has been used to study clinical communication [ - ]. The framework is composed of 4 levels, each building on the prior. Data represent the initial stage, where raw facts such as patient vital signs and laboratory results are collected. Information is the next stage, where data such as vitals trend are organized and processed. Knowledge is the third stage, where information such as diagnosis or assessment is interpreted and applied. Wisdom is the final stage, where knowledge such as considering a patient’s cultural and personal values when making a treatment plan is applied in a practical and integrated manner. By using the DIKW framework, researchers can analyze the flow of communication and identify any gaps or inefficiencies in the sequential progression from data to wisdom [ - ].
Sequence and Flow in Clinical Notes
Considering the importance of sequence and flow to clinical thinking, clinical communication, and EHR workflows in general, it is not surprising that electronic note-writing introduced features that change the flow of information and can impact the output of the note composition. The original SOAP note structure, created by Larry Weed, was intended as a data transformation engine that takes raw clinical data and transforms it into information in the “Subjective” and “Objective” sections . For instance, the patient may report a headache and difficulty sleeping (subjective), and the clinician may find that the patient’s vital signs are within normal limits but with tenderness in the temples upon physical examination (Objective). This information is then converted into knowledge in the “Assessment” section, such as the conclusion that the patient is likely experiencing a headache due to stress and lack of sleep. Finally, the knowledge is transformed into wisdom in the “Plan” section, with a recommendation for over-the-counter pain relief medication and advice for the patient to practice stress management techniques and establish a consistent sleep schedule. In this way, the note writer sequentially [ , ] ascends the DIKW pyramid [ ], while the SOAP note is composed ( ). Modern electronic note-writing workflows have disrupted this time-tested process for data transformation by allowing data and information to flow directly from the EHR into predefined note sections before the clinician thinks about the patient’s condition. Untimely information flow into note templates can thus be viewed as a root cause of poor note quality and a major contributor to the note bloat epidemic.
The objective of this study is to describe the development and impact of a novel set of templates designed to change the flow of information into provider notes ().
We describe the development process, including a description of institutional governance, guiding principles that informed template design, template creation, usability testing, and implementation.
Pragmatic Usability Testing
Usability testing is essential to the EHR implementation life cycle [- ], and testing events can help promote provider adoption [ , ]. Combining “near-live” clinical simulation with “think-aloud” protocols has been shown to help assess user preference and impact on workflow [ ]. “Think-aloud” protocols require users to verbalize their thought process while interacting with a new clinical tool [ - ]. “Near-live” testing allows for a more fluid environment to identify further real-life barriers [ , ]. “Near-live” clinical simulation with “think-aloud” protocols can be deployed together with quantitative usability assessment like the System Usability Scale (SUS) [ - ] as part of a pragmatic usability test strategy [ ] that prioritizes speed and cost-effectiveness. The SUS has been used extensively to assess the usability of the EHR [ , - ], including use to assess electronic note template usability [ ]. The 10 statements on the SUS use a 5-point Likert scale to measure the strength of agreement or disagreement with each statement. The total composite score is a number from 0 to 100, with high SUS scores indicating greater usability and satisfaction.
This is a multihospital nonrandomized prospective improvement study conducted on the inpatient general internal medicine (GIM) service across 3 hospital campuses at the New York University Langone Health System (NYULH). NYULH is a large academic health care system in New York, consisting of over 5000 health care providers across multiple hospitals and >500 ambulatory locations. NYULH’s implementation of a single EHR (Epic Systems Corporation) instance and seamless integration of ancillary systems help to facilitate a single clinical standard throughout the enterprise. To support measurement and continuous improvement for this project, EHR data were queried from the Epic System’s Clarity database using SQL Developer (Oracle Corporation) and exported for analysis of note length and template adoption. The note templates were implemented in the November 2020 version of Epic. Prior to our quality improvement intervention, the institution’s instance of Epic does not allow “copy forward” in the inpatient setting. NYULH did not have any standardized note templates prior to our intervention.
We followed the NYU Grossman School of Medicine institutional review board (IRB) protocol and completed an IRB checklist for activities that may be classified as quality improvement. This work, including our usability study, met the IRB criteria for quality improvement and therefore did not require IRB review or informed consent.
Documentation Improvement Bundle and Template Development
Our documentation improvement efforts began in September 2020, when a group of physician leaders from NYULH representing each hospital campus began to meet biweekly as part of the newly formed Documentation Standards Committee. The committee’s stated goal was to develop and implement standardized documentation and accountability processes for provider notes to improve quality and readability, while eliminating errors, and capturing the complexity of care provided.represents the timeline for the improvement initiative and committee structure, respectively. is a list of key committee stakeholders.
In the first 6 months, during the committee meetings, we conceptualized note bloat. As a result of these committee meetings, we introduced the bundled intervention that included (1) conceptualization of the note bloat problem as a dysfunction in information flow (and ); (2) a set of guiding principles for organizational documentation improvement ( ); (3) a documentation improvement curriculum for inpatient medicine providers; and (4) the design and build of novel new electronic templates that sought to reduce the flow of extraneous information into provider notes by providing link outs to best practice data visualizations.
Description of Template Features and Functions
shows the progress note template developed for the inpatient GIM service. The first implementation of new standard templates was for GIM and included: progress notes, consults, and history and physical (H&P) notes. Within these new templates, very little clinical data flow directly into the note. Instead, each template section has hyperlinks that link out to best visualizations that appear next to the template to support clinical data review. These hyperlinks provide direct access to data-specific sections within the patients’ medical record including medical and surgical history; for example, the data are visualized in a side panel next to the open note. This allows for clinical data review and simultaneous entry of relevant details into the clinical note. Providers are prompted within the template to document a synthesis of the information or knowledge relevant to the note section. Each note section contains a unique combination of hyperlinks with tailored data visualizations to support appropriate clinical thinking in that part of the note. All the hyperlinks disappear at the time of note publication.
The templates generally follow the SOAP note structure, although additional sections were added for clarity or to adapt to the modern practice of medicine. The template sections are titled as follows: Subjective & Notable Events, Physical Exam, Laboratory Test & Imaging Review, Assessment & Plan, and Discharge Milestones & Contingency Planning. This last section continues the sequential progression of the SOAP note structure by moving onto future and contingency planning, including how the patient's care is affected by health system considerations about the advancement of care and hospital discharge.
An additional feature of these novel templates is the presence of patient-specific CDS presenting as disappearing tips that appear in the body of the note.demonstrates a CDS nudge directed at providers to consider the plan for anticoagulation in patients who are on full-dose anticoagulation up to 48 hours prior to surgery. is an example of another CDS nudge to providers to consider the plan for antibiotics, upon discharge, for patients who are currently on intravenous antibiotics. The usability of CDS nudges within note sections was evaluated as part of formal usability testing as described below.
Usability Participation, Procedure, and Analysis
Prior to go-live, pragmatic usability testing was performed with the new templates. Four board-certified physician informaticists served as facilitators for 45-minute one-on-one testing sessions with frontline providers. A purposive sample of inpatient internal medicine providers was selected: 11 from NYU Langone—Long Island Hospital, 3 from NYU Langone—Brooklyn Hospital, and 8 from the Tisch and Kimmel Hospitals on the Manhattan campus. Half of the providers were attending physicians and half were residents or physician assistants. All the providers routinely write notes on hospitalized general medicine service patients.
The templates and hyperlinks to data visualizations were built within the sandbox testing environment of the EHR to function the same as when live in the production system. The testing protocol combined “near-live” clinical simulation of the note-writing processes with “think-aloud” protocol and task-based testing. In this pragmatic usability test, physician informaticists collected data including observational notes, notes on debriefing interviews, and real-time analysis of user-screen interactions. At the end of each testing session, participants completed the 10-question SUS. Summative analysis included a structured summary of themes from physician informaticists debriefing and session notes, average SUS scores, and response distribution by question for the 22 session participants. Themes and SUS scores were presented to the governance committee, and improvements were made to the template prior to go-live ().
Improvement Curriculum and Template Go-Live
Two weeks prior to go-live, all 350 internal medicine providers in the institution were assigned an electronic learning module with an introductory message from the committee chair, elucidation of guiding principles for documentation improvement, and demonstration of the new template in the electronic record. In-person roadshows were conducted in more than 15 academic conferences, quality forums, department, and division meetings across the 4 hospital campuses. The template went live on March 8, 2021. Post live, physician informaticists rounded on all 3 hospital campuses for the first week to support adoption and answer questions about the new approach to electronic documentation.
Post Live Measurement of Note Length and Adoption Statistics
Template utilization and note length in characters were the primary outcomes measured. We used the month of November prior to provider outreach as the baseline preintervention data and the 3 months immediately following implementation in March as the postintervention data.
In usability testing among 22 medicine providers, the new progress note template averaged a usability score of 90.6 out of 100 on the SUS. A total of 77% (17/22) of providers strongly agreed that the new template was easy to use, and 64% (14/22) strongly agreed that they would like to use the template frequently. At 3 months, we observed the following template utilization by note types across 3 campuses: 75% (9048/12,142) of H&P notes, 67% (41,706/62,518) of GIM progress notes, and 34% (677/1987) of inpatient GIM consult notes were written with the new templates. We evaluated template utilization by provider type—attending physicians, advanced practice providers, medical students, residents, and fellows (). Adoption rates for attending physicians among the different provider types were the highest with 85% (2958/3465), 79% (15,862/20,056), and 54% (547/1036) for H&P and progress notes, respectively, followed by advance practice provider H&P and progress notes at 80% (2387/3004) and 70% (5138/7352), respectively. Notes written by the fellows have the lowest adoption rates of 14% (27/199) for consult and 18% (11/60) for progress notes. H&P notes written by residents and medical students had a high adoption rate of 78% (3645/4648) and 71% (39/55), respectively, while the resident consult notes had an adoption rate of 45% (185/413). Progress notes written by residents and medical students had an adoption rate of 59% (10,821/18,430) and 66% (885/1337), respectively.
During this period of time across our institution, the length of notes written using the new templates decreased by 45% (3505/7819), 31% (3427/11,226), and 45% (2768/6191) for general medicine consults, H&P notes, and progress notes, respectively, compared to a baseline measurement period prior to interventions ().
By engaging our physician leaders and gaining input from key stakeholders, we developed 11 guiding principles for general medicine documentation standards that were adopted across 3 campuses. We then incorporated these standards into our template designs along with appropriately placed clinical decision support within the SOAP structure. We further refined the templates through usability testing and feedback from clinicians representing 3 campuses. With the widespread adoption of EHRs, clinical notes have increased in length to include redundant information. This has added to the documentation burden of providers while reducing the readability of these notes. Furthermore, the direct flow of raw unprocessed data from the EHR into clinical documentation compromises the quality and usability of notes. Leveraging our EHR tools and drawing on the original SOAP note structure, we applied longstanding usability practices and note template design to make our notes readable and clinically useful again.
We used note length as a measure of readability and template adoption as a measure of template utilization. Utilization of these new templates resulted in an average reduction of 40% in the note length on the general medicine inpatient service. We had concerns that the introduction of new unfamiliar templates may not be readily accepted by clinicians who prefer their own personalized templates. However, we noted a rapid adoption of these new note templates across all 3 campuses along with high usability scores on the SUS. Attending providers and advanced practice providers had high adoption rates for the H&P and progress notes, with an average of 82% and 75%, respectively. However, we did note the lowest adoption rate by fellows with an average of 16% and a lower adoption rate of progress notes written by residents and medical students with an average of 62%. Our medical students and residents likely created note templates, which leverage their progress notes for rounding. This could result in hesitance to adopt the new templates as it would be disruptive to their rounding workflow . Furthermore, the NYU General Internal Medicine department does not have fellows as part of our training program, and it was likely that other providers were incorrectly assigned as a fellow with our EHR. Our bundled intervention showed that a standardized note template that reduced the flow of extraneous information available elsewhere in the medical record was highly acceptable among most providers. Leadership buy-in with design validation is essential for effective implementation of such templates across large multicampus organizations.
This study has several limitations. This study was limited to the GIM notes. Studies would be required across other specialties to assess if a similar impact is noted in other services. In addition, consult notes had consistently lower adoption rates across all campuses. This could be attributed to a couple of different factors. Consult notes for medicine include medical clearance for surgical procedures. These consult requests were not included during the development of the consult note templates, which led to providers using their own templates to document medical optimization. We also realized that a different group of clinicians were primarily assigned to medicine consult services and were not educated about the presence of the note template, which could have led to reduced utilization of these note templates. Physician notes have several audiences including nursing, care managers, social workers, billing, coding, and other administrative staff. Feedback from such nonphysician staff members who also utilize physician notes was not included during this initial evaluation. Note length is limited as an indicator of note quality and workflow efficiency. Quality review with appropriate control for interrater reliability is necessary to further evaluate the effectiveness of the template . Also, further research is needed to assess the impact of these interventions on provider efficiency [ , ]. Lastly, although copy forward was not available, the windows copy-paste function is available, and future research is needed on the impact of copy and paste on the flow of information.
In our study, using design principles, usability analysis, and stakeholder engagement, we successfully deployed standardized note templates within a hospital’s GIM service. We believe our approach can be applied to other hospital services and disseminated to other hospital systems to improve documentation standards. Further research is needed to study the impact on note quality and workflow efficiency.
The authors thank Nader Mherabi and Suzanne Howard for their ongoing support of our clinical informatics and clinical systems teams. We also thank the entire membership of the NYU Langone Documentation Standards Committee who provided insight and assistance in ideation and implementation of the described interventions.
The data sets generated during and analyzed during the current study are available from the corresponding author on reasonable request.
Conflicts of Interest
- Rule A, Bedrick S, Chiang MF, Hribar MR. Length and redundancy of outpatient progress notes across a decade at an academic medical center. JAMA Netw Open 2021 Jul 01;4(7):e2115334 [FREE Full text] [CrossRef] [Medline]
- Downing NL, Bates DW, Longhurst CA. Physician burnout in the electronic health record era: are we ignoring the real cause? Ann Intern Med 2018;169(1):50-51. [CrossRef] [Medline]
- Kuhn T, Basch P, Barr M, Yackel T, Medical Informatics Committee of the American College of Physicians. Clinical documentation in the 21st century: executive summary of a policy position paper from the American College of Physicians. Ann Intern Med 2015;162(4):301-303 [FREE Full text] [CrossRef] [Medline]
- Goldstein IH, Hwang T, Gowrisankaran S, Bales R, Chiang MF, Hribar MR. Changes in electronic health record use time and documentation over the course of a decade. Ophthalmology 2019;126(6):783-791 [FREE Full text] [CrossRef] [Medline]
- Fackler JC. Stop the noise and get to the point. Crit Care Med 2013;41(2):656. [CrossRef] [Medline]
- Colicchio TK, Cimino JJ, Del Fiol G. Unintended consequences of nationwide electronic health record adoption: challenges and opportunities in the post-meaningful use era. J Med Internet Res 2019;21(6):e13313 [FREE Full text] [CrossRef] [Medline]
- Shoolin J, Ozeran L, Hamann C, Bria W. Association of Medical Directors of Information Systems consensus on inpatient electronic health record documentation. Appl Clin Inform 2013;4(2):293-303 [FREE Full text] [CrossRef] [Medline]
- Hultman GM, Marquard JL, Lindemann E, Arsoniadis E, Pakhomov S, Melton GB. Challenges and opportunities to improve the clinician experience reviewing electronic progress notes. Appl Clin Inform 2019;10(3):446-453. [CrossRef] [Medline]
- Colicchio TK, Cimino JJ. Clinicians' reasoning as reflected in electronic clinical note-entry and reading/retrieval: a systematic review and qualitative synthesis. J Am Med Inform Assoc 2019;26(2):172-184 [FREE Full text] [CrossRef] [Medline]
- Kahn D, Stewart E, Duncan M, Lee E, Simon W, Lee C, et al. A prescription for note bloat: an effective progress note template. J Hosp Med 2018;13(6):378-382. [CrossRef] [Medline]
- Ash JS, Sittig DF, Poon EG, Guappone K, Campbell E, Dykstra RH. The extent and importance of unintended consequences related to computerized provider order entry. J Am Med Inform Assoc 2007;14(4):415-423 [FREE Full text] [CrossRef] [Medline]
- Carayon P, Wetterneck TB, Alyousef B, Brown RL, Cartmill RS, McGuire K, et al. Impact of electronic health record technology on the work and workflow of physicians in the intensive care unit. Int J Med Inform 2015;84(8):578-594 [FREE Full text] [CrossRef] [Medline]
- Varpio L, Day K, Elliot-Miller P, King JW, Kuziemsky C, Parush A, et al. The impact of adopting EHRs: how losing connectivity affects clinical reasoning. Med Educ 2015;49(5):476-486. [CrossRef] [Medline]
- Noblin A, Cortelyou-Ward K, Cantiello J, Breyer T, Oliveira L, Dangiolo M, et al. EHR implementation in a new clinic: a case study of clinician perceptions. J Med Syst 2013;37(4):9955. [CrossRef] [Medline]
- Leonard M, Graham S, Bonacum D. The human factor: the critical importance of effective teamwork and communication in providing safe care. Qual Saf Health Care 2004;13(Suppl 1):i85-i90 [FREE Full text] [CrossRef] [Medline]
- Starmer AJ, O'Toole JK, Rosenbluth G, Calaman S, Balmer D, West DC, I-PASS Study Education Executive Committee. Development, implementation, and dissemination of the I-PASS handoff curriculum: a multisite educational intervention to improve patient handoffs. Acad Med 2014;89(6):876-884 [FREE Full text] [CrossRef] [Medline]
- Pearce PF, Ferguson LA, George GS, Langford CA. The essential SOAP note in an EHR age. Nurse Pract 2016;41(2):29-36. [CrossRef] [Medline]
- Moser EM, Fazio SB, Packer CD, Glod SA, Smith CD, Alguire PC, et al. SOAP to SOAP-V: a new paradigm for teaching students high value care. Am J Med 2017;130(11):1331-1336.e2. [CrossRef] [Medline]
- Shahian D. I-PASS handover system: a decade of evidence demands action. BMJ Qual Saf 2021;30(10):769-774. [CrossRef] [Medline]
- Müller M, Jürgens J, Redaèlli M, Klingberg K, Hautz WE, Stock S. Impact of the communication and patient hand-off tool SBAR on patient safety: a systematic review. BMJ Open 2018;8(8):e022202 [FREE Full text] [CrossRef] [Medline]
- Ronquillo C, Currie LM, Rodney P. The evolution of data-information-knowledge-wisdom in nursing informatics. ANS Adv Nurs Sci 2016;39(1):E1-E18. [CrossRef]
- Matney SA, Maddox LJ, Staggers N. Nurses as knowledge workers: is there evidence of knowledge in patient handoffs? West J Nurs Res 2014;36(2):171-190. [CrossRef] [Medline]
- Dickerson JE. Data, information, knowledge, wisdom, and understanding. Anaesth Intensive Care Med 2021;23(11):737-739. [CrossRef]
- Looman WS, Erickson MM, Garwick AW, Cady RG, Kelly A, Pettey C, et al. Meaningful use of data in care coordination by the advanced practice RN: the TeleFamilies project. Comput Inform Nurs 2012;30(12):649-654 [FREE Full text] [CrossRef] [Medline]
- Cato KD, McGrow K, Rossetti SC. Transforming clinical data into wisdom: artificial intelligence implications for nurse leaders. Nurs Manage 2020;51(11):24-30 [FREE Full text] [CrossRef] [Medline]
- Ang RJ. Use of content management systems to address nursing workflow. Int J Nurs Sci 2019;6(4):454-459 [FREE Full text] [CrossRef] [Medline]
- Giarrizzo-Wilson S. Identifying transfer of care gaps: electronic health record capture of perioperative handoff communications. Loyola University. 2020. URL: https://ecommons.luc.edu/luc_diss/3789 [accessed 2023-02-26]
- Chen Y, Yang L, Hu H, Chen J, Shen B. How to become a smart patient in the era of precision medicine? Adv Exp Med Biol 2017;1028:1-16. [CrossRef] [Medline]
- Sipes C. Nursing informatics: maintaining quality of data and information. In: Hebda T, Hunter K, editors. Application of Nursing Informatics: Competencies, Skills, and Decision-Making. New York: Springer Publishing Company; 2019:181-197.
- Rowley J. The wisdom hierarchy: representations of the DIKW hierarchy. J Inform Sci 2007;33(2):163-180. [CrossRef]
- Wright A, Sittig DF, McGowan J, Ash JS, Weed LL. Bringing science to medicine: an interview with Larry Weed, inventor of the problem-oriented medical record. J Am Med Inform Assoc 2014;21(6):964-968 [FREE Full text] [CrossRef] [Medline]
- Lin CT, McKenzie M, Pell J, Caplan L. Health care provider satisfaction with a new electronic progress note format: SOAP vs APSO format. JAMA Intern Med 2013;173(2):160-162. [CrossRef] [Medline]
- Weed LL. Medical records that guide and teach. N Engl J Med 1968;278(12):652-657. [CrossRef] [Medline]
- Mann DM, Chokshi SK, Kushniruk A. Bridging the gap between academic research and pragmatic needs in usability: a hybrid approach to usability evaluation of health care information systems. JMIR Hum Factors 2018;5(4):e10721 [FREE Full text] [CrossRef] [Medline]
- Kushniruk A, Nohr C, Borycki E. Human factors for more usable and safer health information technology: where are we now and where do we go from here? Yearb Med Inform 2016;25(1):120-125 [FREE Full text] [CrossRef] [Medline]
- Chokshi SK, Mann DM. Innovating from within: a process model for user-centered digital development in academic medical centers. JMIR Hum Factors 2018;5(4):e11048 [FREE Full text] [CrossRef] [Medline]
- Richardson S, Feldstein D, McGinn T, Park LS, Khan S, Hess R, et al. Live usability testing of two complex clinical decision support tools: observational study. JMIR Hum Factors 2019;6(2):e12471 [FREE Full text] [CrossRef] [Medline]
- Clarke MA, Schuetzler RM, Windle JR, Pachunka E, Fruhling A. Usability and cognitive load in the design of a personal health record. Health Policy Technol 2020;9(2):218-224. [CrossRef]
- Kushniruk A, Hall S, Baylis T, Borycki E, Kannry J. Approaches to demonstrating the effectiveness and impact of usability testing of healthcare health policy and technology. Stud Health Technol Inform 2019;257:244-249. [Medline]
- Morita PP, Cafazzo JA. Challenges and paradoxes of human factors in health technology design. JMIR Hum Factors 2016;3(1):e11 [FREE Full text] [CrossRef] [Medline]
- Press A, McCullagh L, Khan S, Schachter A, Pardo S, McGinn T. Usability testing of a complex clinical decision support tool in the emergency department: lessons learned. JMIR Hum Factors 2015;2(2):e14 [FREE Full text] [CrossRef] [Medline]
- Richter Lagha R, Burningham Z, Sauer BC, Leng J, Peters C, Huynh T, et al. Usability testing a potentially inappropriate medication dashboard: a core component of the dashboard development process. Appl Clin Inform 2020;11(4):528-534 [FREE Full text] [CrossRef] [Medline]
- Blecker S, Pandya R, Stork S, Mann D, Kuperman G, Shelley D, et al. Interruptive versus noninterruptive clinical decision support: usability study. JMIR Hum Factors 2019;6(2):e12469 [FREE Full text] [CrossRef] [Medline]
- Hultman G, Marquard J, Arsoniadis E, Mink P, Rizvi R, Ramer T, et al. Usability testing of two ambulatory EHR navigators. Appl Clin Inform 2016;07(02):502-515. [CrossRef]
- Brooke J. SUS: a retrospective. J Usability Stud 2013;8(2):29-40.
- Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. Int J Hum Comput Interact 2008;24(6):574-594. [CrossRef]
- Lewis JR. The system usability scale: past, present, and future. Int J Hum Comput Interact 2018;34(7):577-590. [CrossRef]
- Kelly MM, Sklansky DJ, Nackers KAM, Coller RJ, Dean SM, Eickhoff JC, et al. Evaluation and improvement of intern progress note assessments and plans. Hosp Pediatr 2021;11(4):401-405. [CrossRef] [Medline]
- Bersani K, Fuller TE, Garabedian P, Espares J, Mlaver E, Businger A, et al. Use, perceived usability, and barriers to implementation of a patient safety dashboard integrated within a vendor EHR. Appl Clin Inform 2020;11(1):34-45 [FREE Full text] [CrossRef] [Medline]
- Tran K, Leblanc K, Valentinis A, Kavanagh D, Zahr N, Ivers NM. Evaluating the usability and perceived impact of an electronic medical record toolkit for atrial fibrillation management in primary care: a mixed-methods study incorporating human factors design. JMIR Hum Factors 2016;3(1):e7 [FREE Full text] [CrossRef] [Medline]
- Belden JL, Koopman RJ, Patil SJ, Lowrance NJ, Petroski GF, Smith JB. Dynamic electronic health record note prototype: seeing more by showing less. J Am Board Fam Med 2017;30(6):691-700 [FREE Full text] [CrossRef] [Medline]
- Epstein JA, Cofrancesco J, Beach MC, Bertram A, Hedian HF, Mixter S, et al. Effect of outpatient note templates on note quality: NOTE (notation optimization through template engineering) randomized clinical trial. J Gen Intern Med 2021;36(3):580-584 [FREE Full text] [CrossRef] [Medline]
- Wiebe N, Otero Varela L, Niven DJ, Ronksley PE, Iragorri N, Quan H. Evaluation of interventions to improve inpatient hospital documentation within electronic health records: a systematic review. J Am Med Inform Assoc 2019;26(11):1389-1400 [FREE Full text] [CrossRef] [Medline]
- Stetson PD, Bakken S, Wrenn J, Siegler E. Assessing electronic note quality using the physician documentation quality instrument (PDQI-9). Appl Clin Inform 2012;3(2):164-174 [FREE Full text] [CrossRef] [Medline]
|CDS: clinical decision support|
|DIKW: Data, Information Knowledge, Wisdom|
|EHR: electronic health record|
|GIM: general internal medicine|
|HITECH: Health Information Technology for Economic and Clinical Health|
|H&P: history and physical|
|IPASS: illness severity, patient summary, action list, situation awareness, and contingency planning and synthesis by receiver|
|IRB: institutional review board|
|NYULH: New York University Langone Health System|
|SBAR: situation, background, assessment, and recommendation|
|SOAP: subjective, objective, assessment, and plan|
|SUS: System Usability Scale|
Edited by A Mavragani; submitted 21.07.22; peer-reviewed by D Chartash, D Newman, J Bell; comments to author 24.10.22; revised version received 23.01.23; accepted 15.02.23; published 12.04.23Copyright
©Jonah Feldman, Adam Goodman, Katherine Hochman, Eesha Chakravartty, Jonathan Austrian, Eduardo Iturrate, Brian Bosworth, Archana Saxena, Marwa Moussa, Dina Chenouda, Frank Volpicelli, Nicole Adler, Joseph Weisstuch, Paul Testa. Originally published in JMIR Formative Research (https://formative.jmir.org), 12.04.2023.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.