This paper is in the following e-collection/theme issue:

Formative Evaluation of Digital Health Interventions (2410) Usability of Apps and User Perceptions of mHealth (974) mHealth in a Clinical Setting (615)

Published on in Vol 6, No 5 (2022): May

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/34129, first published .
Implementation of a Personalized Digital App for Pediatric Preanesthesia Evaluation and Education: Ongoing Usability Analysis and Dynamic Improvement Scheme

Implementation of a Personalized Digital App for Pediatric Preanesthesia Evaluation and Education: Ongoing Usability Analysis and Dynamic Improvement Scheme

Implementation of a Personalized Digital App for Pediatric Preanesthesia Evaluation and Education: Ongoing Usability Analysis and Dynamic Improvement Scheme

Authors of this article:

Yaron Connelly1, 2 Author Orcid Image ;   Roni Lotan1 Author Orcid Image ;   Yitzhak Brzezinski Sinai3, 4 Author Orcid Image ;   Dan Rolls1 Author Orcid Image ;   Amir Beker1 Author Orcid Image ;   Eilone Abensour5 Author Orcid Image ;   Orit Neudorfer1, 6 Author Orcid Image ;   Daniel Stocki3, 4 Author Orcid Image
Article Authors Cited by (3) Tweetations (2) Metrics

    Original Paper

    1GistMD, Tel Aviv, Israel

    2The Israeli Center for Emerging Technologies in Healthcare, Samir Medical Center, Zerifin, Israel

    3Department of Anesthesiology and Intensive Care, Tel Aviv Medical Center, Tel Aviv, Israel

    4Sackler Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel

    5Faculté de Pharmacie, Université de Paris, Paris, France

    6Dizengoff Pediatric Community Center, Clalit Health Services, Tel Aviv, Israel

    Corresponding Author:

    Yaron Connelly, MA

    GistMD

    Stricker St 16

    Tel Aviv, 6200622

    Israel

    Phone: 972 8 9542573

    Email: yaron.connelly@gmail.com


    Background: Preanesthesia evaluation is a basic practice preceding any surgical procedure, aimed at tailoring individualized anesthetic plans for patients, improving safety, and providing patients with educational knowledge and tools in preparation for the surgery day. In the last 2 decades, eHealth and mobile health (mHealth) settings have gradually replaced part of the face-to-face encounters as the platform for preanesthesia communication between doctor and patient, yielding a range of benefits as demonstrated in recent publications. Nevertheless, there is a lack of studies examining the effectiveness of surgical mHealth apps focusing on the pediatric preanesthetic setting and addressing their usability among families.

    Objective: This study describes a dynamic approach for the development process of GistMD’s preanesthesia mHealth system, a mobile-based educational and management system designed for the pediatric setting.

    Methods: The study was conducted in 4 departments at a 1500-bed quaternary, academic medical center in Tel Aviv, Israel. During the study period, the link to the preanesthesia system was sent via SMS text messages to families whose children were about to undergo surgery. The system included preanesthesia questionnaires, educational videos, downloadable instructions, and consent forms. Continuous collection and examination of usability data were conducted during the implementation term including responsiveness, effectiveness, and satisfaction indicators. The information collected in each stage was used to draw conclusions regarding potential usability gaps of the system and to plan product adjustments for the following period.

    Results: During 141 days of implementation, the link to the GistMD preanesthesia management system was sent to 769 families, and product-fit actions were implemented during this term: (1) changing text message scheduling for addressing learnability and accessibility, resulting in a significant increase of 27% (χ21=12.65, P<.001) in view rates and 27.4% (χ21=30.01, P<.001) in satisfaction rates; (2) reducing the number of screens to increase efficiency and operability, leading to a significant decrease of 8.6% in cases where users did not perform any activity on the system after logging in (χ21=6.18, P=.02); (3) conducting a patient-focused campaign in 2 departments aimed at addressing memorability, leading to significant increases in 8 of the 12 usability indicators.

    Conclusions: Our results indicate that mHealth product-fit decisions originating from theory-based approaches and ongoing usability data analysis allow tailoring of the most appropriate responses for usability gaps, as reflected in increased use rates and satisfaction. In the case of the preanesthesia management system in the pediatric setting, increased usability conveyed important benefits for patients and families. This work suggests a framework and study methods that may also be applicable in other mHealth settings and domains.

    JMIR Form Res 2022;6(5):e34129

    doi:10.2196/34129

    Keywords

    mHealth apps; preanesthesia; pediatric setting; mHealth usability; usability analysis; mobile health; mHealth; pediatrics; anesthesia 


    Every surgical procedure that involves anesthesia is preceded by a preanesthesia evaluation session, conducted to develop a plan for anesthesia and assess the potential risks. Generally, it includes 2 essential components. First, to tailor an anesthetic plan for each patient, anesthesiologists conduct the preanesthesia clinical evaluation by gathering information from medical records, physical examinations, and patient interviews [1-3]. This basic practice aims to improve patient safety; in addition, it is important for health care organizations, as it can minimize preoperative costs, operation cancellations, and operating room time loss resulting from managing suboptimal plans [4]. Second, the process is also designed to provide patients with educational knowledge and tools that will enable them to be better prepared for the surgery day and more engaged in the anesthesia process they are about to undergo, facilitating shared decision-making and reducing preoperative anxiety, of which anesthesia is a leading cause [5-10]. The preanesthesia process is particularly important in the pediatric setting as children's anesthesia carries the risk of unique complications, such as respiratory adverse events and involves emotional engagement of the child and its parents, who usually experience significant anxiety and seek to be informed by care providers [11,12].

    In the last 2 decades, telemedicine, eHealth and mobile health (mHealth) settings have gradually replaced part of the face-to-face encounters as the platform for preanesthesia communication between doctor and patient [13], and a considerable amount of literature demonstrated the range of benefits resulting from that shift. For example, a study conducted at 4 veteran affairs medical centers in the United States revealed that the electronic consultation system allowing primary care providers and anesthesiologists access to a shared electronic record can improve workflows of anesthesiologists and allow the development of anesthetic plans without face-to-face preoperative visits [14]. Another study conducted in the United States demonstrated that telemedicine screening visits, prior to preadmission testing center appointments, decreased the time spent in hospital on the preoperative day, prevented case cancellations, and highly increased patient experience and satisfaction [15]. Nonetheless, one of the most important values that is addressed using these technologies from a patient perspective is the access to quality care. For instance, a study in Canada, where nearly 15% of the residents live in remote areas, showed that 9 out of 10 patients that live far from the hospital and 8 out of 10 anesthesiologists were highly satisfied with the experience of using telemedicine-based preanesthesia assessment [16]. In fact, similar results were demonstrated among patients from a central area with shorter travel distances and expenses [17]. The vital role of telemedicine capabilities was well demonstrated during the COVID-19 pandemic [18,19], highlighting the need to keep developing innovative solutions that will provide remote access for the preanesthesia domain, regardless of distance-from-hospital considerations.

    Although worldwide smartphone penetration reached 48.3% in 2021 [20] and more than 400,000 mHealth apps are available in app stores [21], mHealth apps in the wide context of surgical and operative settings are increasingly emerging and offer new opportunities for patients [22,23]. The efficacy of these apps was recently measured in several studies that underline the potential role of mHealth platforms in promoting patients’ comprehension and readiness prior to surgery, adherence to proper postoperative behavior and patient outcomes [23]. A South Korean study presented a significant increase in patient knowledge regarding surgical safety issues before the operation when using guidance apps [24]. In another study, anxiety and depression scores among patients with breast cancer were significantly lower when they accessed additional information provided by an mHealth app [25]. Postoperative behavior was examined in patients who underwent bariatric surgery and demonstrated a greater tendency to adopt a healthy lifestyle because of using tailored educational apps [26]. Despite the unique need, only a few studies have examined the effectiveness of surgical mHealth apps focusing on the pediatric preanesthetic setting and were designed for parents and their children. Some of these studies have found that preoperative anesthesia education provided by mobile apps can reduce anxiety and improve patient satisfaction [27,28], but, to the best of our knowledge, none of the studies addressed their usability among parents and children. In contrast, there are several examples of usability studies that are intended for health care providers, professional anesthesiologists, facilitating information sharing and management [29-32], or postoperative settings [33]. Leveraging the benefits of mHealth requires families to be highly engaged in using the apps [34]. It is evident that an incredibly low portion of the 400,000 health care apps has been successfully implemented; the total number of worldwide downloads sharply decreased recently, and hospitals can engage only 2% of their patients in mHealth activities [21]. This environment highlights the importance of evaluating mHealth apps via the prism of usability, assuming that one of the reasons that considerable proportions of users, patients, or professionals may not use the apps owing to low-quality or poor user interface (UI) or user experience (UX) components and not necessarily because of their low potential value [34]. The acceptable basic usability characteristics of systems and software products emerged in 1990 and are based on the definition of the International Organization for Standardization that encompasses effectiveness, efficiency, and satisfaction [35,36]. The Nielsen model expanded the conceptualization of usability and included learnability, memorability, and error protection [37]. The 6 main themes of usability should reflect a variety of subcharacteristics such as UI aesthetics, defined as the degree to which a product or system has attributes that make it easy to operate and control, and the degree to which a product or system can be used by people with the widest range of characteristics [36].

    Considering the lack of literature evaluating the usability of mHealth products that are designed for patients and focused on the preanesthesia period in a pediatric setting, this study aims to describe the implementation process of a mobile-based system designed for this purpose.


    Study Objectives

    The main objective of the study is to report a dynamic, quick, and timely approach for mHealth product development during its implementation period applied to modify the web-based app to family needs and address usability gaps as quickly as possible. This agile process included ongoing collection of usability data from day 1 and theory-grounded thinking. The secondary aim is to describe the lessons learned and suggest recommendations to increase the usability of these types of systems.

    Web-Based App

    The mHealth system used in this study was developed through a partnership between an industry vendor (GistMD, Ltd) and a 1500-bed general academic medical center in Israel (Tel Aviv Sourasky Medical Center). The medical center is the largest acute care facility in Israel. GistMD is a digital health company founded in 2018 and develops platforms that enable scalable production of personalized content and media for improving patient education, engagement, and adherence.

    The system was designed to address three main functions in the pediatric preanesthsesia meeting: (1) providing families and children with personalized education regarding the anesthesia process and the required preparations prior to surgery, (2) performing a remote preanesthesia evaluation possibly avoiding the need for a face-to-face encounter prior to the day of surgery, and (3) giving families the opportunity to download instructions, consent forms, and other relevant information prior to the day of surgery.

    The process includes the following steps. A text message is sent to the families’ mobile phones in conjunction with scheduling the surgery for the children and entering minimal demographic data into the system (gender and age group 0-8 or 9-18 years). The message is sent from the medical center and contains a hyperlink for the web-based system. The web-based app is compatible with all types of smartphones. Initially, the guardians are asked to indicate their preferred language for all the content in the app. Currently available languages are Hebrew, English, Arabic, Russian, and Spanish. The app customizes the content according to the medical condition for which the child was referred for surgery and demographics (eg, language, age, gender, and background diseases), displaying highly personalized material. In total, the system’s recombination engine can produce 80 different versions of the video, whereas the families can reboot the system at any stage and dynamically adjust the preferences or customize a new version of the video (eg, when different users want to watch the video in different languages). The guardians were also requested to answer a 10-item preanesthesia evaluation questionnaire aimed at identifying patients with a medical background requiring face-to-face evaluation by an anesthesiologist prior to the day of surgery. The questionnaire is an in-house development of the Tel Aviv Medical Center, validated among 200 patients for multilingual versions since 2019. It has become a standard of care at the medical center since the middle of 2020. Immediately after filling out the questionnaire, the family has the option to watch a tailored 5-minute animated video aimed at assisting them to better prepare for the day of surgery. After watching the video, parents can download various documents to review, including the informed consent for anesthesia, written instructions for fasting guidelines before anesthesia, a checklist of documents to bring on the day of surgery, and a list of new onset symptoms that require attention prior to surgery. The family is requested to provide feedback by stating whether the video was helpful or not. As illustrated in Figure 1, the characters and the storyline in the video, as well as the documents, are adjusted according to the combination of demographics entered.

    ‎
    Figure 1. Illustration of various combinations of visual content.
    View this figure

    Study Population

    Between August 9, 2020, and December 31, 2020, the app was offered to all families whose children were scheduled for surgery in 4 pediatric surgical departments at Dana-Dwek Children’s Hospital, the pediatric facility of the Tel Aviv Sourasky Medical Center in Israel. The medical center serves a population of 1 million people from Tel Aviv and the greater metropolitan area, a central location that is typically characterized by higher socioeconomic levels. Dana-Dwek Children’s Hospital provides medical services and treatment for children of all ages, from newborns to 18 years of age and approximately 3500 pediatric surgeries are performed at the hospital annually. The 4 departments selected for the study represent approximately 80% of the surgical capacity of the pediatric hospital in the study term and specialize in plastic surgery, urology, orthopedics, and otolaryngology.

    Data Collection

    Surgery coordinators in the 4 pediatric surgical departments were equipped with an operation interface that supports the web-based app. The operation interface and the web-based app are cloud-based and external to the hospital’s information technology infrastructure. After the coordinator scheduled a surgery and the system sent a text message to the guardians of the child, usability data were automatically retrieved for each user without the need for intervention or active collection among families. The collected data were encoded and anonymized, and the personal details of the patients or their family could not be identified.

    Ethical Considerations

    This manuscript is not considered human subjects research and is not presenting any medical information, and thus does not require ethical approval.

    Measurements

    Usability indicators measured in this study were focused on the degree of usage of the various functions of the app. All measures were coded as dichotomous variables, with 1 indicating that the users used the function and 0 indicating that they did not. At the basic level, the initial indicator is the responsiveness to logging on to the system, showing whether the users open the link they received in the text message. Similar threshold indicators are common in similar usability tests of apps [34]. At the second level, effectiveness indicators examined whether the users used each further function (filled out and completed the questionnaire, played the video, downloaded the informed consent and the documents with instructions). Additional indicators refer to determining whether users meaningfully and efficiently used the educational function. Previous studies have measured the usability of educational content in health apps by assessing the extent of use, such as the number of modules completed or the average viewing time [26,31]. In addition to the measurement that examined whether the users clicked “play” and started watching the video, meaningful use of the educational function was measured in this study by examining whether the participants viewed 75% or more of the video’s total duration. The first 75% of the video was chosen as the cutoff point, as it contains the essential content to be provided to the families, whereas the remaining 25% contains, for example, wishes from the medical staff and credit roll. In this regard, reverse indicators examining underuse of the system were also measured by examining whether the participants viewed 50% or less of the video’s total duration and whether the users accessed the link but did not follow on with any activity. Finally, the satisfaction indicator refers to the user perspective regarding the opportunities that the system suggests. Unlike satisfaction that corresponds to the degree of enjoyability of apps [38], the measure used in this study examined whether it was appropriate for the needs of the participants and was perceived as providing them utility [36,39]. In this study, satisfaction was measured by asking participants, “Do you find the system helpful?” The response options were “Yes” and “No.” Answering the question was not mandatory and participants could use all functions without providing an answer.

    Procedures

    For a quick product-fit process, we applied dynamic and ongoing processes of usability assessment. First, the implementation process for the web-based app was divided into 4 periods, and toward the end of each period, an examination of the usability indicators was conducted. All data were collected automatically, and the information gathered in each stage was used to draw hypotheses regarding potential current usability gaps of the system. Next, up-to-date literature and theories on the usability of software applications and mHealth in particular were reviewed to tailor product adjustments for addressing the gaps in the following period. Table 1 describes the thinking processes that yielded the product-fit strategies and actions taken to address the usability gaps during the market-product fit process in all stages of implementation.

    Table 1. Product-fit actions by strategies and related usability gaps.
    PeriodCases (N)Duration (days)Usability gapDecision on strategyDecision on action
    1: Launching28634Poor usage rates of basic featuresAddressing learnability and accessibility issues by considering the daily habits of end usersChange in text message scheduling
    25622Unsatisfied rates of underuse tendencyImproving efficiency and operability by applying lean and agile thinking for product modificationReduce number of screens
    323854Usability gaps between different departmentsAddressing memorability problems by sending notificationsFamily- focused campaign
    4: End of implementation18931N/AaN/AFollow-up and summarizing

    aN/A: not applicable.

    Period 1: Addressing Learnability and Accessibility
    Identify the Gap

    The web-based system was launched on August 9, 2020. At 34 days after the initial distribution, we conducted a preliminary evaluation of its usability indicators. The findings (as specified in the results section below) revealed that although users opened the link for the system and filled out the questionnaire, the usage rates of users who played the video and started watching (33%) and downloading the documents (1%) were extremely low. These poor results led us to conduct a deeper viewing analysis suggesting that although users get the link for the system in the morning (as was the custom at first), more than a third of them preferred to watch the video in the afternoon, and those who watched the video in morning hours tended to watch shorter parts of it.

    Choose a Strategy and Tailor the Solution

    In the context of usability, learnability should reflect the degree to which the system can be used to achieve the desired goals of learning and reap its benefits. Learnability can be facilitated by integrating the system with the daily life and habits of users, while accounting for the amount of time required for users to perform the process [35,37]; this type of modification can also address the accessibility aspect of usability while allowing more users with the widest range of characteristics and capabilities to use the system [36]. Identifying the gap between the time when users receive the link to the system and their preferred hours to use it in practice led us to conclude that the required step is to change the hour when the link is sent to users from morning to afternoon, when users are more likely to use the feature and invest more effort for the time-consuming activity of watching the video.

    Period 2: Addressing Efficiency and Operability
    Identify the Gap

    The second period occurred between September 14 and October 6, 2020, and it yielded 56 cases. Toward the end of the period, we re-evaluated the usability indicators and analyzed the underuse trends. As specified in the results section below, although all the usability indicators increased, particularly the rate at which families that opened the link increasing by 10%, we expected that the rate at which users logged into the system by opening the link without any activity would decrease in relation to the high value it showed in the first period, but it remained stable with a high and dissatisfying rate of 11%. The increase in the rate of system logins, alongside the high rate of underuse, led us to hypothesize that users may want to use the system but find it cumbersome and it may be possible to improve UX.

    Choose a Strategy and Tailor the Solution

    When it comes to software products, the efficiency aspect of usability can be attributed to the stability of the software as well as the quality and aesthetics of the UI that make the product easy to operate and control [35,36]; therefore, user-centered interface designing is strongly related to usability [40,41].

    A key strategy for improving usability by simplifying the UX relies on “agile” and “lean” thinking of designing software products [41,42] and mHealth apps in particular [35,43]. Agile and lean design underlines the need to produce high customer value while minimizing the elements that do not provide value for providers or users. From these perspectives, individual functions are implemented in the smallest possible steps [41,44,45]. The solution chosen to mitigate the underuse tendency was reducing one of the screens [46]. Initially, instructions for the family prior to surgery were displayed on a screen separate from the video player, and the download features of the informed consent and the printable detailed instructions. Users were required to click on a button to move from the first screen to the second. After the modification, all functions were merged into 1 screen, allowing a more compact app whose flow of use is shorter and simpler. All features were accessible to users by just scrolling down. Figure 2 presents screenshots from the app that illustrate the differences between the full version and the simplified one.

    ‎
    Figure 2. Illustration of the differences between the 2-screen and 1-screen versions of the app.
    View this figure
    Period 3: Addressing Memorability
    Identify the Gap

    After the measures aimed to mitigate the tendency to underuse the system, the third period of implementation took place between October 7 and November 30, 2020, and this yielded 238 cases. In this stage, usability indicators were examined across different departments in the first 3 periods. As specified in the results section, the system performance in 2 departments (C and D) was significantly lower compared to that in the other 2 departments (A and B).

    Choose a Strategy and Tailor the Solution

    Memorability refers to the level of ease with which users, families in this case, can recall how to use an app even after discontinuing its use for some time [47]. Effective notification management is important for maintaining memorability of mobile apps among users and consequently increasing the usability potential [48]. In practice, departments C and D were targeted in a focused campaign aimed to strengthen memorability and thus increase their usability performance and reduce the gaps between them and departments A and B. Families whose children are about to undergo surgery received telephone and SMS text message notifications regarding the opportunity to use the web-based system and the benefits of using it. Figure 3 illustrates a screenshot of the message.

    ‎
    Figure 3. Illustration of a notification sent to families as part of the campaign.
    View this figure
    Period 4: Follow-up and Examining Trends Throughout the Implementation Term

    At the end of the implementation term, usability indicators across departments were compared to assess the impact of product-fit actions applied in the third period. In addition, trends related to all usability indicators and with respect to the entire sample were examined. The final implementation period was between December 1 and 31, 2020, and it yielded 189 cases.

    Statistical Analysis

    SPSS software (version 26; IBM Corp) was used to perform data analysis. To examine the effect of the strategic actions of product fit, chi-square tests were performed for comparing all usability indicators (dichotomous variables) during each of the periods in comparison to the prior periods. The percentage of users who opened the link was calculated considering all participants to whom a message was sent. The percentage of users with respect to all other usability metrics was calculated only from the total number of users who opened the link.


    Sample

    During the 141 days of the implementation term, the GistMD preanesthesia system was delivered to 769 families of children who were about to undergo surgery at the Tel Aviv Sourasky Medical Center. The system was initially launched in 4 departments. Table 2 presents the characteristics of the pediatric patients whose families received the link to the system prior to their surgery from the various departments.

    Table 2. Patient characteristics by department (N=769).
    CharacteristicDepartment ADepartment BDepartment CDepartment DTotal
    Age in years, n (%)

    		0-866 (67.3)145 (82.9)158 (66.7)70 (27)439 (57.1)

    		9-1832 (32.7)30 (17.1)79 (33.3)189 (73)330 (42.9)
    Gender, n (%)

    		Male67 (68.4)154 (88)152 (64.1)133 (51.4)506 (65.8)

    		Female31 (31.6)21 (12)85 (35.9)126 (48.6)263 (34.2)

    Step 1: Change in Text Message Scheduling

    The launch period was between August 9 and September 13, 2020, yielding a total of 286 cases. The initial findings indicate that 227 (79.4%) families received and opened the link provided by the system, 195 (85.9%) opened the link and responded to the questionnaire, which is the first step in the system’s operational flow; however, only 75 (33%) of them played the video and started watching it. Moreover, only 3 (1%) and 4 (1%) families downloaded the informed consent form and the instructions document, respectively. Finally, the number of users who answered “YES” to the question “Was the system useful to you?” corresponded to only 7% (15/227) of the families who received and opened the link.

    The second period was between September 14 and October 6, 2020, and it included 56 families who received the link, with only 89% (n=50) opening it. Chi-square tests showed marginal significance indicating a 9.9% increase in the second period compared to the first one (χ21=2.99, P=.06). However, strong statistical significance was observed for the other usability indicators. First, 30 (60%) of the parents opened the link and played the video, which is an increase of 27% in comparison to the first period (χ21=12.65, P<.001). Second, the proportion of parents who answered the questionnaire but did not watch the video decreased from 52.9% (120/227) in the launching period to 28% (14/50) in the second period (χ21=10.14, P=.001). Third, the download rates of informed consent forms and instruction documents increased by 18.7% (χ21=31.96, P<.001) and 20.2% (χ21=32.76, P<.001), respectively. Finally, 34% (17/50) of the families actively indicated that the system was helpful for them, an increase of 27.4% in comparison to the first period (χ21=30.01, P<.001).

    Step 2: Reduce Number of Screens

    The third implementation period was between October 7 and November 30, 2020, and it included 238 cases. Results indicate that the families who opened the link without any activity in the system significantly decreased from 12% (6/50) in the second period to 3.4% (7/238) in the third (χ21=6.18, P=.02). The number of cases in which the video was stopped before reaching its midpoint also decreased from 20% (6/30) to 8.6% (12/140). Chi-square tests demonstrated a marginal significance (χ21=3.41, P=.07). It is noteworthy that all the values of the usability parameters, which increased in the second period, were retained or increased in the third, but without significant differences; for instance, the satisfaction indicator rose from 34% (17/50) in the second period to 42% (88/206) in the third period (χ21=0.86, P=.17).

    Step 3: Patient-Focused Campaign

    The fourth period was form December 1 to 31, 2020, in which 124 cases were retrieved and analyzed. Table 3 presents usability comparisons across different departments in the first 3 periods. The system performance in 2 departments (C and D) was lower than that in the others (A and B). For example, the link opening rate in department A was 90.8% (59/65), whereas that in department D was only 71% (142/200) (χ21=10.47, P<.001). In addition, the difference in the informed consent download rate between department B (26/129, 20.2%) and department C (13/153, 8.5%) is also significant (χ21=8.12, P=.005).

    Considering these gaps, departments C and D were targeted in a focused campaign, and a follow-up examination was performed in period 4. As observed in Table 3, the campaign led to a significant increase in some of the usability indices in departments C and D. However, during the campaign, some of the usability indicators of departments A and B, which were not part of the campaign, decreased, except for 1 (download of informed consent document); the decreases were not significant.

    Table 3. Usability indicators before and after the focused campaign across departments.a
    DepartmentPeriod 1 (N=238)Period 2 (N=56)Period 3 (N=238)Periods 1 to 3 (N=580)Period 4 (N=189)χ2 (df)P value
    Department A (not targeted during the campaign)n=8n=13n=44n=65n=33


    		Opened the link (%)889291911003.25 (1).08

    		Responded to the evaluation questionnaire (%)100679892910.01 (1).6

    		Played the video and started watching (%)71588075730.38 (1).52

    		Watched more than 75% of the video (%)57586864521.46 (1).16

    		Downloaded informed consent form (%)017352794.2 (1).03

    		Downloaded instructions document (%)025232050.38 (1).38

    		Indicated that the app is helpful (%)0254536360.001 (1).56
    Department B (not targeted during the campaign)n=76n=11n=56n=143n=32


    		Opened the link (%)86919690.2910.005 (1).62

    		Responded to the evaluation questionnaire (%)8810098931002.14 (1).15

    		Played the video and started watching (%)31806950.4662.17 (1).1

    		Watched more than 75% of the video (%)23606141.9480.4 (1).34

    		Downloaded informed consent form (%)3403720.2240.23 (1).4

    		Downloaded instructions document (%)3403318.6170.03 (1).55

    		Indicated that the app is helpful (%)8404425.6412.9 (1).07
    Department C (targeted during the campaign)n=77n=25n=70n=172n=65


    		Opened the link (%)87889189910.17 (1).44

    		Responded to the evaluation questionnaire (%)78969888.91007.27 (1).003

    		Played the video and started watching (%)16507043.88124.21 (1)<.001

    		Watched more than 75% of the video (%)13326336.66614.99 (1)<.001

    		Downloaded informed consent form (%)014168.53623.22 (1)<.001

    		Downloaded instructions document (%)014147.83422.56 (1)<.001

    		Indicated that the app is helpful (%)8324225.56427.88 (1)<.001
    Department D (targeted during the campaign)n=125n=7n=68n=200n=59


    		Opened the link (%)70867171833.42.04

    		Responded to the evaluation questionnaire (%)90839290.1961.58 (1).17

    		Played the video and started watching (%)44675448.6612.33 (1).08

    		Watched more than 75% of the video (%)34504438554.35 (1).03

    		Downloaded informed consent form (%)117218.580.004 (1).61

    		Downloaded instructions document (%)217219.2100.05 (1).51

    		Indicated that the app is helpful (%)65040194311 (1)<.001

    aThe italicized values are statistically significant.

    Step 4: Summarizing the Implementation Phase

    As observed in Figure 4, analysis of the usability data for the entire sample and throughout all the stages of implementation reveals that 4 of the indices peaked in the fourth period (ie, opening the link, responding to the questionnaire, watching the video, and the satisfaction indicator). However, 3 of the indices peaked in the third period.

    They demonstrated a slight decrease in the fourth period (ie, watching more than 75% of the video, responding to the questionnaire, and downloading the instruction document). Further examination of the trends only among the departments targeted during the campaign (C and D) showed that all indicators demonstrated an increase between periods 3 and 4, including those that demonstrated a decrease at the level of the entire sample. All indicators showed overall increases during the implementation period (from period 1 in comparison to the end of period 4). Chi-square tests were performed for examining the differences between indicator values at the end of period 1 and those at the end of the implementation period (period 4), revealing that all were significant at the level of P<.001.

    ‎
    Figure 4. Trends in usability indicators throughout all the implementation stages.
    View this figure

    Overview

    eHealth and mHealth platforms are gradually replacing face-to-face encounters for preanesthesia communication and evaluation, underlining their potential to streamline the process that is crucial to ensure successful surgery. Although realizing the potential of these digital solutions depends on adequate engagement among the users (ie, patients, professionals, and caregivers), it is essential to examine their usability indicators and identify barriers that might reduce the likelihood of optimal use among those who can benefit from these products. Nevertheless, studies addressing the usability characteristics of surgical mHealth apps focusing on the pediatric preanesthetic setting and designed for parents and their children are scarce. To bridge this gap, this study examined the usability of web-based preanesthesia apps that are designed for the pediatric setting and describes a product-fit process during its implementation period. Specifically, we describe 3 product modifications performed to improve the usability of the system, based on insights that emerged from the synergy between existing user data as well as academic and professional theoretical knowledge.

    Principal Findings

    Our results indicate that choosing a dynamic and theory-grounded approach to product-fit adjustments during the implementation period leads to significant increases in all the usability metrics of the system (see Figure 4). Each period of the study was characterized by a different gap that was identified and addressed with a tailored response considering the conceptual principles of usability. The first action that aimed to address learnability and accessibility by considering the daily habits of end users (ie, change in text message scheduling) led to higher numbers of users playing the educational video and starting to watch it as well as higher questionnaire completion rates. This finding is consistent with past studies that have shown that patients having the option to receive customized flexible text messages scheduled in their mHealth apps increased their effective usage of these apps [49], and this eventually improved their health outcomes [50]. In the context of the first action taken, existing studies also suggest that users should be directed to use the app at a time that best suits them and implies that the solution should bring together the concept of usability and data science. This broad perspective indicates that proper scheduling of push notifications for smartphones has become a major challenge these days, where users receive large amounts of various information on their mobile device that cause information overload and influence their ability to pay attention to the information that we seek to provide them [51]. The data from this study revealed that users prefer to use the system in the afternoon, which may indicate that there are families with children about to undergo surgery, and they may need time without interruptions for learning and preparing for an emotionally involved event. Alternatively, but not in contradiction, if we choose to “speak about usability,” we will ask ourselves whether the families may want to coordinate app usage at a time when they can do so together with their children to enable them to be engaged in the process.

    The second action, aimed to address efficiency and operability by reducing the number of “screens,” was reflected in our collected data indicating reduced instances of app underuse. This finding suggests that interfaces failing to deliver a positive UX may be a barrier to usage for an audience of users who have expressed a willingness to use the app, passed the initial threshold, and clicked on the link. By reducing the number of screens, we tried to apply a lean approach for the app that aimed to provide better customer value using lower capacity; in fact, this approach can present the entire process demonstrated in this study in a different light. In recent years, “lean thinking” demonstrates one of the theoretical meeting points between usability and UX of apps as well as for developing software products using the agile process in which the developers initially release a Minimum Viable Product (MVP) [41]. The UX of the product designed using an agile process will be tested according to the lean approach by receiving user feedback and constantly examining the usability indices of the product [52]. After analyzing the accumulated data, a new cycle will begin, and an updated MVP will be released for users in another agile process. The dynamic process of the app development described in this study also involved cycles of product modifications derived from data analysis of its usability metrics. This iterative scheme is previously demonstrated in studies that described the development process of mHealth apps and its potential to produce a better product [53] as well as increase its usability [54].

    The third action aimed to address the memorability of the app among families and was implemented in response to the data analysis we performed to examine the usability indicators in each of the departments and the findings related to 2 departments that demonstrated low usability performance in comparison to the others. The focused notification campaign was able to bring about a considerable increase in the usability metrics at the departments where we hoped to achieve improvement (C and D). Previous studies have also highlighted the importance of better connectivity between providers and users of mHealth apps; using notifications for this purpose had a positive influence on the product assessment [55] and usability [56].

    Although we achieved the desired result, we also found that increases in the metrics for departments A and B were characterized by higher usability indices in the baseline assessment (periods 1 to 3); therefore, these metrics were not targeted during the campaign, where extreme and modest decreases in some of the metrics during the fourth period were observed (See Table 3). The relatively weak performance in departments A and B at period 4 affected the overall results and can explain the negative trend in the values of the 3 indicators that demonstrated a decrease in the fourth period with respect to the entire sample. Contrary to the overall negative trend, departments C and D showed an increase in all indices (see Figure 4). These findings suggest that latent usability barriers may emerge when high usability indicators are demonstrated, and this highlights the need to examine usability patterns at diverse levels of analysis, such as tuning the app according to population-specific or department-specific needs. From a different viewpoint, usability may also be improved in groups of already highly engaged users, and this can further motivate them to use the app.

    Lessons Learned

    There are 2 main lessons to be learned based on our positive experience with the dynamic implementation of the web-based app. First, mHealth product-fit decisions originating from usability data analysis can be reflected in increased app use and satisfaction levels. In a system designed for the pediatric setting, increasing usability involves providing important benefits (ie, education, reducing anxiety) to patients and families who initially expressed a willingness to use the system (ie, logged in and checked its functionality) but did not use it optimally, if at all they used the system. In this study, several cycles of usability analyses were performed followed by product modifications to suit the user requirements. It is possible to link this process to the concepts of agile and lean design and development, but its main strength is different and lies in its inherent ability to facilitate theory-grounded thinking aimed at interpreting data and registering metadata. The combination of the lean design principles, data sciences theory, and conceptualization of product usability characteristics enabled modifications and successful improvements to the app. Second, we realized that satisfactory usability metrics do not necessarily indicate that product-fit modifications should not be made. In the last action, which included a campaign targeting patients from 2 departments, we did not concentrate product-fit efforts in the 2 departments where we assumed the app performance was good. From the decreased usability indices observed in these departments, we can learn that maintaining usability is an ongoing process that requires deepening of the theory-grounded thinking and efforts to refine the methods of analyzing the data retrieved from the usability patterns of our users.

    Study Strengths and Limitations

    The present study has 2 main strengths. First, to the best of our knowledge, this is the first study to examine usability of surgical mHealth apps that focused on the pediatric preanesthetic setting and was designed for parents and their children. Addressing the unique needs of children and families prior to surgery is particularly important owing to the emotional involvement in the preparation for the operation and information seeking by the parents. Usability analyses of mHealth solutions are aimed at maximizing the potential of exposure and usage of these solutions among patients and families who need them the most. Second, although most studies examining the usability of apps in the surgical setting relied on small samples, this study is based on comparative examination of 769 cases, thus collecting better quality data for understanding the usability of the app among patients and their families.

    The notable limitations of the study concern the sample and its potential lack of representativeness. First, the fact that the system required using smartphones excluded families without access to these devices, or those having low technological orientation or other limitations preventing them from using it in practice. In this context, the reliance on smartphones was chosen to strengthen the elements of personalization and anonymization for the children and their families, which could have weakened if we had expanded the distribution channels of the system. Second, the study was conducted in a medical center in a central metropolitan area that is characterized by high socioeconomic levels, and this may lead to sample bias based on income or education levels. Nevertheless, because no personal information such as socioeconomic status or health literacy was collected from families, we could not control these factors and preform systematic sampling. These constraints might limit the generalizability of our findings and conclusions, but it helped us obtain a broad picture of how the system was used and even yielded statistically significant findings. Finally, because this study was designed to examine a newly launched mHealth system and to apply ongoing, quick, and timely solutions for emerging usability gaps, we did not consider collecting available data because it was time-consuming. These data, such as patient testimonials, may have contributed to the understanding of system usability. Future studies may be able to leverage knowledge derived in this manner using the approach we have proposed in this study.

    Looking Toward the Future

    The findings this study yielded the current version of the system, which showed stable usability performance. Therefore, future studies may be conducted with considerably less time constraints. A major aspect that should be addressed in future studies is expanding our toolbox of measurements to assess the usability of future versions of the system, including collection of qualitative feedback from patients before and during implementation of new versions, to gain deep insights about UX and the use of validated questionnaires in the field of usability after implementation. When we think about the desired growth direction of pediatric preanesthesia apps, we believe that increased usage and satisfaction levels of patients should also be accompanied by increased satisfaction levels of hospital personnel. The app that is the focus of this study has high utility for professional anesthesiologists and the hospital management as well as patients and their families, and these aspects have not been addressed here. In our further studies, we would like to examine, for example, the extent to which the system makes it possible to eliminate face-to-face meetings and how this will affect staff satisfaction with the system and the attempt to persuade patients to use the app. We assume that a dynamic, theory-grounded process like the one presented here will be developed, in which we will examine the usability of the app from the perspective of the treating team; this will be add to the knowledge gained from the patients, close the feedback loop, and may also enable autotuning of the system in future.

    Conclusions

    Usability analysis of mHealth solutions is crucial for maximizing the potential benefits that patients can receive. In this study, we developed a successful dynamic process of usability examination and product-fit adjustments of a preanesthesia app designed for the pediatric setting. A dynamic and ongoing process is suggested for analyzing and interpreting usability data. This process includes timely data collection and using of theory-based thinking to tailor the most appropriate response for addressing usability gaps. Because every change is implemented to address a need arising from earlier usage, this framework may facilitate values such as communication, reliability, and responsiveness that in turn strengthen the appropriateness recognizability of the app from the user perspective. This framework can also potentially be applicable in other mHealth settings and domains. Future studies will be able to refine the method to apply it in different settings and expand the areas of knowledge that will use it.

    Acknowledgments

    We thank Professor Idit Matot, Director of the Anesthesia, Pain and Intensive Care Division, and Director of General Surgery Division and Professor Ronni Gamzu, Chief Executive Officer of Tel Aviv Sourasky Medical Center, for their support, time, and efforts. This study was supported by a grant from the Israel Innovation Authority.

    Authors' Contributions

    YC and RL contributed to data collection and analysis, and they were the primary drafters of the manuscript, tables, and figures. AB, DR, EA, and ON provided critical inputs regarding the study vision and design. AB, YBS, and DS critically revised the paper for intellectual content. All authors have approved the manuscript for publication.

    Conflicts of Interest

    YC and RL are employees and equity holders of GistMD. DS, YBS, and ON are advisors and equity holders in GistMD. AB and DR are cofounders of and equity holders in GistMD. EA has no conflicts of interest.

    1. Lozada MJ, Nguyen JTC, Abouleish A, Prough D, Przkora R. Patient preference for the pre-anesthesia evaluation: telephone versus in-office assessment. J Clin Anesth 2016 Jun;31:145-148. [CrossRef] [Medline]
    2. Almeshari M, Khalifa M, El-Metwally A, Househ M, Alanazi A. Quality and accuracy of electronic pre-anesthesia evaluation forms. Comput Methods Programs Biomed 2018 Jul;160:51-56. [CrossRef] [Medline]
    3. Shih PY, Zheng MC. Usability study of a pre-anesthesia evaluation app in an university hospital: before the revision of user interface. In: Stephanidis C, Antona M, editors. HCI International 2020 - Posters. HCII 2020. Communications in Computer and Information Science. Cham: Springer; Jul 2020:480-487.
    4. Committee on StandardsPractice Parameters, Apfelbaum JL, Connis RT, Nickinovich DG, American Society of Anesthesiologists Task Force on Preanesthesia Evaluation, Pasternak LR, et al. Practice advisory for preanesthesia evaluation: an updated report by the American Society of Anesthesiologists Task Force on Preanesthesia Evaluation. Anesthesiology 2012 Mar;116(3):522-538 [FREE Full text] [CrossRef] [Medline]
    5. Jlala HA, French JL, Foxall GL, Hardman JG, Bedforth NM. Effect of preoperative multimedia information on perioperative anxiety in patients undergoing procedures under regional anaesthesia. Br J Anaesth 2010 Mar;104(3):369-374 [FREE Full text] [CrossRef] [Medline]
    6. West AM, Bittner EA, Ortiz VE. The effects of preoperative, video-assisted anesthesia education in Spanish on Spanish-speaking patients' anxiety, knowledge, and satisfaction: a pilot study. J Clin Anesth 2014 Jun;26(4):325-329. [CrossRef] [Medline]
    7. Lemos MF, Lemos-Neto SV, Barrucand L, Verçosa N, Tibirica E. Preoperative education reduces preoperative anxiety in cancer patients undergoing surgery: Usefulness of the self-reported Beck anxiety inventory. Braz J Anesthesiol (English Edition) 2019 Jan;69(1):1-6. [CrossRef]
    8. Glenn J, Wolfe K, Borgmann K. The evaluation of video teaching on preoperative anxiety in the outpatient pediatric surgical patient. J Perianesth Nurs 2020 Aug;35(4):e3. [CrossRef]
    9. Southerland WA, Beight LJ, Shapiro FE, Urman RD. Decision aids in anesthesia: Do they help? Curr Opin Anaesthesiol 2020 Apr;33(2):185-191. [CrossRef] [Medline]
    10. Lee A, Chui PT, Gin T. Educating patients about anesthesia: a systematic review of randomized controlled trials of media-based interventions. Anesth Analg 2003 May;96(5):1424-1431. [CrossRef]
    11. De Francisci G, Papasidero AE, Spinazzola G, Galante D, Caruselli M, Pedrotti D, et al. Update on complications in pediatric anesthesia. Pediatr Rep 2013 Feb;5(1):e2 [FREE Full text] [CrossRef] [Medline]
    12. Chang CM, Li Y, Or LL, Pikkarainen M, Pölkki T, Wang W, et al. A qualitative study exploring Singaporean parents' perceptions on preparing their child for surgery. J Pediatr Nurs 2020 Sep;54:e69-e77. [CrossRef] [Medline]
    13. Berg K. Telehealth in anesthesia, an update. In: Goudra BG, Singh PM, Green MS, editors. Anaesthesia for Uncommon and Emerging Procedures. Cham: Springer; Mar 2021:73-80.
    14. Afable MK, Gupte G, Simon SR, Shanahan J, Vimalananda V, Kim EJ, et al. Innovative use of electronic consultations in preoperative anesthesiology evaluation at VA medical centers in New England. Health Aff (Millwood) 2018 Feb;37(2):275-282. [CrossRef] [Medline]
    15. Mullen-Fortino M, Rising KL, Duckworth J, Gwynn V, Sites FD, Hollander JE. Presurgical assessment using telemedicine technology: impact on efficiency, effectiveness, and patient experience of care. Telemed J E Health 2019 Feb;25(2):137-142. [CrossRef] [Medline]
    16. Wong DT, Kamming D, Salenieks ME, Go K, Kohm C, Chung F. Preadmission anesthesia consultation using telemedicine technology: a pilot study. Anesthesiology 2004 Jun;100(6):1605-1607 [FREE Full text] [CrossRef] [Medline]
    17. Kamdar NV, Huverserian A, Jalilian L, Thi W, Duval V, Beck L, et al. Development, implementation, and evaluation of a telemedicine preoperative evaluation initiative at a major academic medical center. Anesth Analg 2020 Dec;131(6):1647-1656 [FREE Full text] [CrossRef] [Medline]
    18. Mihalj M, Carrel T, Gregoric ID, Andereggen L, Zinn PO, Doll D, et al. Telemedicine for preoperative assessment during a COVID-19 pandemic: recommendations for clinical care. Best Pract Res Clin Anaesthesiol 2020 Jun;34(2):345-351 [FREE Full text] [CrossRef] [Medline]
    19. Garg R, Hoda W. Emergence of telehealth for anesthesiologists in COVID-19-boon for all!. J Anaesthesiol Clin Pharmacol 2020 Jul;36(3):417-419 [FREE Full text] [CrossRef] [Medline]
    20. How many people have Smartphones worldwide (Sept 2021)? www.bankmycell.com.   URL: https://www.bankmycell.com/blog/how-many-phones-are-in-the-world [accessed 2021-09-20]
    21. Georgiou M. Developing a healthcare app in 2022: What do patients really want? Imaginovation Insider. 2022.   URL: https://imaginovation.net/blog/developing-a-mobile-health-app-what-patients-really-want/ [accessed 2022-04-28]
    22. Simpao AF, Lingappan AM, Ahumada LM, Rehman MA, Gálvez JA. Perioperative smartphone apps and devices for patient-centered care. J Med Syst 2015 Sep;39(9):102. [CrossRef] [Medline]
    23. Machado RCC, Turrini RNT, Sousa CS. Mobile applications in surgical patient health education: an integrative review. Rev Esc Enferm USP 2020 Mar;54:e03555. [CrossRef]
    24. Cho S, Lee E. Effects of the smartphone application "Safe Patients" on knowledge of patient safety issues among surgical patients. Comput Inform Nurs 2017 Dec;35(12):639-646. [CrossRef] [Medline]
    25. Foley NM, O'Connell EP, Lehane EA, Livingstone V, Maher B, Kaimkhani S, et al. PATI: Patient accessed tailored information: a pilot study to evaluate the effect on preoperative breast cancer patients of information delivered via a mobile application. Breast 2016 Dec;30:54-58. [CrossRef] [Medline]
    26. Mundi MS, Lorentz PA, Grothe K, Kellogg TA, Collazo-Clavell ML. Feasibility of smartphone-based education modules and ecological momentary assessment/intervention in pre-bariatric surgery patients. Obes Surg 2015 Oct;25(10):1875-1881. [CrossRef] [Medline]
    27. Rogers G. Using telemedicine for pediatric preanesthesia evaluation: a pilot project. J Perianesth Nurs 2020 Feb;35(1):3-6. [CrossRef]
    28. Ji L, Zhang X, Fan H, Han M, Yang H, Tang L, et al. drawMD APP-aided preoperative anesthesia education reduce parents anxiety and improve satisfaction. Patient Educ Couns 2016 Feb;99(2):265-270. [CrossRef] [Medline]
    29. English LL, Dunsmuir D, Kumbakumba E, Ansermino JM, Larson CP, Lester R, et al. The PAediatric Risk Assessment (PARA) mobile app to reduce postdischarge child mortality: design, usability, and feasibility for health care workers in Uganda. JMIR Mhealth Uhealth 2016 Feb;4(1):e16 [FREE Full text] [CrossRef] [Medline]
    30. Bhansali R, Armstrong J. Smartphone applications for pediatric anesthesia. Paediatr Anaesth 2012 Apr;22(4):400-404. [CrossRef] [Medline]
    31. Monroe KS, Evans MA, Mukkamala SG, Williamson JL, Jabaley CS, Mariano ER, et al. Moving anesthesiology educational resources to the point of care: experience with a pediatric anesthesia mobile app. Korean J Anesthesiol 2018 Jun;71(3):192-200 [FREE Full text] [CrossRef] [Medline]
    32. Clebone A, Strupp KM, Whitney G, Anderson MR, Hottle J, Fehr J, Pedi Crisis Application Working Group. Development and usability testing of the Society for Pediatric Anesthesia Pedi Crisis mobile application. Anesth Analg 2019;129(6):1635-1644. [CrossRef]
    33. Dunsmuir D, Wu H, Sun T, West NC, Lauder GR, Görges M, et al. A postoperative pain management mobile app (Panda) for children at home after discharge: usability and feasibility. JMIR Perioper Med 2019 Jul;2(2):e12305 [FREE Full text] [CrossRef] [Medline]
    34. Scott AR, Alore EA, Naik AD, Berger DH, Suliburk JW. Mixed-methods analysis of factors impacting use of a postoperative mHealth app. JMIR Mhealth Uhealth 2017 Feb;5(2):e11 [FREE Full text] [CrossRef] [Medline]
    35. Liew MS, Zhang J, See J, Ong YL. Usability challenges for health and wellness mobile apps: mixed-methods study among mHealth experts and consumers. JMIR Mhealth Uhealth 2019 Jan;7(1):e12160 [FREE Full text] [CrossRef] [Medline]
    36. Software and data quality - usability. ISO25000.   URL: https://iso25000.com/index.php/en/iso-25000-standards/iso-25010/61-usability [accessed 2021-09-20]
    37. Nielsen J. Usability inspection methods. 1994 Presented at: CHI94: ACM Conference on Human Factors in Computer; April 24-28, 1994; Boston p. 413-414. [CrossRef]
    38. Putrino D, Zanders H, Hamilton T, Rykman A, Lee P, Edwards DJ. Patient engagement is related to impairment reduction during digital game-based therapy in stroke. Games Health J 2017 Oct;6(5):295-302 [FREE Full text] [CrossRef] [Medline]
    39. Nievas-Soriano BJ, García-Duarte S, Fernández-Alonso AM, Bonillo-Perales A, Parrón-Carreño T. Users evaluation of a Spanish eHealth pediatric website. Comput Methods Programs Biomed 2021 Nov;212:106462 [FREE Full text] [CrossRef] [Medline]
    40. Fredheim H. Why user experience cannot be designed? Smashing Magazine.   URL: https://www.smashingmagazine.com/2011/03/why-user-experience-cannot-be-designed/ [accessed 2022-04-28]
    41. Müller S. Digitala Vetenskapliga Arkivet. 2019 Aug 06.   URL: https://www.diva-portal.org/smash/record.jsf?dswid=6646&pid=diva2%3A1336046 [accessed 2021-09-20]
    42. Hibbs C, Jewett S, Sullivan M. The Art of Lean Software Development: A Practical and Incremental Approach. Sebastopol: O'Reilly Media, Inc; Jan 2009:71-75.
    43. Bosse C, Kelly K. Improving EHR usability using LEAN methodology. Stud Health Technol Inform 2016;225:870. [CrossRef] [Medline]
    44. A brief history of Lean. Lean Enterprise Institute.   URL: https://www.lean.org/explore-lean/a-brief-history-of-lean/ [accessed 2021-09-20]
    45. Klein L. UX for Lean Startups: Faster, Smarter User Experience Research and Design. Sebastopol: O'Reilly Media, Inc; May 2013:63-84.
    46. Merrick L. 15 UX design secrets for building Lean, profitable mobile apps. https://medium.com/. 2016 Sep.   URL: https:/​/medium.​com/​@LoganTjm/​15-ux-design-secrets-for-building-lean-profitable-mobile-apps-3df7c75c2e8d [accessed 2021-09-20]
    47. Zhang D, Adipat B. Challenges, methodologies, and issues in the usability testing of mobile applications. Int J Hum Comput Interact 2005 Jul;18(3):293-308. [CrossRef]
    48. Zeitz K, Marchany R, Tront J. Speed isn't enough: usability and adoption of an optimized alert notification system. IEEE Technol Soc Mag 2016 Mar;35(1):47-55. [CrossRef]
    49. Setiawan IMA, Zhou L, Alfikri Z, Saptono A, Fairman AD, Dicianno BE, et al. An adaptive mobile health system to support self-management for persons with chronic conditions and disabilities: usability and feasibility studies. JMIR Form Res 2019 Apr;3(2):e12982 [FREE Full text] [CrossRef] [Medline]
    50. Schwebel FJ, Larimer ME. Using text message reminders in health care services: a narrative literature review. Internet Interv 2018 Sep;13:82-104. [CrossRef]
    51. Okoshi T, Tsubouchi K, Tokuda H. Real-world product deployment of adaptive push notification scheduling on smartphones. In: KDD '19: Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.: Association for Computing Machinery; 2019 Presented at: 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining; August 4-8, 2019; Anchorage. [CrossRef]
    52. Mardhia M, Anggraini E. Implement a Lean UX model: integrating students' academic monitoring through a mobile app. 2019 Presented at: 2019 International Conference of Advanced Informatics: Concepts, Theory and Applications (ICAICTA); September 20-21, 2019; Yogyakarta p. 1-5. [CrossRef]
    53. Pathak LE, Aguilera A, Williams JJ, Lyles CR, Hernandez-Ramos R, Miramontes J, et al. Developing messaging content for a physical activity smartphone app tailored to low-income patients: user-centered design and crowdsourcing approach. JMIR Mhealth Uhealth 2021 May;9(5):e21177 [FREE Full text] [CrossRef] [Medline]
    54. Lopes F, Rodrigues M, Silva AG. User-centered development of a mobile app for biopsychosocial pain assessment in adults: usability, reliability, and validity study. JMIR Mhealth Uhealth 2021 May;9(5):e25316 [FREE Full text] [CrossRef] [Medline]
    55. Vaghefi I, Tulu B. The continued use of mobile health apps: insights from a longitudinal study. JMIR Mhealth Uhealth 2019 Aug;7(8):e12983 [FREE Full text] [CrossRef] [Medline]
    56. Bidargaddi N, Almirall D, Murphy S, Nahum-Shani I, Kovalcik M, Pituch T, et al. To prompt or not to prompt? A microrandomized trial of time-varying push notifications to increase proximal engagement with a mobile health app. JMIR Mhealth Uhealth 2018 Nov;6(11):e10123 [FREE Full text] [CrossRef] [Medline]

    ‎
    mHealth: mobile health
    MVP: Minimum Viable Product
    UI: user interface
    UX: user experience

    Edited by A Mavragani; submitted 07.10.21; peer-reviewed by D Dunsmuir, B Nievas Soriano; comments to author 15.11.21; revised version received 23.02.22; accepted 08.04.22; published 05.05.22

    Copyright

    ©Yaron Connelly, Roni Lotan, Yitzhak Brzezinski Sinai, Dan Rolls, Amir Beker, Eilone Abensour, Orit Neudorfer, Daniel Stocki. Originally published in JMIR Formative Research (https://formative.jmir.org), 05.05.2022.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.