In June 2018, we searched web-based databases (PubMed and Embase) to identify articles related to the evaluation of the usability of mobile apps for health and wellness. Studies from this literature review were assessed to create a list of domains for rating mHealth apps relevant to diverse, low-income populations. Search criteria included “smartphone application underserved community,” “usability of commercially available applications for diverse patients,” “mHealth apps usability testing underserved,” “mheatlh app underserved,” “mobile health phone applications for diverse populations,” on PubMed and “mobile application'/exp OR 'mobile application' OR 'mobile phone'/exp OR 'mobile phone,” “‘mHealth under-served' OR (('mhealth'/exp OR mhealth) AND unders-erved)” on Embase.

An extensive literature review indicated important domains of mHealth apps for our target population, including the following: language, literacy, graphics, multimedia, usability, patient-centeredness, data entry mode, data exportability, cost, evidence based content, platform, extent to which the platform was up to date, connectivity, Americans with Disabilities Act (ADA) accessibility, privacy, social network, cultural sensitivity (incorporating diversity in language, graphics, and data), messaging or reminder capability, and benchmarking (comparing a user’s performance to the performance of others on the app). These domains were grouped into larger categories, including the following: Usability (graphics, multimedia, usability, and ADA accessibility), Population focus (language, literacy, patient-centeredness, cost, social network, cultural sensitivity, benchmarking, and messaging or reminder capability), Technology (data entry mode, data exportability, platform, connectivity requirement, and privacy), and Clinical Impact (evidence-based content and extent to which the platform was up to date).

Each domain was then weighted in terms of importance to the target population by independent coders (RGM, KGB, and JM). These coders are all primary care physicians at an urban safety net hospital (in General Internal Medicine, Family Medicine, and Pediatrics, each with 10-25 years of experience at this institution) and care for overlapping members of these communities by age. They were asked to rate each domain on a scale of 1 to 5 (1=“not important to be included” and 5=“must be included”). Then, a reviewer (SS) adjudicated these weights and averaged the “weight” of each domain respectively. Finally, these weights were directionally confirmed with the hospital-based patient and family advocacy committee.

To simulate how patients would access recommendations for mHealth apps, we used the most common search engine, Google, to search “Top Ten Mobile Health Apps for Diabetes” and “Top Ten Mobile Health Apps for Smoking Cessation.” The first search result referred to articles in a popular health website “Healthline,” which provided the top 10 mobile health apps related to each topic [21,22]. The website rated these apps on the basis of quality, reliability, reviews, and community nominations. As no such list existed for medication adherence focused apps, these apps were chosen from the list recommended by a previous study [23]. We chose apps in these 3 areas because they represent different aspects of medical care delivery: preventative care (smoking cessation), chronic disease management (diabetes management), and general therapeutic intervention (medication adherence).

Ten apps in smoking cessation, diabetes management, and medication adherence were identified and rated by a coder (SS) in each domain (language, literacy, graphics, multimedia, usability, patient-centered, data entry mode, data exportability, cost, evidence-based content, platform, extent the platform was up to date, connectivity, ADA accessibility, privacy, social network, cultural sensitivity, messaging or reminder capability, and benchmarking) with a score of 0 to 3. A score of 0 was assigned if the specific domain was not applicable, and a score of 3 was assigned for the highest applicability (eg, language—available in 3 or more languages; Table 1). Domains were defined on the basis of current research in each respective domain. For example, for “Social Network,” higher points were assigned for “competition” than for “social support” because previous studies have shown that social comparison was more important for physical activity [24]. As another example, the definition for “Usability” was based on the design of the app and the ability to navigate it easily. We specifically did not use the industry standard assessment (ie, System Usability Scale) owing to concerns about the applicability for diverse, low–socioeconomic status communities. Lastly, we decided whether apps were “evidence based” if they cited specific research or if the theory underlying their apps had existing evidence (ie, gasification and social networks).

The domain score was multiplied by the domain weight (of importance to the target population) to produce a weighted final score for that domain for the app in question. The weighted scores across all domains were added to assign each app a final overall score of usefulness and applicability for diverse, low-income populations.

In July 2019, after the mHealth apps were chosen and scored, we presented the top 3 apps in each health category to a patient advisory board at a safety net hospital in Boston, Massachusetts. The group consisted of 4 participants on the day of presentation. They were Caucasian women between 35 and 75 years of age from the catchment area of the safety net institution. They were asked to give their overall thoughts on our research idea and the functionality of mHealth apps they considered important to manage their health on a daily basis.

Domains of usability and functionality for mHealth apps were assigned weighted scores depending on the importance the clinical experts gave to those categories for diverse, low-income communities. These were directionally confirmed with a patient advisory board. Domains of greatest importance were identified through this process (Table 2). Literacy was found to be the highest scoring domain by clinical experts, with all raters assigning a score of 5 out of 5. Language, usability, cost, evidence based content, and cultural sensitivity had an average weight of 4.5 out of 5. Graphics was rated 4 out of 5. Social media connectivity and timing of the app’s latest update were assigned a score of 3.5 out of 5. Multimedia, data exportability, patient centeredness, benchmarking, and messages or reminders were all rated an average of 3 out of 5. ADA accessibility was rated 2.5 out of 5. Data entry and privacy domains were weighted 2 out of 5, being the least important for the specific patient population. Raters were within 1 point of each other on all domains, with 8 out of 19 domain weightings being identical. The higher-level categories into which domains can be grouped also differentiated in terms of importance, with clinical impact and population focus being most highly weighted: Clinical Impact (average weight across component domains 4), Population Focus (average weight across component domains 3.88), Usability (average weight across component domains 3.5), and Technology (average weight across component domains 2.8).

When using the weighted scoring methodology to rate mHealth apps across smoking cessation, diabetes management, and medication adherence, the framework was able to sufficiently distinguish among 10 apps within each category (Table 3). Weighted scores ranged from 108.5 to 153 for smoking cessation apps, 119 to 147 for diabetes management apps, and 113 to 137.5 for medication adherence apps (Table 4). The top 2 scoring apps in each category were QuitNow! (156/201), Smoke Free-Quit Smoking Now (149.5/201), Glucosio (147/201), MyNetDiary (146/201), Medisafe (137.5/201), and MyMeds (126/201) (Multimedia Appendix 1).

mHealth apps have the potential to improve individuals’ management of chronic diseases and to extend the reach of the health care provider visit. Considering the increasing predominance of smart device use to access the internet among diverse, low-income communities, mHealth apps hold enormous potential for impact on health and could help bridge health equity divides. To date, however, studies have shown that more research is needed to rate the practical functionality of these mobile apps, specifically for this target population [25,26]. By developing a framework to rate the usefulness of mHealth applications for low income, diverse, patient populations, and showing its effectiveness to differentiate between apps, we aimed to provide a way to curate mHealth apps for better access and engagement among diverse populations.

Existing frameworks such as the Mobile App Rating Scale (MARS) include ratings for domains grouped into categories with broad population applicability, including engagement, functionality, aesthetics, subjective quality, and information. As identified by researchers who created the MARS system, this grading system is agnostic to the needs of specific populations and may not be indicative of the usefulness of apps for specific groups. As a result, it is difficult to apply the grading criteria of MARS to diverse, low-income patient populations [26]. The evaluation mechanism described in this study incorporates domains that are applicable for general consumption as contained in MARS and highlighted by Anderson et al [27] and others as important to engagement (“evidence based content,” “privacy,” “up to date,” “patient centered,” “benchmarking,” and “social network”), but expands beyond MARS specifically for diverse, low-income patient populations [26-29].

In contrast to the more generalized domains, our research led us to evaluate a greater number of criteria with population-specific focus. The domains expanded upon those in the MARS criteria to include: language, literacy, cultural sensitivity, data entry, data exportability, multimedia, ADA accessibility, cost, platform, and message or reminder function. As identified by prior studies, multimedia availability, such as videos, helped increase mobile app engagement, particularly for individuals with low literacy levels [16,17,19,20,30,31]. Messages or a reminder function helped app users stay motivated in their plan, manage medications, and organize personal health information [29,32-36]. Cost was a domain included in our scoring framework because some past studies showed increased engagement if patients paid for the app. However, there is a question as to whether this would be an undue barrier for patients with low income [27]. Importantly and consistent with our hypothesis, language, literacy, and cultural sensitivity were found to be some of the most important qualities for an app to have to be relevant and useful for diverse, low-income populations, according to both our clinician and patient reviewers [14,16,18-20,29,37-40].

The highest rated mHealth apps in our sample shared notable, similar qualities, regardless of what chronic illness or health state they were developed to manage. Most apps placed importance on offering multiple languages, being written for lower literacy levels, and incorporating graphics; these scored high on usability. However, apps such as MyNetDiary, for diabetes management, which were more difficult to navigate, did offer additional resources to learn about navigating through the app, such as an instructional video. Additionally, privacy, evidence-based content, data entry, data exportability, and patient-centered content were present in most (but not all) of the chosen apps. Benchmarking (comparing a user’s performance to the performance of others on the app), social support networks, and multimedia use scores were not robust in any of the chosen apps. ADA accessibility and cultural sensitivity (incorporating diversity in language, graphics, and data) were also lacking. Based on these results, the apps rated most highly in accordance with our framework conveyed information in a manner that a more diverse user base would be able to effectively engage with. However, as a whole, the mobile apps did not focus any specific attention to engaging a diverse population. For example, although many apps were inclusive by providing multiple language options, they lacked content that was tailored to a culturally diverse population. Overall, although these apps were the best of what is currently on the market, they are far from being ideal for our target population.

The patient advisory board was both a versatile and key feature of our research effort providing both a patient perspective on the mHealth apps and on the domains included in the framework we developed. Thus, we gained insight to patients’ perspectives and were able to compare their priorities in engaging with these apps to those of health care providers in the same system. Domains including language, literacy, evidence-based content, cultural sensitivity, and providing up-to-date information were ranked highly by both groups. However, some domains such as graphics, social network, and benchmarking were not as important to patients as they were thought to be by providers. In contrast to our qualitative findings, prior studies have shown that apps with simple interfaces that favor graphics over text tend to be more usable with lower literacy populations [19,37]. Additionally, social media functionality has also been found to reduce barriers to sharing information and learning from others [32,41,42]. Therefore, while other studies have shown these domains to be important to engagement, our patients did not rank them highly in terms of importance to usability. These differences could be explained by the small sample size and demographics of our patient advisory group.

There were several limitations to this work, which can be used as learning points for future research. First, in selecting which mHealth apps to evaluate we used a common website which had its own subjective way of choosing which apps were the “best.” However, this approach to finding the “top 10” apps on a website likely replicates the way in which patients would search for and find mHealth apps to download and use. The mHealth app market is in a constant state of flux and growth, and with it, the “top 10.” Second, some of the mHealth apps initially identified for evaluation were not available for download. Churn in the mHealth market may impact engagement with some mHealth apps. However, this better reflected which mHealth apps would be accessible to our patients. Third, as this was a small study with no independent funding, we only had one reviewer grade the mHealth apps across each domain. Though individual user subjectivity is inherently part of the process of evaluating mHealth apps, this process may result in some subjectivity to our analysis of the individual apps. Furthermore, our patient advisory board was not necessarily representative of the diverse patient population of the hospital. While they themselves were patients at the safety net institution, they were all White women aged 35-75 years. As a result, their opinions may not be generalizable and could be subject to conscious and unconscious biases. While this limitation is significant, we also found it important to have some patient perspective and validation rather than none at all. Finally, it is important to acknowledge our mHealth rubric and research was conducted before the COVID-19 pandemic, and much of mHealth has evolved since then. Our reliance on technology is more than ever, and we may be more dependent on mobile app technologies. However, the nature of this research should withstand the evolution of the mHealth app market, and even seek to improve upon it as it can be applied to make this growing field even more accessible for a diverse user base.

We adapted a framework for evaluating mHealth apps for diverse, low-income populations from the MARS model and created modified domains to characterize critical features as identified by patients and clinical experts who care for these patient populations. Although these app domains are rated separately, it is important to remember that often these different domains work synergistically. Some domains may increase user engagement and retention while others are focused on increasing access, inclusivity, or privacy. Multiple languages remain a worthy goal, but we contend mHealth app developers should also consider incorporating culturally sensitive and specific information such as recipes, videos, and motivational tools.

This novel framework is invaluable as it can be applied to evaluate individual mHealth apps in the context of therapeutic interventions, as well as by app developers to identify those domains important for engagement of diverse, low-income populations. However, right now our app ranks relative performance. To establish specific scoring thresholds and determine cut-off ratings for apps we need to apply this rating framework to a large number of apps. The plan would be to accomplish this during a follow-up study. For now, we believe distinguishing relative performance is important so developer can use an iterative approach to design apps. They can use scores as a performance indicator as they improve or refine existing apps. If developers are able to report app performance, it may be a way to filter apps and create inter-app competition to improve performance.

In the future, the study team plans to incorporate the selected apps in smoking cessation, diabetes management, and medication adherence into our clinical processes as “prescriptions” for patients. We will be able to track these “prescriptions,” assess patient engagement, and determine impact on health outcomes. In addition, we hope to continue to use our framework to find, scale, and spread mHealth apps that will be most useful for our target population in other areas of health including exercise and mental health. By using this novel framework to identify mHealth apps for recommendation and for future mHealth app development, health care providers, policy makers and developers alike may be able to better incorporate this burgeoning technology into both clinical practice and patient homes for greater impact on health outcomes for all patients, further narrowing digital and health outcome divides.