This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.
Some latest estimates show that approximately 95% of Americans own a smartphone with numerous functions such as SMS text messaging, the ability to take high-resolution pictures, and mobile software apps. Mobile health apps focusing on vaccination and immunization have proliferated in the digital health information technology market. Mobile health apps have the potential to positively affect vaccination coverage. However, their general functionality, user and disease coverage, and exchange of information have not been comprehensively studied or evaluated computationally.
The primary aim of this study is to develop a computational method to explore the descriptive, usability, information exchange, and privacy features of vaccination apps, which can inform vaccination app design. Furthermore, we sought to identify potential limitations and drawbacks in the apps’ design, readability, and information exchange abilities.
A comprehensive codebook was developed to conduct a content analysis on vaccination apps’ descriptive, usability, information exchange, and privacy features. The search and selection process for vaccination-related apps was conducted from March to May 2019. We identified a total of 211 apps across both platforms, with iOS and Android representing 62.1% (131/211) and 37.9% (80/211) of the apps, respectively. Of the 211 apps, 119 (56.4%) were included in the final study analysis, with 42 features evaluated according to the developed codebook. The apps selected were a mix of apps used in the United States and internationally. Principal component analysis was used to reduce the dimensionality of the data. Furthermore, cluster analysis was used with unsupervised machine learning to determine patterns within the data to group the apps based on preselected features.
The results indicated that readability and information exchange were highly correlated features based on principal component analysis. Of the 119 apps, 53 (44.5%) were iOS apps, 55 (46.2%) were for the Android operating system, and 11 (9.2%) could be found on both platforms. Cluster 1 of the k-means analysis contained 22.7% (27/119) of the apps; these were shown to have the highest percentage of features represented among the selected features.
We conclude that our computational method was able to identify important features of vaccination apps correlating with end user experience and categorize those apps through cluster analysis. Collaborating with clinical health providers and public health officials during design and development can improve the overall functionality of the apps.
IT has revolutionized all aspects of the world, including our health care system. IT has enhanced the overall efficiency and accessibility of patient care [
A significant concern that is often communicated by mHealth app users is data privacy. In the United States, the Health Insurance Portability and Accountability Act (HIPAA) ensures that health care entities provide adequate measures to protect patient data. Many consumer-based apps that track and monitor health data are not HIPAA compliant. Health care stakeholders in the United States recommend that mobile apps designed for health care be HIPAA compliant [
Vaccine hesitancy—the delay or refusal to be vaccinated despite available vaccination services—is a complex phenomenon that involves emotional, cultural, social, spiritual, and political factors [
Results from a recent systematic review reported a lack of evidence supporting the use of vaccination apps geared toward children, as shown through vaccination uptake, knowledge, and decision-making [
The primary aim of our study was to develop a computational method to explore the descriptive, usability, information exchange, and privacy features of various vaccination apps, which can inform vaccination app design. We also aimed to assess these apps using a content analysis approach and identify potential flaws in app functionality. This study analyzed these data according to their respective operating platforms and collectively. The content analysis approach used was adapted from previous studies [
For this study, vaccination-related apps were operationalized as apps that allowed tracking, scheduling, and general dissemination of vaccination information [
The search and selection process for the apps was conducted from March to May 2019. We identified a total of 211 apps across both platforms, with iOS and Android representing 62.1% (131/211) and 37.9% (80/211) of the apps, respectively. As iOS apps represented more than half of the apps collected, we chose a random sample of apps from each platform to generate a sample of 132 apps (62.6% of the total sample). We oversampled the Android apps that were collected to provide a balance of Android apps that would be represented in the feature space. Moreover, oversampling is a common technique used when there is an underrepresentation of 1 class [
To comprehensively characterize the features of the vaccination apps retrieved, we systematically developed an inclusive codebook with 4 categories (
Summary of the codebook features with descriptions.
Category | Features | Description |
Descriptive | App name, developer; platform (iOS, Android, or both), category in the app store (medical, health and fitness, travel, or local), size in MB, ranking in its respective category if applicable, overall star rating if applicable, age rating if applicable, and cost (completely free, free to download with in-app purchase, or paid) | These descriptive characteristics gave an overview of the immunization app and such information could generally be found on the app store’s description page without the need to download or install the app [ |
Users and diseases | Target users and target diseases; for target users, we analyzed whether the app provided information on a specific user group (eg, children, parents, women, physicians, and age group); target diseases pertained to the description of a specific disease or general information concerning vaccinations and scheduling | In this category, we evaluated the targeted users and diseases of the apps. Some apps could be used by multiple, potentially overlapping groups of adult users, such as travelers and women, for which we created a specific group with binary response only. The targeted users included the following: minors, parents, travelers, women, people of all ages, and health care providers and staff. For targeted diseases, 0 was associated with no user-defined diseases, and 1 was for specific diseases such as seasonal influenza and measles-mumps-rubella [ |
Information exchange features | Account requirement for full app functionality, information presented about specific types of vaccines, educational information about vaccination and immunization in general, immunization tracking, customization of schedule, identification of nearby vaccination clinics, reminders of upcoming vaccination events, and personalized vaccination recommendations | In this category, we further explored and quantified vaccination-related core features of the apps. |
Privacy and readability | Health Insurance Portability and Accountability Act–compliance feature; presence of in-app privacy statement; presence of privacy statement in the app store; presence of multilingual (at least 2 languages) privacy statement; and the average length of the privacy content (in English) using the following 7 readability measures: Simple Measure of Gobbledygook, Flesch Reading Ease score, Gunning Fog Index, Flesch-Kincaid Grade, Coleman-Liau Index, Automated Readability Index, and Linsear Write Formula [ |
Here, we considered an important element in mobile health–related research and app development, which is privacy-related features to address privacy concerns around sensitive and private vaccine health information. These features would provide information on how user-generated data would be collected, stored, shared, and transmitted on the web and offline [ |
We analyzed and evaluated the content of the apps using the aforementioned codebook through a combination of content analysis [
PCA is an important preprocessing step. Prior studies have used PCA to show children’s interactions with education apps [
After identifying the proportion of variance, we determined the value of each feature contained within each PC. We used the loading values of each PC to determine this information. These values represent the correlations between the PC and the original used features. A correlation that is close to 1 or −1 indicates how important the feature is to the component. We extracted the top 5 features for each PC with the highest variance. Using these values, we reduced the number of features to represent the apps from 42 to 10. The key idea of PCA is to reduce the number of variables in the data set but preserve as much information or representation of that information in the new data set as possible [
Cluster analysis is used to define classes within a set of data. Clustering can be conducted using supervised and unsupervised methods. We used the unsupervised k-means clustering method to group our apps. This clustering algorithm is well documented, with successfully separating data for analysis; moreover, it has performed similar to or better than other clustering approaches [
The data used in this study satisfied two research activities that did not require IRB approval, Quality Assurance and Improvement. IRB approval is not required if the study involves the practice of program evaluation, self-assessment of programs or business practices, and other quality improvement projects where methods rather than humans are the subject of the study. It also satisfies the conditions of a pilot study where the activities are intended to refine data collection procedures – time to participate, testing survey questions, etc. where any data collected are only used to plan and/or improve a future research study.
Of the 42 features, 12 (28%) were used for the descriptive app category. Of these 12 features, 9 (75%) were used for the (targeted) users and diseases category, and 8 (67%) were used for the information exchange category. Finally, 31% (13/42) of features represented the privacy and readability category. Of the 119 apps, 53 (44.5%) were iOS apps, 55 (46.2%) were for the Android operating system, and 11 (9.2%) could be found on both platforms. The Flesch-Kincaid Grade readability score (readability tests designed to indicate how difficult the content is to understand) had an average of 6.4 (SD 6.6) for both platforms combined (
Select app features characteristics (N=119).
App features | iOS (n=53) | Android (n=55) | Both (n=11) | Total | |
Number of ratings, mean (SD) | 13.53 (62.34) | 1772.8 (8136.84) | 61.91 (79.07) | 831.11 (5600.44) | |
Size in MB, mean (SD) | 37.54 (41.2) | 11.48 (17) | 14.4 (19.47) | 23.36 (32.66) | |
Star rating, mean (SD) | 0.83 (1.59) | 2.63 (2.08) | 2.71 (2.07) | 1.84 (2.06) | |
Age rating, mean (SD) | 9.34 (5.4) | 2.62 (4.99) | —a | 5.37 (6.15) | |
Length of privacy policy (words)b, mean (SD) | 850.38 (1483.42) | 790.42(1227.05) | 874.64 (1206.42) | 824.91 (1329.78) | |
Flesch-Kincaid Grade, mean (SD) | 6.13 (6.89) | 6.01 (6.38) | 9.63 (6.28) | 6.4 (6.6) | |
|
|||||
|
Yes | 7 (13) | 2 (4) | 0 (0) | 9 (8) |
|
No | 46 (87) | 53 (96) | 13 (100) | 110 (92) |
aNot available.
bSeveral apps identified contained privacy policy statements written in a different language
cHIPAA: Health Insurance Portability and Accountability Act.
Results from the dimensionality reduction of the feature space showed that PC1 explained approximately 24.7% of the data and PC2 explained 8.3% of the data (
Principal component analysis showing that 24.7% of the data variance is explained by principal component 1 and 8.3% of the data variance is explained by principal component 2.
PC1 (features related to readability)
Automated Readability Index
Simple Measure of Gobbledygook formula
Flesch-Kincaid Grade
Reading text page success
Linsear Write Formula
PC2 (features related to user customization)
Reminder for vaccination
Customized schedule
Vaccination tracking
Personalized recommendations
Targeted at parents
The top 5 features from PC1 and PC2 were used to create a cluster graph that represented the optimal number of clusters for the new feature space (
Total within-cluster sum of squares and average silhouette width. The optimal number of clusters is 5 (left) for the total within-cluster sum of squares measure and 6 (right) for the average silhouette width measure.
K-means clusters with selected new features represented (N=119).
Features | App cluster, n (%) | |||||
|
1 (n=27) | 2 (n=32) | 3 (n=29) | 4 (n=19) | 5 (n=12) | |
|
||||||
|
Android | 12 (44) | 15 (47) | 13 (45) | 9 (47) | 6 (50) |
|
iOS | 11 (41) | 13 (41) | 14 (48) | 9 (47) | 6 (50) |
|
Both | 4 (15) | 4 (12) | 2 (7) | 1 (6) | 0 (0) |
|
||||||
|
Yes | 16 (59) | 4 (12) | 8 (28) | 6 (32) | 5 (42) |
|
No | 11 (41) | 28 (88) | 21 (72) | 13 (68) | 7 (58) |
|
||||||
|
Yes | 25 (93) | 4 (12) | 0 (0) | 18 (95) | 10 (83) |
|
No | 2 (7) | 28 (88) | 29 (100) | 1 (5) | 2 (17) |
|
||||||
|
Yes | 27 (100) | 32 (100) | 0 (0) | 0 (0) | 0 (0) |
|
No | 0 (0) | 0 (0) | 29 (100) | 19 (100) | 12 (100) |
In this study, we developed a codebook to conduct a content analysis of vaccination apps and explored the use of computational approaches to identify the feature importance of vaccination apps, reduce the dimensionality of our feature space, and categorize vaccination apps using k-means clustering in an unsupervised case approach. When examining the feature importance of the 119 vaccination apps and 42 features, we found that the most important features could be categorized and explained through PC1 and PC2. For PC1, the top features found in this component were predominately associated with the
mHealth technology has the potential to improve the efficiency and convenience of health care information exchange. Our findings can be categorized into two major themes: (1) features that limit the functionality of apps and (2) features that impede the overall user experience. Although most apps are moderately received by their users, based on the app rating feature, there were salient weaknesses identified through the use of PCA. This further suggests that the limitations within the reviewed vaccination apps must be addressed. On the basis of the k-means cluster analysis and the selected features, only 1 cluster of vaccination apps did not provide evidence for user vaccine schedule customization. Functionality improvements to mHealth apps could allow for a connection between patients and medical professionals to provide timely care. Systematic incorporation of information exchange features and improving policy readability would result in notable enhancements to future apps, as well as those that are currently on the market and fail to incorporate these features.
We concluded that most vaccination apps were not developed alongside health professionals. There is no standard for expert involvement in app development for any sector, and integrating medical experts in the development of mHealth apps is important, considering the increased use of mHealth technology in health care spaces [
Health literacy involves the ability of individuals to find, understand, and use services to educate themselves to make health-related decisions [
Transferring vaccination records from paper to digital requires strict data standards and interoperability to ensure security [
The validity of our research is upheld through a diligent acquisition and analysis of the 119 Android and iOS apps. We used 2 computational approaches to reduce the feature space and cluster our apps. Furthermore, PCA allows for the identification of specific features correlated to the larger PCs. Following the use of PCA and k-means clustering, our data provide a visual representation that is palatable for diverse audiences. The method used in our work has implications for other domain areas to examine the most important features when considering app design.
Despite a rigorous procurement and analysis of the 119 apps, our research contains several limitations. First, the apps that did not meet the 12-month time frame of representation on their respective platforms were removed from the analysis [
Second, this study was started in 2019, before the COVID-19 pandemic. Vaccination hesitancy along with misinformation has exacerbated vaccination uptake concerns. The landscape related to vaccination campaigns and the use of vaccination apps has changed significantly since this study started. Therefore, changes in apps that address misinformation, vaccine hesitancy, and telehealth services should be considered in future studies. Third, we used 2 exploratory machine learning approaches that can be affected by the data set size, number of features, and number of clusters. Instead of k-means clustering, the use of a hierarchical clustering method can account for grouping concerns during the cluster assignment step. Future work may incorporate other computational techniques to analyze these nuanced differences.
Finally, the researchers conducting this study are a US-based team; therefore, this research is intended to facilitate future app development. This research is also intended to supplement the further improvement of vaccination apps currently used in the United States. Not all the 119 apps featured in our research are based in the United States; this adds to the limitations of the research as it may complicate health recommendations that adhere to government and regional guidelines. Per the variation in countries where the apps are based, HIPAA compliance may not apply to other nations, and this may additionally complicate comparisons. Despite some apps being based in other countries, many internationally focused apps have followed or referenced the Centers for Disease Control and Prevention recommendations for vaccination schedules.
The use of vaccines as a tool in personal and public health remains a cornerstone of disease prevention. Despite the advancement of vaccine technology and the promotion of vaccines as safe and effective, vaccine hesitancy has led to the resurgence of preventable childhood diseases. This resurgence threatens the effectiveness of vaccines as a public health tool. Technology, particularly mHealth apps, enables the intersection of public health and IT to potentially manifest positive vaccine health behaviors in individuals. Understanding the descriptive, usability, information exchange, and privacy features of these 119 mHealth apps has the potential to provide researchers and health care professionals information concerning features that should be considered when designing vaccination apps as a public health instrument.
There is conflicting literature on the overall effectiveness of mHealth apps to assist with improving vaccination coverage; however, our research yields recommendations for mHealth vaccination apps developed in the future. One recommendation is to incorporate a transdisciplinary research approach to mHealth app development, in which medical professionals, app developers, public health experts, and users can collaborate throughout the app development process. This ensures engagement from multiple stakeholders and reliable information exchange between agencies and users. As noted in the previous section, although our study was conducted before the COVID-19 pandemic, our findings could prove relevant for the ongoing monitoring of COVID-19 metrics, vaccination documentation, and beyond. One such example for mHealth apps is contact tracing for COVID-19 or serving as a liaison for information exchange between experts and users. A recent study described the most frequently installed features of contact-tracking apps as alert systems and government accountability [
We conclude that our computational method was able to identify important features of vaccination apps correlating with end user experience and categorize those apps through cluster analysis (
Completed k-means analysis using 5 clusters.
List of names for the apps that are represented in each cluster using the k-means clustering method.
electronic health record
Health Insurance Portability and Accountability Act
mobile health
principal component
principal component analysis
The authors would like to acknowledge the Office of Undergraduate Research online pilot, Paper Chase project, at the University of North Carolina Charlotte. The authors would also like to acknowledge Drs Jessamyn Bowling and Alicia Dahl, and Ms Lisa Krinner for supporting the students during this project.
GSJ designed and conducted the computational experiments and analyzed the results of the study. GSJ, DN, EP, RS, ML, and RA assisted with the writing and editing of every section of the manuscript. PA completed the descriptive statistics used in the study. QX assisted with the development of the codebook. SC assisted with the development of the codebook and provided overall direction for the project codebook implementation.
None declared.