Published on in Vol 6 , No 1 (2022) :January

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/32273, first published .
A Smartphone App to Increase Immunizations in the Pediatric Solid Organ Transplant Population: Development and Initial Usability Study

A Smartphone App to Increase Immunizations in the Pediatric Solid Organ Transplant Population: Development and Initial Usability Study

A Smartphone App to Increase Immunizations in the Pediatric Solid Organ Transplant Population: Development and Initial Usability Study

Original Paper

1Adult and Child Consortium for Health Outcomes Research and Delivery Science, Aurora, CO, United States

2University of Colorado School of Medicine, Aurora, CO, United States

3Children's Hospital Colorado, Aurora, CO, United States

4Colorado School of Public Health, Aurora, CO, United States

5Bruyere and Ottawa Hospital Research Institutes, The Ottawa Hospital, University of Ottawa, Ottawa, ON, Canada

6CANImmunize Inc, Ottawa, ON, Canada

Corresponding Author:

Amy G Feldman, MSc, MD

Children's Hospital Colorado

13123 E 16th Ave

Aurora, CO, 80045

United States

Phone: 1 720 777 5354

Fax:1 720 777 7277

Email: amy.feldman@childrenscolorado.org


Background: Vaccine-preventable infections result in significant morbidity, mortality, and costs in pediatric transplant recipients. However, at the time of transplant, less than 20% of children are up-to-date for age-appropriate immunizations that could prevent these diseases. Smartphone apps have the potential to increase immunization rates through their ability to provide vaccine education, send vaccine reminders, and facilitate communication between parents and a multidisciplinary medical group.

Objective: The aim of this study was to describe the development of a smartphone app, Immunize PediatricTransplant, to promote pretransplant immunization and to report on app functionality and usability when applied to the target population.

Methods: We used a mixed methods study design guided by the Mobile Health Agile Development and Evaluation Lifecycle. We first completed a formative research including semistructured interviews with transplant stakeholders (12 primary care physicians, 40 parents or guardians of transplant recipients, 11 transplant nurse coordinators, and 19 transplant subspecialists) to explore the acceptability of an immunization app to be used in the pretransplant period. Based on these findings, CANImmunize Inc developed the Immunize PediatricTransplant app. We next held 2 focus group discussions with 5-6 transplant stakeholders/group (n=11; 5 parents of transplant recipients, 2 primary care physicians, 2 transplant nurse coordinators, and 2 transplant subspecialists) to receive feedback on the app. After the app modifications were made, alpha testing was conducted on the functional prototype. We then implemented beta testing with 12 stakeholders (6 parents of transplant recipients, 2 primary care doctors, 2 transplant nurse coordinators, and 2 transplant subspecialists) to refine the app through an iterative process. Finally, the stakeholders completed the user version of the Mobile Application Rating Scale (uMARS) to assess the functionality and quality of the app.

Results: A new Android- and Apple-compatible app, Immunize PediatricTransplant, was developed to improve immunization delivery in the pretransplant period. The app contains information about vaccine use in the pretransplant period, houses a complete immunization record for each child, includes a communication tool for parents and care providers, and sends automated reminders to parents and care providers when immunizations are due. During usability testing, the stakeholders were able to enter a mock vaccine record containing 16 vaccines in an average of 8.1 minutes (SD 1.8) with 87% accuracy. The stakeholders rated engagement, functionality, aesthetics, and information quality of the app as 4.2/5, 4.5/5, 4.6/5, and 4.8/5, respectively. All participants reported that they would recommend this app to families and care teams with a child awaiting solid organ transplant.

Conclusions: Through a systematic, user-centered, agile, iterative approach, the Immunize PediatricTransplant app was developed to improve immunization delivery in the pretransplant period. The app tested well with end users. Further testing and agile development among patients awaiting transplant are needed to understand real-world acceptability and effectiveness in improving immunization rates in children awaiting transplant.

JMIR Form Res 2022;6(1):e32273

doi:10.2196/32273

Keywords



Due to lifelong immunosuppression, solid organ transplant recipients are at increased risk of life-threatening infections [1-3]. Vaccine-preventable infections (VPIs) occur in up to 15% of pediatric solid organ transplant recipients in the first 5 years posttransplant, a rate of up to 87 times higher than in the general pediatric population [4,5]. These VPIs result in lengthy hospitalizations, morbidity, and mortality. In addition, VPIs can increase the cost of transplantation by US $120,000 [4,5]. To prevent these infections, it is crucial for transplant candidates to receive all age-appropriate vaccines in the pretransplant period. However, less than 20% of pediatric liver transplant recipients are up to date on age-appropriate immunizations at the time of transplant [6]. In a recent qualitative study with 82 transplant stakeholders (including transplant hepatologists, nephrologists, cardiologists, infectious diseases physicians, transplant nurse coordinators, primary care physicians, and the parents of transplant recipients), the following barriers to pretransplant immunization were identified: (1) gaps in knowledge about the timing and safety of pretransplant vaccines; (2) lack of communication, coordination, and follow-up between team members regarding immunizations; (3) difficulty remembering when vaccines were due; and (4) lack of a centralized immunization record that could be easily accessible by all team members [7].

While human resources are an important component in the overall strategy to address immunization rates, provider-driven interventions (ie, phone call reminders and individually created calendar reminders) are expensive and difficult to sustain. Health information technology tools, including health care mobile apps for use on a personal smartphone or desktop computer, have been demonstrated to be a sustainable strategy for facilitating patient provider communication, disseminating high-quality evidence-based information to end users on a global scale, increasing adherence to medical regimens and tracking when medical interventions are due, and improving outcomes in chronic illnesses [8-12]. mHealth (mobile health) apps have been successfully created and implemented to facilitate immunization delivery [13-20]. However, mHealth apps have never been utilized to improve immunization rates in high-risk populations (such as children awaiting transplant) who require a tailored vaccine schedule (such as the accelerated vaccine schedule) and comanagement by multiple providers (including the primary care physician and transplant team) who in turn may operate on different electronic medical records (EMRs). The goal of this study was to describe the development of a smartphone app, Immune PediatricTransplant, to promote pretransplant immunization and to report on app functionality and usability when piloted in the target population.


mHealth Agile Development and Evaluation Lifecycle

The mHealth Agile Development and Evaluation Lifecycle guided this study. This lifecycle focuses on safety and efficacy while also allowing for rapid and iterative development and evaluation that is required to create high-quality, effective, thoroughly tested, evidence-based digital tools. The mHealth Agile Development and Evaluation Lifecycle is divided into 5 stages: phase 0—project identification; phase 1—user experience, design, development, and alpha testing; phase 2—beta testing; phase 3—clinical trial evaluation; and phase 4—postmarket surveillance [21] (Figure 1). This study focused on the first 3 phases of this cycle—project identification; user experience design, development, and alpha testing; and beta testing.

Figure 1. mHealth Agile Development and Evaluation Lifecycle (adapted from Wilson et al [21]). UI: user interface; UX: user experience.
View this figure

Approach and Recruitment

We used a sequential user-centered design process approach focusing on the first 3 phases of the mHealth Agile Development and Evaluation Lifecycle (Figure 2). All English-speaking transplant hepatologists, cardiologists, nephrologists, infectious diseases physicians, primary care physicians, and parents of children who received liver, heart, and kidney transplants between January 1, 2011, and August 30, 2019, at the Children’s Hospital Colorado (CHCO), Ann & Robert H. Lurie Children’s Hospital of Chicago (Lurie) and Children’s Hospital of Philadelphia (CHOP) were invited to participate. Health care providers were invited to participate via an email invitation. The parents of transplant recipients were approached in clinic or were sent a written invitation. The participants received a reimbursement for participating in the study (US $10 per interview, US $150 per focus group, and US $150 per think-aloud walk-through app trial testing). Institutional review board approval was obtained from the University of Colorado (all research took place through the University of Colorado; CHOP and Lurie only assisted with recruitment), and verbal informed consent was obtained by the interviewer before starting the interview, focus group or think-aloud walk-through app trial [7].

Figure 2. Development process of the Immunize PediatricTransplant app. uMARS: user version of the Mobile Application Rating Scale.
View this figure

Phase 0: Project Identification Through Formative Research to Understand the Barriers to Pretransplant Immunization and Assess Acceptability of a Pretransplant Immunization App

To develop an evidence and theory-based app, we identified barriers to pretransplant immunization based on relevant literature, prior research by our group, and the domains from the Theoretic Domains Framework [4-6,22-26]. We conducted semistructured interviews with 82 transplant stakeholders (including the parents of transplant recipients, transplant nurse coordinators, transplant infectious diseases physicians, primary care physicians, and transplant nephrologists, cardiologists, and hepatologists) to identify barriers to pretransplant immunization and to explore whether an immunization app would be useful in addressing these barriers. We used a team-based inductive approach to analyze the results [7]. Our focus of this manuscript is to present new findings regarding use and acceptability of an immunization app.

Phase 1: User-Centered App Design, Development, and Alpha Testing

Based on the formative research, we worked with CANImmunize Inc to develop a new app for Apple and Android smartphones. The content of the app targets immunization barriers identified in Step 1. Specifically, the app (1) incorporates information about pretransplant vaccine safety and timing; (2) houses an easily accessible cloud-based immunization record for each child; (3) includes a chat or communication feature for providers and parents to communicate about immunizations; and (4) provides reminders for parents and providers when vaccines are due based on the accelerated Centers for Disease Control and Prevention vaccine schedule for transplant candidates.

We held 2 focus group discussions with 5 transplant stakeholder participants in the first group and 6 transplant stakeholder participants in the second group (N=11; 5 parents of transplant recipients, 2 primary care physicians, 2 transplant nurse coordinators, and 2 hepatologists) to obtain feedback on the initial version of the app. We implemented a focus group protocol and semistructured discussion guide consistent with the recommended focus group methodology [27]. We showed the participants screenshots from the app and asked for their thoughts. Both focus groups were led by a moderator, which was held in English and lasted 60 minutes. With the permission of the participants, the focus groups were videotaped, and notes were taken. Based on the findings from the focus groups, we made modifications to the app. Upon completion of the initial functional prototype, CANImmunize Inc completed alpha testing to identify and repair any layout issues, software bugs, or other technical problems.

Phase 2: Beta Testing

We invited 12 transplant stakeholders (6 parents of transplant recipients, 2 primary care physicians, 2 transplant nurse coordinators, and 2 hepatologists) to perform a lab usability testing of the app. Once they consented, the participants downloaded and trialed the app during a think-aloud walk-through over Zoom. Think-aloud walk-throughs are a standard approach for software development and app testing [28-30]. We asked each of the 12 stakeholders to set up an account, create a new child record, input a 16-vaccine immunization record provided by the research team, connect with the research physician, and utilize various app features (eg, including enabling touch/face ID, setting up notifications to be received via text message, and reading through the informational features of the app). This database captured all information inputted into the app and saved the time stamp at which each change was made to the user’s data. This was used to assess how long the participant spent entering the complete vaccine record, the accuracy with which they entered the vaccine record, and the stakeholder’s success in connecting with the research physician. After each participant completed the testing, we asked them to independently complete the user version of the Mobile Application Rating System (uMARS), a validated tool for end users to assess the quality of mHealth apps. The uMARS is a 26-item measure that includes subscales to assess engagement, functionality, aesthetics, and information quality of the app [31]. After every 3-4 usability tests, we incorporated feedback into a new version of the app.


Approach and Recruitment

We interviewed 82 stakeholders including parents or guardians of heart, liver, and kidney transplant recipients, primary care physicians who took care of transplant recipients, transplant infectious diseases physicians transplant nurse coordinators, and transplant hepatologists, nephrologists, and cardiologists (Table 1).

Table 1. Participant demographic characteristics (N=82).
CharacteristicsValues
Transplant subspecialist, n (%)16 (20)
Transplant IDa physician, n (%)3 (4)
Transplant nurse coordinator, n (%)11 (13)
Primary care provider, n (%)12 (15)
Parent or guardian, n (%)40 (49)
Transplant center, n (%)

Children’s Hospital Colorado35 (43)

Children’s Hospital of Philadelphia27 (33)

Ann and Robert H. Lurie Children’s Hospital20 (24)
Gender, n (%)

Female68 (83)

Male14 (17)
Years in practice,b n (%)

0-510 (24)

6-106 (14)

11-2010 (24)

Over 20 years16 (38)
Interview venue, n (%)

Hospital (office or conference meeting room)18 (22)

Telephone64 (78)

aID: infectious diseases.

bexcluding parents (n=42).

Phase 0: Project Identification Through Formative Research to Understand the Barriers to Pretransplant Immunization and Assess Acceptability of a Pretransplant Immunization App

Despite being from diverse geographic regions and having experience with different types of organ transplant (heart, liver, and kidney), when asked about feasibility and potential benefit, 80/82 (98%) participants believed that a mobile health app would be useful to help address and overcome these immunization barriers in the pretransplant period. Parent participants commented that an immunization app could provide educational material on “how vaccines work and what diseases they prevent.” All participants reported that a health information technology tool could improve communication by “getting everyone on the same page, especially when different providers used different EMRs.” Provider participants emphasized that having easy access to a centralized immunization record would “be great, especially for out of state patients whose information is not in the state immunization information system.” Finally, parent and provider participants stated that automated vaccine reminders “would be a huge help in reminding them to get needed vaccines” (Table 2).

Table 2. Selective illustrative quotations about how a transplant-specific immunization app might help address immunization barriers in the pretransplant period
ThemeIllustrative Quotation
Increase factual knowledge
  • I don’t know a lot of the scientific words of what the doctor says—if the app could explain the vaccine, it would be amazing. [parent]
  • It would be great to have information about what the vaccine is, what it protects against, how often it needs to be given, and why its extra important for a soon-to-be immunocompromised child. [parent]
  • I’d like a place to verify vaccine information; I don’t want to mess it up. [primary care provider]
  • If there was a tool where I could enter the child’s age, the vaccines they had received, and if needed anticipated transplant date and it would give me their individualized accelerated schedule that would be terrifically efficient. [infectious diseases physician]
  • Because most people haven’t seen vaccine-preventable infections like measles or mumps, the diseases aren’t as scary as they should be. Providing a brief blurb about what the disease can look like would make families more inclined to follow through on a vaccine. [transplant nurse coordinator]
  • Having education material about each vaccine would be great—a family could click on it and get a recall of why that vaccine’s important. [hepatologist]
Enhance communication and coordination
  • Being able to get everyone on the same page to get questions answered would be great. [parent]
  • An app sounds wonderful—if there could be communication between me, my primary care provider, and the transplant team. Everyone could be on the same track. [parent]
  • It’s exhausting trying to get a hold of someone when you have a question, and you can’t go forward until you reach them. The tool would improve communication a hundredfold. [parent]
  • Efficient communication to everybody sounds pretty great. [primary care provider]
  • Families live on their phone, that is the way to communicate with them. [hepatologist]
Centralize vaccine records
  • I have my child’s vaccines on a card, but if I lose that card or forget to write a new vaccine on the card, I don’t know where that information would be. [parent]
  • When you’re dealing with the stress of a super sick kid you can’t remember every detail like when vaccines were given. It would be great just to open up the app. [parent]
  • A health tool could be a repository for immunizations, particularly for those children from out of state or those children with gaps in their records. [primary care provider]
  • Being able to see in real time the vaccine record would be great. [transplant nurse coordinator]
  • Centralization of records would be great because right now they’re in multiple places. [hepatologist]
Help track when vaccines are due
  • Reminders would be so big—if all of us transplant families could get reminders, we would be able to get immunizations done on time. [parent]
  • If the app could alert not just me but also the doctor’s office that my child had a shot due that would be really helpful. [parent]
  • A reminder on your phone seems simple buts it’s a huge deal for a transplant patient and their family. [parent]
  • Anything that makes it easier for people to remember when a vaccine is due would help us improve immunization rates. [hepatologist]

Phase 1: User-Centered App Design, Development, and Alpha Testing

A total of 11 stakeholders attended 2 focus groups. There were 5 parents of transplant recipients (1 parent of a heart transplant recipient, 2 parents of liver transplant recipients, and 2 parents of kidney transplant recipients), 2 transplant nurse coordinators, 2 primary care physicians, and 2 transplant subspecialists. Of the 11 stakeholders, 9 (82%) were female.

Overall, the participants were enthusiastic about the idea of using an app to help facilitate immunization delivery in the pretransplant period. The participants gave specific suggestions on the app features they would find helpful, including the ability to use the app on both their phone and desktop computer and the ability to use finger-touch capability or facial recognition to login.

This app would definitely have made things easier when my child was going through transplant.
[parent]
I think this app is a great idea for caring for kids with complex medical needs.
[primary care physician]

When asked their preference on information delivery about the timing and safety of vaccines in the pretransplant period, the participants uniformly liked the idea of informational text rather than an informational video or interactive learning session.

I don’t think I would ever have time to watch a video with my kids running around.
[parent]
Short text bullets of information make it easy to skim through information while you’re waiting for a visit to start.
[parent]
I would like information about which specific vaccines my child can and can’t receive before and after transplant.
[parent]
I like text that I can easily reference when I have questions.
[hepatologist]

When talking about how to enter the child’s prior immunization records into the app to create a centralized vaccine record, the participants gave valuable input. They all recommended a scroll down feature to select the month or year in which each vaccine was given as opposed to a free text entry stating that “a scroll down feature would minimize entry errors.” A few participants suggested “visual recognition” whereby the app could take a picture of the child’s previous vaccine record and then input that information directly into the app’s record. All participants were concerned about the amount of time it would take to enter a complete vaccine record into the app. However, they stated that they would be willing to make the initial time commitment if thereafter they could always be able to email or print a copy of the vaccine record from the app.

I would definitely be willing to spend 30 minutes or even an hour entering my child’s vaccine records if that meant I could have all the records in one place moving forward.
[parent]
If I could print out the vaccine record from the app it would be great and worth the initial time needed to enter the vaccines.
[parent]

When discussing the communication tool or chat feature, the parents and providers were excited about the possibility of being able to facilitate communication between the family, the primary care doctor, and the transplant team. Multiple people mentioned that when the primary care provider and the transplant team use different EMRs, it makes communication very difficult. A few providers expressed concern about having an additional patient communication tool that they would need to check and respond to.

As a parent, it would be huge for the primary care physician and the transplant team to talk directly so I don’t have to be the go-between.
[parent]
My primary care physician and transplant doctor used two different platforms when my daughter went through transplant, and it was very frustrating. This app would have been ideal.
[parent]
As a primary care physician, we often feel out of the loop when a child is awaiting transplant; this would have helped to close that loop.
[primary care physician]
I worry a little about now having to check the EMR communication portal and the app.
[hepatologist]

Phase 2: Beta Testing

A total of 12 stakeholders tested the Immunize PediatricTransplant app (10 on an Apple phone and 2 on an Android phone). There were 6 parents of transplant recipients (2 heart transplant recipients, 2 liver transplant recipients, and 2 kidney transplant recipients), 2 transplant nurse coordinators, 2 primary care physicians who had cared for transplant patients, and 2 transplant subspecialists. Of the 12 participants, 10 (83%) were female. User testing identified 6 issues related to usability and functionality. Four usability issues were identified including changing the appearance of the home page to make icons more prominent, changing the specific icon for the chat or communication function, changing the appearance of vaccine information sheets to be more readable, and allowing users to bulk enter vaccines given on the same date or enter multiple dates for the same vaccine. In terms of functionality, the participants suggested 2 features: (1) the inclusion of a PDF tutorial on how to use the app; and (2) the placement of priority stars next to the most administered vaccines in the vaccine entry section in order to help the parents easily identify likely vaccines for each age. Through iterative app changes after every 3-4 user walk-through trials, we modified the app to address all of these issues (Figure 3).

Time stamps from the database identified that, on average, it took 8.1 minutes (SD 1.8 minutes) for the 12 users to enter a 16-vaccine immunization record. The participants entered the vaccine record with 87% (14/16) accuracy (range 69%-100%). All participants were able to connect successfully with the research physician.

Overall, the participants were enthusiastic about the app. All participants stated that the app had the potential to increase knowledge about vaccines and VPIs and improve immunization delivery in the pretransplant period.

Overall, the app is excellent. It’s a very easy to use app…you don’t have to be tech-savvy to use this app.
[parent]
I think this app would be great not just for transplant patients but all children I care for who require care by multiple subspecialists.
[primary care provider]

All 12 participants completed the uMARS after they finished usability testing of the app. The average scores for engagement, functionality, aesthetics, and information were 4.2/5, 4.5/5, 4.6/5 and 4.8/5 respectively. Moreover, 100% of the participants reported that they would likely or definitely recommend this app to everyone.

Figure 3. Screenshots from the Immunize PediatricTransplant app.
View this figure

Principal Findings

In this paper, we described the initial design and user-centered iterative development Immunize PediatricTransplant, a mobile health app designed to increase immunization rates in the pediatric transplant population. Immunize PediatricTransplant addresses and attempts to overcome the barriers to pretransplant immunization by providing educational information about vaccines and VPIs, creating a cloud-based central vaccine record that is easily accessible to families and all members of the health care team regardless of which EMR they use, having a chat feature to enable communication between the family and multiple health care providers, and sending automated reminders (based on the accelerated vaccine schedule used for transplant candidates) to remind families and providers when vaccines are due.

While there are other immunization apps on the market that are able to store a vaccine record and provide general information about vaccines (ie, Docket Immunization Records app [Docket Health Inc], Passport Health, Apple Health app [Apple Inc]), Immunize PediatricTransplant is unique for multiple reasons. First, to our knowledge Immunize PediatricTransplant is the first app that incorporates a communication tool so that multiple providers can communicate directly with the family and other providers about vaccines. Second, Immunize PediatricTransplant is the first app to include an automated vaccine reminder tool that sends out vaccine reminders using a unique vaccine schedule (the accelerated pretransplant schedule). Studies have shown that recall reminders are effective in improving immunization rates regardless of patient age, setting, or vaccination type [32,33]. Additionally, a meta-analysis of 13 randomized controlled studies showed that digital push technologies are more likely to impact vaccine uptake than nondigital interventions [34]. Third, Immunize PediatricTransplant is unique in providing tailored vaccine information for children with a specific health condition (transplant) who require a unique vaccine schedule (the accelerated schedule). The transplant community is a highly motivated and engaged user group that is ideal for pilot testing an mHealth intervention. Fourth, Immunize PediatricTransplant creates a patient-centered cloud-based personal health record across health care networks. Personal health records, which put consumers in control of their health information, are a key feature in health information exchange [10]. If successful in future pragmatic trials, the app could be modified and used not just for transplant candidates and recipients, but also for children with other diseases who have low vaccine rates despite being at high risk for infectious complications (eg, inflammatory bowel disease, rheumatoid arthritis, and lupus) [35-43]. Overall, the participants were extremely positive in their review of the app with 100% of stakeholders recommending the app for transplant families and providers. However, they did express that input of a child’s vaccine record could be improved in future iterations of the app. Currently, the app relies on either the parent or member of the health care team manually entering the name and date of all prior vaccines that the child has received. This leaves potential for error. In this pilot study, 87% of vaccines were entered correctly. Although 7/12 participants (58%) entered the information regarding vaccines with 100% accuracy, there was one participant who had difficulty and only entered 69% of the information correctly. The participants suggested that in the future it would be ideal if one could take a photograph of their child’s prior vaccine record with their phone’s camera, upload the photograph into the Immunize PediatricTransplant app, and then have the app automatically recognize vaccine names and dates. Optimal character recognition with natural language processing has recently been shown to have the potential to accurately identify clinically relevant information contained within the EMR [44-47].

Alternatively, a participant suggested that future iterations of the app could download vaccine information directly from the EMR or state immunization information systems, a feature that has been highlighted as an important future direction in mHealth [10]. However, there is no uniform EMR utilized across health systems in the United States. As of 2017, there were over 600 health information technology developers supplying certified health information technology [48]. In addition, many primary care providers and transplant teams who comanage transplant patients are on different EMRs. Likewise, if a child received vaccines at multiple locations (eg, at the primary care physician’s office, at a community pharmacy, and at their transplant physician’s office), vaccine information may exist in pieces across different EMRs. Unfortunately, state immunization information systems are only accessible by certain in-state providers (not all specialists have access), and transplant patients are often cared for by out-of-state transplant centers. The Immunize PediatricTransplant app allows for all providers, regardless of EMR, to access the complete vaccine record, receive vaccine reminders, and communicate easily with all members of the child’s transplant team.

Certain participants voiced concern that the app could add to the workload of the providers; that it would be another tool that the providers would be responsible for checking and responding to. In future trials of the app among children awaiting liver, heart, and kidney transplant, survey questions will be designed to further understand whether the app helps facilitate communication between various providers and families or adds additional work burden to the providers.

Strengths

The development of the Immunize PediatricTransplant app was strengthened by the use of various qualitative research techniques (semistructured interviews, focus group discussions, and lab usability think-aloud testing) to ensure that we had a thorough understanding of the pretransplant immunization process including all potential barriers faced by transplant candidates and their providers. A user-centered design at all stages of the development ensured that the app would meet the needs and preferences of all various transplant stakeholders (parents, primary care providers, transplant nurses, and transplant physicians). Finally, the uMARS survey data and data entered into the app’s database complemented the qualitative data to ensure that the app was feasible and acceptable to all transplant stakeholder groups.

Limitations

Several limitations are potentially present in this pilot study of the Immunize PediatricTransplant app. First, the app was designed and trialed by English speaking transplant stakeholders from 3 large pediatric transplant centers (CHOP, Lurie, and CHCO). As a result, acceptability, feasibility, and usability findings may not be generalizable to all transplant providers and families. Second, participation in the study was voluntary; therefore, it is possible that there was participant bias the whereby providers and parents with enthusiasm for mHealth might have been more likely to participate. These individuals may also have above-average technical skills. Third, some app features (such as the chat or communication feature, vaccine reminder notifications, and the outbreak map) were unable to be fully tested during the beta testing since they require a live app environment. In a future study, we plan to trial this app, including the aforementioned features, in real time among parents and providers with a child currently awaiting heart, liver, and kidney transplant. Finally, the app is dependent on the manual entry of immunizations by the family or provider. If vaccine entry is incomplete or incorrectly entered, then individualized vaccine recommendations may be inaccurate. In a future study trialing the app among families awaiting transplant, we plan to assess the degree of accuracy in parental or provider vaccine entry.

Conclusions and Future Directions

Despite the high risk for infection posttransplant, the majority of pediatric transplant recipients are underimmunized at the time of transplant. A novel app, Immunize PediatricTransplant, has now been developed, which may overcome the barriers to pretransplant immunization including providing knowledge about vaccines, a communication portal, an easily accessible vaccine record, and an automated vaccine reminder system. This developmental study suggests that the app is functional and acceptable to transplant stakeholders. Future randomized clinical trials among all pediatric solid organ transplant candidates across the United States (agile development phase 3) are needed to trial the app in real time to see if it can improve vaccine rates at the time of transplantation. Additionally, future clinical trials will allow us to evaluate whether the app is useful in educating families about novel vaccines (such as COVID-19) in the pediatric population.

Acknowledgments

The authors would like to acknowledge all transplant stakeholders who participated in this research. AF is supported by National Institute of Health (NIH)/National Center for Advancing Translational Sciences Colorado Clinical and Translational Sciences Awards Grant KL2 TR002534, as well as Agency for Healthcare Research and Quality Grant K08 HS026510-01A1. The contents are the authors’ sole responsibility and do not necessarily represent official NIH views.

Conflicts of Interest

KW and CB are cofounders of CANImmunize Inc. CB is also the chief technology officer of CANImmunize Inc. KW is a paid member of the independent data safety board for the Medicago vaccine trial. MMC is a senior developer at CANImmunize Inc.

  1. Burroughs M, Moscona A. Immunization of pediatric solid organ transplant candidates and recipients. Clin Infect Dis 2000 Jun;30(6):857-869. [CrossRef] [Medline]
  2. Kaplan LJ, Daum RS, Smaron M, McCarthy CA. Severe measles in immunocompromised patients. JAMA 1992 Mar 04;267(9):1237-1241. [Medline]
  3. McGregor RS, Zitelli BJ, Urbach AH, Malatack JJ, Gartner JC. Varicella in pediatric orthotopic liver transplant recipients. Pediatrics 1989 Feb;83(2):256-261. [Medline]
  4. Feldman AG, Sundaram SS, Beaty BL, Kempe A. Hospitalizations for Respiratory Syncytial Virus and Vaccine-Preventable Infections in the First 2 Years After Pediatric Liver Transplant. J Pediatr 2017 Mar;182:232-238.e1. [CrossRef] [Medline]
  5. Feldman AG, Beaty BL, Curtis D, Juarez-Colunga E, Kempe A. Incidence of Hospitalization for Vaccine-Preventable Infections in Children Following Solid Organ Transplant and Associated Morbidity, Mortality, and Costs. JAMA Pediatr 2019 Mar 01;173(3):260-268 [FREE Full text] [CrossRef] [Medline]
  6. Feldman AG, Sundaram SS, Beaty BL, Torres R, Curtis DJ, Kempe A. Immunization Status at the Time of Liver Transplant in Children and Adolescents. JAMA 2019 Nov 12;322(18):1822-1824 [FREE Full text] [CrossRef] [Medline]
  7. Feldman AG, Marsh R, Kempe A, Morris MA. Barriers to Pretransplant Immunization: A Qualitative Interview Study of Pediatric Solid Organ Transplant Stakeholders. J Pediatr 2020 Dec;227:60-68 [FREE Full text] [CrossRef] [Medline]
  8. Krishna S, Boren SA. Diabetes self-management care via cell phone: a systematic review. J Diabetes Sci Technol 2008 May;2(3):509-517 [FREE Full text] [CrossRef] [Medline]
  9. Scherr D, Kastner P, Kollmann A, Hallas A, Auer J, Krappinger H, MOBITEL Investigators. Effect of home-based telemonitoring using mobile phone technology on the outcome of heart failure patients after an episode of acute decompensation: randomized controlled trial. J Med Internet Res 2009 Aug 17;11(3):e34 [FREE Full text] [CrossRef] [Medline]
  10. Kao C, Liebovitz DM. Consumer Mobile Health Apps: Current State, Barriers, and Future Directions. PM R 2017 May;9(5S):S106-S115. [CrossRef] [Medline]
  11. Domin A, Spruijt-Metz D, Theisen D, Ouzzahra Y, Vögele C. Smartphone-Based Interventions for Physical Activity Promotion: Scoping Review of the Evidence Over the Last 10 Years. JMIR Mhealth Uhealth 2021 Jul 21;9(7):e24308 [FREE Full text] [CrossRef] [Medline]
  12. Tabi K, Randhawa AS, Choi F, Mithani Z, Albers F, Schnieder M, et al. Mobile Apps for Medication Management: Review and Analysis. JMIR Mhealth Uhealth 2019 Sep 11;7(9):e13608 [FREE Full text] [CrossRef] [Medline]
  13. Wilson K, Atkinson KM, Penney G. Development and release of a national immunization app for Canada (ImmunizeCA). Vaccine 2015 Mar 30;33(14):1629-1632. [CrossRef] [Medline]
  14. Wilson K, Atkinson KM, Westeinde J. Apps for immunization: Leveraging mobile devices to place the individual at the center of care. Hum Vaccin Immunother 2015 Jun 25;11(10):2395-2399 [FREE Full text] [CrossRef] [Medline]
  15. Wilson K, Atkinson KM, Deeks SL, Crowcroft NS. Improving vaccine registries through mobile technologies: a vision for mobile enhanced Immunization information systems. J Am Med Inform Assoc 2016 Jan;23(1):207-211 [FREE Full text] [CrossRef] [Medline]
  16. CANImmunize. App Store Preview.   URL: https://itunes.apple.com/ca/app/immunize-canada/id574951915?mt=8 [accessed 2021-12-14]
  17. Feldman AG, Atkinson K, Wilson K, Kumar D. Underimmunization of the solid organ transplant population: An urgent problem with potential digital health solutions. Am J Transplant 2020 Jan;20(1):34-39 [FREE Full text] [CrossRef] [Medline]
  18. Zaidi S, Shaikh SA, Sayani S, Kazi AM, Khoja A, Hussain SS, et al. Operability, Acceptability, and Usefulness of a Mobile App to Track Routine Immunization Performance in Rural Pakistan: Interview Study Among Vaccinators and Key Informants. JMIR Mhealth Uhealth 2020 Feb 13;8(2):e16081 [FREE Full text] [CrossRef] [Medline]
  19. Mekonnen ZA, Gelaye KA, Were M, Tilahun B. Effect of Mobile Phone Text Message Reminders on the Completion and Timely Receipt of Routine Childhood Vaccinations: Superiority Randomized Controlled Trial in Northwest Ethiopia. JMIR Mhealth Uhealth 2021 Jun 15;9(6):e27603 [FREE Full text] [CrossRef] [Medline]
  20. Mekonnen ZA, Gelaye KA, Were MC, Gashu KD, Tilahun BC. Effect of mobile text message reminders on routine childhood vaccination: a systematic review and meta-analysis. Syst Rev 2019 Jun 28;8(1):154 [FREE Full text] [CrossRef] [Medline]
  21. Wilson K, Bell C, Wilson L, Witteman H. Agile research to complement agile development: a proposal for an mHealth research lifecycle. NPJ Digit Med 2018;1:46 [FREE Full text] [CrossRef] [Medline]
  22. Rubin LG, Levin MJ, Ljungman P, Davies EG, Avery R, Tomblyn M, Infectious Diseases Society of America. 2013 IDSA clinical practice guideline for vaccination of the immunocompromised host. Clin Infect Dis 2014 Feb;58(3):e44-100. [CrossRef] [Medline]
  23. Feldman AG, Feudtner C, Kempe A. Reducing the Underimmunization of Transplant Recipients. JAMA Pediatr 2018 Feb 01;172(2):111-112. [CrossRef] [Medline]
  24. Feldman AG, Kempe A, Beaty BL, Sundaram SS, Studies of Pediatric Liver Transplantation (SPLIT) Research Group. Immunization practices among pediatric transplant hepatologists. Pediatr Transplant 2016 Dec;20(8):1038-1044. [CrossRef] [Medline]
  25. Harris K, Baggs J, Davis RL, Black S, Jackson LA, Mullooly JP, et al. Influenza vaccination coverage among adult solid organ transplant recipients at three health maintenance organizations, 1995-2005. Vaccine 2009 Apr 14;27(17):2335-2341. [CrossRef] [Medline]
  26. Cane J, O'Connor D, Michie S. Validation of the theoretical domains framework for use in behaviour change and implementation research. Implement Sci 2012;7:37 [FREE Full text] [CrossRef] [Medline]
  27. Krueger RA, Casey MA. Focus Groups: A Practical Guide for Applied Research 3rd Edition. Thousand Oaks, California, USA: SAGE Publications; 2000.
  28. Al Ayubi SU, Parmanto B, Branch R, Ding D. A Persuasive and Social mHealth Application for Physical Activity: A Usability and Feasibility Study. JMIR Mhealth Uhealth 2014 May 22;2(2):e25 [FREE Full text] [CrossRef] [Medline]
  29. Lim J, Cloete G, Dunsmuir DT, Payne BA, Scheffer C, von Dadelszen P, et al. Usability and Feasibility of PIERS on the Move: An mHealth App for Pre-Eclampsia Triage. JMIR Mhealth Uhealth 2015;3(2):e37 [FREE Full text] [CrossRef] [Medline]
  30. Nikolaus S, Bode C, Taal E, Vonkeman HE, Glas CA, van de Laar MA. Acceptance of New Technology: A Usability Test of a Computerized Adaptive Test for Fatigue in Rheumatoid Arthritis. JMIR Hum Factors 2014;1(1):e4 [FREE Full text] [CrossRef] [Medline]
  31. Stoyanov SR, Hides L, Kavanagh DJ, Wilson H. Development and Validation of the User Version of the Mobile Application Rating Scale (uMARS). JMIR Mhealth Uhealth 2016;4(2):e72 [FREE Full text] [CrossRef] [Medline]
  32. Szilagyi PG, Bordley C, Vann JC, Chelminski A, Kraus RM, Margolis PA, et al. Effect of patient reminder/recall interventions on immunization rates: A review. JAMA 2000 Oct 11;284(14):1820-1827. [CrossRef] [Medline]
  33. Jacobson Vann JC, Jacobson RM, Coyne-Beasley T, Asafu-Adjei JK, Szilagyi PG. Patient reminder and recall interventions to improve immunization rates. Cochrane Database Syst Rev 2018 Jan 18;1:CD003941 [FREE Full text] [CrossRef] [Medline]
  34. Atkinson KM, Wilson K, Murphy MS, El-Halabi S, Kahale LA, Laflamme LL, et al. Effectiveness of digital technologies at improving vaccine uptake and series completion - A systematic review and meta-analysis of randomized controlled trials. Vaccine 2019 May 21;37(23):3050-3060. [CrossRef] [Medline]
  35. Lu Y, Bousvaros A. Immunizations in children with inflammatory bowel disease treated with immunosuppressive therapy. Gastroenterol Hepatol (N Y) 2014 Jun;10(6):355-363 [FREE Full text] [Medline]
  36. Waszczuk K, Waszczuk E, Mulak A, Szenborn L, Paradowski L. A 'cocoon immunization strategy' among patients with inflammatory bowel disease. Eur J Gastroenterol Hepatol 2015 Mar;27(3):249-253. [CrossRef] [Medline]
  37. Soon IS, deBruyn JCC, Wrobel I. Immunization history of children with inflammatory bowel disease. Can J Gastroenterol 2013 Apr;27(4):213-216 [FREE Full text] [CrossRef] [Medline]
  38. Selby L, Kane S, Wilson J, Balla P, Riff B, Bingcang C, et al. Receipt of preventive health services by IBD patients is significantly lower than by primary care patients. Inflamm Bowel Dis 2008 Feb;14(2):253-258. [CrossRef] [Medline]
  39. Lawson EF, Trupin L, Yelin EH, Yazdany J. Reasons for failure to receive pneumococcal and influenza vaccinations among immunosuppressed patients with systemic lupus erythematosus. Semin Arthritis Rheum 2015 Jun;44(6):666-671 [FREE Full text] [CrossRef] [Medline]
  40. Harris JG, Maletta KI, Ren B, Olson JC. Improving Pneumococcal Vaccination in Pediatric Rheumatology Patients. Pediatrics 2015 Sep;136(3):e681-e686. [CrossRef] [Medline]
  41. Grein IHR, Groot N, Lacerda MI, Wulffraat N, Pileggi G. HPV infection and vaccination in Systemic Lupus Erythematosus patients: what we really should know. Pediatr Rheumatol Online J 2016 Mar 08;14(1):12 [FREE Full text] [CrossRef] [Medline]
  42. Yazdany J, Tonner C, Trupin L, Panopalis P, Gillis JZ, Hersh AO, et al. Provision of preventive health care in systemic lupus erythematosus: data from a large observational cohort study. Arthritis Res Ther 2010;12(3):R84 [FREE Full text] [CrossRef] [Medline]
  43. Kremers HM, Bidaut-Russell M, Scott CG, Reinalda MS, Zinsmeister AR, Gabriel SE. Preventive medical services among patients with rheumatoid arthritis. J Rheumatol 2003 Sep;30(9):1940-1947. [Medline]
  44. Kostrinsky-Thomas AL, Hisama FM, Payne TH. Searching the PDF Haystack: Automated Knowledge Discovery in Scanned EHR Documents. Appl Clin Inform 2021 Mar;12(2):245-250. [CrossRef] [Medline]
  45. Goodrum H, Roberts K, Bernstam EV. Automatic classification of scanned electronic health record documents. Int J Med Inform 2020 Dec;144:104302 [FREE Full text] [CrossRef] [Medline]
  46. Laique SN, Hayat U, Sarvepalli S, Vaughn B, Ibrahim M, McMichael J, et al. Application of optical character recognition with natural language processing for large-scale quality metric data extraction in colonoscopy reports. Gastrointest Endosc 2021 Mar;93(3):750-757. [CrossRef] [Medline]
  47. Nguyen A, O'Dwyer J, Vu T, Webb PM, Johnatty SE, Spurdle AB. Generating high-quality data abstractions from scanned clinical records: text-mining-assisted extraction of endometrial carcinoma pathology features as proof of principle. BMJ Open 2020 Jun 11;10(6):e037740 [FREE Full text] [CrossRef] [Medline]
  48. Health Care Professional Health IT Developers. Health IT.   URL: https:/​/dashboard.​healthit.gov/​quickstats/​pages/​FIG-Vendors-of-EHRs-to-Participating-Professionals.​php [accessed 2021-12-14]


CHCO: Children’s Hospital Colorado
CHOP: Children’s Hospital of Philadelphia
EMR: electronic medical record
mHealth: mobile health
uMARS: user version of the Mobile Application Rating Scale
VPIs: vaccine-preventable infections


Edited by G Eysenbach; submitted 27.07.21; peer-reviewed by T Payne, S Weinland; comments to author 19.09.21; revised version received 23.09.21; accepted 23.11.21; published 13.01.22

Copyright

©Amy G Feldman, Susan Moore, Sheana Bull, Megan A Morris, Kumanan Wilson, Cameron Bell, Margaret M Collins, Kathryn M Denize, Allison Kempe. Originally published in JMIR Formative Research (https://formative.jmir.org), 13.01.2022.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.