Abstract
Background: Visits to medical subspecialists are common, with follow-up timing often based on heuristics rather than evidence. Unnecessary visits contribute to long wait times for new patients. Specialists could enhance visit timing and reduce frequency by systematically monitoring patients’ symptoms between visits, especially for symptom-driven conditions like rheumatoid arthritis (RA). We previously designed an intervention using a mobile health (mHealth) app to collect patient-reported outcomes (PRO). One of several aims of the app was to assist rheumatologists in determining visit timing for patients with RA. The intervention did not reduce visit frequency.
Objective: To explore possible reasons for the lack of association between the intervention and visit frequency, we describe app usage, assess usability, and identify barriers and facilitators for using between-visit PRO data to reduce visits when patients’ symptoms are stable.
Methods: We analyzed patients’ use of the app by reporting adherence (percent of PRO questionnaires completed during the 12-month study) and retention (use in the last month of the study). To examine rheumatologists’ experiences, we summarized views of the electronic health record (EHR)–embedded PRO dashboard and EHR inbox messages suggesting early or deferred visits. We assessed app usability using the interactive mHealth App Usability Questionnaire for Ease of Use and Usefulness for patients and the System Usability Scale for rheumatologists. We assessed rheumatologist-level effects of intervention usage using Kruskal-Wallis rank sum and equality of proportion tests. We identified barriers and facilitators through interviews and surveys.
Results: The analysis included 150 patients with RA and their 11 rheumatologists. Patients answered a median of 53.3% (IQR 34.1%-69.2%) of PRO questionnaires; this proportion varied by rheumatologist (range 40.7%‐67%). Over half of the patients used the app during the final month of the study (56%, range 51%‐65%, by rheumatologists); the median number of months of use was 12 (IQR 9-12). Rheumatologists viewed the dashboard 78 times (17.6% of 443 visits) with significant differences in viewing rates by rheumatologist (range 10%‐66%; P<.01). There were 108 generated messages sent to rheumatologists suggesting a deferred visit (24.4% of 443 visits) with significant differences in message counts received per visit by rheumatologist (range 10.8%‐22.6%; P=.03). Rheumatologists’ reported barriers to offering visit deferrals included already scheduling as far out as they were comfortable and rescheduling complexities for staff. Based on 39 patient interviews and 44 surveys, patients reported 2 main barriers to app usage: questionnaire frequency not being tailored to them and reduced motivation after not discussing PRO data with their rheumatologist. A total of 5 interviewed patients received the option to defer their visits, of which 3 elected to defer the appointment and 2 chose to keep it.
Conclusions: While an mHealth app for reporting RA PROs was used frequently by patients, using these data to reduce the frequency of unneeded visits was not straightforward. Better engagement of clinicians may improve the use of PRO data.
doi:10.2196/60854
Keywords
Introduction
Visits to medical specialists represent a substantial component of health care services. About half of patient visits to physicians in the United States are to specialists, totaling roughly 500 million visits per year at a cost of billions of dollars [,]. The current approach to care delivery inefficiently allocates specialty care resources: visits are scheduled largely in preset intervals (eg, 3, 6, or 12 months) that are not evidence-based and have a weak relationship to clinical need [,]. Sometimes these schedules are required for insurance purposes, that is, to have medications covered. Evidence for optimal visit timing is scant [,]. As a result, specialists’ schedules fill with potentially unneeded visits, creating inefficiencies, wasteful spending, and patient burdens [], and contribute to longer wait times for other patients who need specialist care; these access issues are especially pronounced for rural, minority, and other vulnerable populations [].
We previously designed a mobile health (mHealth) intervention to improve patient care in rheumatology []. It was well accepted by patients, and we became interested in whether one additional value of the app could be to reduce visit follow-up frequency when clinically appropriate [,]. The rationale was that a reasonable proportion of visits were for stable patients where remote symptom reporting could help clinicians defer visits. We focused on rheumatology because this specialty has a great need for optimizing visit timing due to projected workforce shortages in the next decade []. Furthermore, prior works suggest a potential for reducing visit frequency, in part, because a substantial number of visits, for rheumatoid arthritis (RA) in particular, do not result in a change in treatment, suggesting limited clinical use [,-]. The mHealth intervention involved patients using a smartphone app to record their symptoms using patient-reported outcome (PRO) questionnaires; based on these data, their rheumatologists received notifications that either suggested the need for an earlier visit if symptoms were acutely worsening or the potential to defer a scheduled visit if symptoms were stable (see Methods for details).
Our initial clinical trial of this intervention found no association with reduced visit frequency. To explore possible reasons for this finding and inform further intervention development, we examined quantitative and qualitative data on patient and rheumatologist engagement and data from surveys and interviews. We describe the app intervention usage, assess usability, and identify possible barriers and facilitators to improve visit follow-up frequency using between-visit PRO data.
Methods
Setting and Study Population
The prior study was conducted at 2 rheumatology practices affiliated with Brigham and Women’s Hospital, which is part of Mass General Brigham, a large academic medical center in Boston, MA, United States. All clinics within this health system share an electronic health record (EHR) system (Epic Systems, Inc). The intervention was implemented by engaging 11 clinically focused rheumatologists who previously provided input on the intervention design [,]. As described previously, we identified eligible patients by searching upcoming patient visit schedules for the 11 rheumatologists [,]. We used prior diagnosis codes to identify patients with RA. Only established patients were included, and therefore all prescheduled visits were follow-ups. We recruited these patients via patient portal messages, phone calls, and in-person prior to scheduled visits.
mHealth Intervention
The intervention was designed with extensive input from patients and rheumatologists through several iterations as part of a user-centered design process [,,,]. Briefly, the intervention included a patient-facing mHealth app that used push notifications to prompt patients to complete one of four types of validated PRO questionnaires regarding pain interference, function, fatigue, and RA disease activity []. Each questionnaire was repeated every 8 days in a staggered fashion so that patients completed one questionnaire every 2 days. Rheumatologists were able to view all results within the patient’s EHR chart. In addition, the PRO data could trigger 3 types of EHR inbox messages to the rheumatologists. First, all rheumatologists received a message that data were available for viewing 48 hours before a visit with a patient who had entered PRO data. Second, if PRO data suggested several weeks of worsening symptoms, the rheumatologist received a message with a suggestion to offer an earlier visit. Finally, if PRO data suggested stable symptoms during the visit interval from the prior visit until 2 weeks prior to a scheduled visit, rheumatologists received a message suggesting that the patient be offered a deferred visit. The specific logic that determined when these messages are sent has been described previously [].
Study Design and Data Sources
As noted, this initial trial of the mHealth app found no change in visit frequency attributable to the PRO data and EHR inbox messages []. We sought to explore potential reasons for these findings and inform further intervention development by (1) describing use of the mHealth app by patients and use of the PRO data by rheumatologists, (2) assessing mHealth app and EHR-embedded PRO data usability, and (3) identifying barriers and facilitators. Patient and rheumatologist perspectives were assessed through surveys during and after the study period. Data sources included the EHR (for demographic and clinical characteristics of study patients and visit timing), mHealth app and dashboard usage logs (for patient adherence and retention, rheumatologist views of the PRO data, and messages sent suggesting early or deferred visits), interviews with patients (for barriers and facilitators), and surveys of patients and rheumatologists (for usability and barriers and facilitators).
For analyses of intervention usage and usability, rheumatologists with 15 or more patients in the study were analyzed as individuals, and the rest were aggregated. Because only one of the instances in which an earlier visit was suggested resulted in an earlier visit, we focused the analysis primarily on aspects related to deferred visits [].
Intervention Usage Analysis
We used descriptive statistics to summarize patient adherence and retention rates with the mHealth app. We used the Kruskal-Wallis rank sum test to assess differences in median patient PRO completion percentage across rheumatologists. We used descriptive statistics to summarize volumes and timing of EHR inbox messages sent to the rheumatologists suggesting the potential for an early or deferred visit and view of the EHR-embedded PRO data. For both measures, we tested for equality of proportions across rheumatologists.
Usability Analysis
We summarized results from mHealth App Usability Questionnaire (MAUQ) items, which were administered to patients after they completed the study []. The MAUQ is a validated instrument measuring mHealth app usability. Specifically, we used the interactive versions of the ease of use and satisfaction (MAUQ_E) and usefulness (MAUQ_U) subscales of the questionnaire because they were most relevant to this intervention. The MAUQ scales range from 1 (worst, ie, strongly disagree with all questions) to 7 (best, ie, strongly agree with all questions), with 4 meaning neutral.
For rheumatologists, we summarized the results of the System Usability Scale (SUS) administered after all the rheumatologists’ patients had completed the study [,]. The SUS is a widely used usability measure with a range from 0 (worst) to 100 (best), with an average score across evaluations of 68‐70.
Barriers and Facilitators Analysis
We invited patients who had received a suggested early or deferred visit to participate in an interview. We conducted those interviews virtually and transcribed the recordings. Using conventional content analysis, one research team member (RSR) created a codebook and summarized the results using the framework method, and a second research team member (DHS) reviewed a portion of the results to confirm findings [,]. We used similar methods to analyze free-text responses from patient survey results administered at study completion. When presenting qualitative results from interviews, we used “some” to indicate a finding was present in fewer than 5 responses, “many” to denote greater than 5 responses, and “most” to denote greater than half of the responses.
For rheumatologists, we summarized data from a survey administered after study completion about how often they offered patients the option to defer a visit and the reasons for not offering that option.
All quantitative analyses were conducted in R (version 4.2.2; R Foundation for Statistical Computing).
Ethical Considerations
This study was reviewed and approved by the institutional review board at Mass General Brigham (number 2021P000790). All written and audiovisual data (which were not anonymized) were stored on secure servers accessible only to the research team. Written consent was obtained for all patients to participate in the trial and for the use of their medical data in analyses, verbal consent was obtained for interviews, and consent information was presented before all electronic survey questions. Participants were compensated with a gift card for the trial and a US $25 check for the interviews.
Results
Participants
Characteristics of the 150 intervention patients included in the study are shown in 
Table 1. Patients were mostly female and White.
| Characteristics | Values |
| Sex, n (%) | |
| Female | 125 (83.3) |
| Male | 25 (16.7) |
| Age (years), median (IQR) | 62 (52-69) |
| Race, n (%) | |
| Asian | 4 (2.7) |
| Black | 5 (3.3) |
| White | 136 (90.7) |
| Declined or others | 5 (3.3) |
| Seropositivity status, n (%) | |
| Missing | 36 (24) |
| 0 | 39 (26) |
| 1 | 75 (50) |
| CRPa,b, median (IQR) | 1.4 (0.7‐4.5) |
| DMARDc,d use, n (%) | |
| csDMARDe | 93 (62) |
| btsDMARDf | 90 (60) |
| Other medications, n (%) | |
| NSAIDsg | 30 (20) |
| Corticosteroid | 55 (36.7) |
| Opioids | 25 (16.7) |
| Comorbidities, n (%) | |
| Hypertension | 26 (17.3) |
| Obesity | 15 (10) |
| Depression | 7 (4.7) |
| Osteoporosis | 19 (12.7) |
aCRP: C-reactive protein.
bTwo values were missing for CRP.
cDMARD: disease-modifying antirheumatic drug.
dIt is common for patients to use more than 1 DMARD.
ecsDMARD: conventional synthetic disease-modifying antirheumatic drug.
fbtsDMARD: biologic or targeted synthetic disease-modifying antirheumatic drug.
gNSAID: nonsteroidal anti-inflammatory drug.
Intervention Usage
Four rheumatologists each had at least 15 patients in the intervention and were analyzed as individual observations; the other 7 rheumatologists were aggregated and treated as one observation. The median percentage of PRO questionnaires completed was 53.3% across all patients (Table 2). This completion rate varied by rheumatologists, ranging from 40.7% (the rheumatologist with the most patients) to 67% (the rheumatologist with the third most patients). Differences in completion rates between all rheumatologists or groups were not statistically significant based on the Kruskal-Wallis rank sum test (P=.50). The median number of months patients answered questionnaires was 12 (IQR 9-12), with little variation by rheumatologist; 56% (84/150) of patients completed at least one questionnaire in the final month of the study. See app usage by rheumatologists in Table 2 and over time in 
Figure 1.
| Rheumatologista | Patients, n | Patient visits, n | Percent PROb questionnaires completed,c median (IQR) | Months with answered questionnaires,d median (IQR) | Patients who completed at least 1 PRO in the final month of the study, n (%) |
| All rheumatologists | 150 | 443 | 53.3 (34.1-69.2) | 12 (9-12) | 84 (56) |
| Four rheumatologists with the greatest volume of study patients | |||||
| 1st highest | 51 | 148 | 40.7 (25.8-63.7) | 11 (8-12) | 26 (51) |
| 2nd highest | 25 | 68 | 48.9 (34.1-64.8) | 12 (9-12) | 13 (52) |
| 3rd highest | 20 | 53 | 67 (49.5-78.6) | 12 (11-12) | 13 (65) |
| 4th highest | 15 | 49 | 46.7 (44-61) | 12 (9-12) | 8 (53) |
| All other rheumatologists | |||||
| 5th-11th highest | 39 | 125 | 58.8 (47.3-72) | 12 (11-12) | 24 (62) |
aOrdered by volume of study patients who receive care from that rheumatologist.
bPRO: patient-reported outcome.
cThe total possible PRO questionnaires offered were 182.
dNumber of months within the 12-month study in which patients answered at least 1 PRO questionnaire.
There was variation by rheumatologist in the percentage of their received messages that suggested a deferred visit, ranging from 18.9% of patient visits (the rheumatologist with the most patients) to 37.7% (the rheumatologist with the third most patients) (Table 3). In a 6-sample test for equality of proportions, there was a statistically significant difference in the percent of deferred visit suggestions received by rheumatologists (P=.03). There were no significant differences among early visits. For rheumatologist usage of the PRO data as measured by the number of views, there was variation by rheumatologist ranging from 10.1% of patient visits (rheumatologist with the most patients) to 66% (rheumatologist with the third most patients). In a 6-sample test for equality of proportions, the difference was statistically significant (P<.01), which means that at least one value (number of views of PRO data per patient visit) differed significantly from the others.
| Rheumatologista | Patients, n | Patient visits, n | Messages sent to rheumatologist suggesting visit deferral, n (%) | Messages sent to rheumatologist suggesting early visit, n (%) | Rheumatologist views of PROb data in EHRc, n (%) |
| All rheumatologists | 150 | 443 | 108 (24.4) | 31 (6.9) | 78 (17.6) |
| Four rheumatologists with the greatest volume of study patients | |||||
| 1st highest | 51 | 148 | 28 (18.9) | 8 (5.4) | 15 (10.1) |
| 2nd highest | 25 | 68 | 13 (19.1) | 5 (7.4) | 11 (16.2) |
| 3rd highest | 20 | 53 | 20 (37.7) | 5 (9.4) | 35 (66) |
| 4th highest | 15 | 49 | 18 (36.7) | 3 (6.1) | 7 (14.3) |
| All other rheumatologists | |||||
| 5th-11th highest | 39 | 125 | 29 (23.2) | 10 (8) | 10 (8) |
aOrdered by volume of study patients who receive care from that provider.
bPRO: patient-reported outcome.
cEHR: electronic health record.
Usability
Of the 150 intervention patients, 88 completed the MAUQ surveys with a median MAUQ_E of 6.3 (IQR 5.6-6.9) and median MAUQ_U of 4.8 (IQR 4.1-5.6) (Table 4 and 
Figure 2). Based on Figure 2, most patients found the app easy to use across all questions. However, the usefulness of the app was less clear for patients. Most patients agreed that the app “would be useful for my health and well-being,” but about half of patients were neutral about its usefulness on the other questions. There was a small range across rheumatologists of the aggregated MAUQ_E (6.2‐6.9) and a more substantial range for the MAUQ_U (4.6‐6.3).
Of 11 participating rheumatologists, 10 completed the SUS to assess the EHR-embedded dashboard and reported a mean of 68.25 (range 45‐82.5). (The average score for the SUS across a wide range of apps is 68 (SD 12.5) [].)
| Rheumatologista | Patients, n | MAUQ_Eb, median (IQR) | MAUQ_Ub, median (IQR) |
| All rheumatologists | 88 | 6.3 (5.6-6.9) | 4.8 (4.1-5.6) |
| Four rheumatologists with the greatest volume of study patients | |||
| 1st highest | 31 | 6.2 (5.5-6.6) | 4.6 (3.4-5.3) |
| 2nd highest | 17 | 6.3 (6.2-7.0) | 4.6 (4.2-5.0) |
| 3rd highest | 12 | 6.7 (6.4-7.0) | 5.2 (4.3-6.3) |
| 4th highest | 4 | 6.9 (6.2-7.0) | 6.3 (5.7-6.6) |
| All other rheumatologists | |||
| 5th-11th highest | 24 | 6.3 (5.6-6.9) | 4.8 (3.8-5.3) |
aOrdered by volume of study patients who receive care from that provider.
bThe mHealth App Usability Questionnaire for Ease of Use (MAUQ_E) and Usefulness (MAUQ_U) were rescaled from 1‐5 to 1‐7 (worst to best) to be consistent with their standard use, with 1 indicating strongest disagreement with all questions and 7 indicating strongest agreement with all questions.
Barriers and Facilitators
We interviewed 39 patients shortly after their responses triggered a message to their rheumatologist suggesting a possible early or deferred visit. We also assessed free text comments from 44 patients who completed the final study survey. Facilitators and barriers are summarized in Table 5. The most cited benefit of the intervention mentioned by patients was that the app improved awareness of their symptoms: “I really enjoyed using this app and felt like it gave me a great way to check in with myself on my symptoms and take a pulse on how my RA was” [Patient 78]. Patients described benefits both from being aware of when their symptoms were worse and from recognizing when they were feeling well, such as when a medication change was working (“I’m in the green every time, it’s amazing!” [Patient 141]).
| Intervention component | Facilitators | Barriers |
| PROa questionnaires | Increased awareness of symptoms was perceived as a benefit; graph of responses helped assess the impact of medications |
|
| Discussing PRO scores with rheumatologists | Discussion motivated patients to continue using the app and be aware of their symptoms |
|
| Offers for deferred visits | Perceived time saved from avoiding unneeded visits was viewed positively |
|
| Offers for early visits | Potential for early visits helped make patients feel looked after; patients trust in rheumatologist’s judgment made them likely to accept an earlier visit |
|
| User experience with technology | App was viewed as easy to use |
|
aPRO: patient-reported outcome.
bRA: rheumatoid arthritis.
Although most of the patients we spoke with did not discuss the app with their rheumatologists, many did, and most of those patients were appreciative and motivated to use the app because of it: “She said ‘well I can see on this graph’...it was definitely comforting because I’m in the good part...so it was good she brought it up” [Patient 28]; “I did discuss the app data with my doctor...It was very helpful to review them with her” [Patient 180]. In a few cases, patients described examples in which decisions regarding medications were made based on discussion of the app data. In 2 cases, a patient described how the app data helped inform the decision during a visit to further increase visit intervals. Some of the patients who discussed the app data with their rheumatologists said the app added little to their care: “I was cognizant that [my doctor] looked at the app and has referred to it in discussing my condition, but she is a sufficiently good and attentive communicator that I didn’t notice much in the way of changed communications with her (which I considered excellent to begin with)” [Patient 145].
However, many patients were not aware that their rheumatologist has access to or viewed the data: “My doctor and I have never discussed the use of this app. I’m not even sure she knows about the app” [Patient 144]. This reduced motivation to use the app: “I was disappointed that the app was not used at all, really, in my care. I felt that the information just went into cyberspace and was never heard from again” [Patient 119]; “I don’t see any connection between the app and the care I receive with my particular rheumatologist...I found it interesting that she didn’t bring it up so that gave me the message that she doesn’t find it helpful in her decision making” [Patient 30]. One patient noted a disconnect between her impression of the primary purpose of the app (for her rheumatologist to oversee her PRO responses and reach out if needed) and her rheumatologist’s view of the app’s main purpose (for patient self-management).
Of the 5 interviewed patients who reported receiving a call offering them the option to defer their visit because of stable symptoms they reported in the app, 3 accepted the offer to defer the visit: “Not having to take the trip down to Boston from Maine, as long as [my doctor] thought was fine, we said okay and we rescheduled it [from September until November]” [Patient 200]. One of those patients had a symptom flare shortly after electing to defer the visit: “the following week I had a bad rheumatoid attack and I thought ‘oh shoot I should have kept that appointment’ but I think it was okay” [Patient 190]. Two decided to keep their appointment: “when she called me I was in the middle of...about 5 different joints that were inflamed and so I said to her that I’d like to keep the visit” [Patient 145]. When asked about the potential to receive such a call, almost all patients interviewed had a positive view: “If I’m feeling well and it’s just kind of a check-in, then I wouldn’t need to drive into Boston for it” [Patient 70]; “I’d do a cartwheel...it is such a hassle [to drive into Boston]...traffic can be brutal” [Patient 141].
The only patient we interviewed who received a call from the clinic offering them the option for an earlier visit based on their PRO data believed that they did not need to come in early for their RA. When asked about the potential to receive such a call, almost all patients interviewed had a positive view: “when you have to call in...you’re hesitant to do that because you know you’re always wondering ‘oh is this something that’s just going to pass in a little while’...whereas if you guys could actually see the graph changing and call...it is comforting” [Patient 28]; “I think that would be great!...I’d feel like I’m not just spinning the wheel for nothing, answering the questions” [Patient 70]. However, a few believed they were already in good communication with their rheumatologist and would have contacted them if they needed help: “Very likely I would’ve already been there...I would’ve already thought to give her a call and let her know I’m not feeling great” [Patient 30].
About half of patients interviewed or who provided free text comments on the survey mentioned challenges related to the app questionnaire: some found them too frequent (“I did not see the value and found the number of surveys excessive and as I am in remission” [Patient 84]); some found them not frequent enough (“Every day would be better than every other day because you know if you do it every day then you know when you miss a day” [Patient 103]); and some found them inadequate for capturing their relevant symptoms (“This app did NOT capture the total wellbeing of the patient. It is strictly about joints. My RA involves my lungs” [Patient 3]).
Of the 11 rheumatologists, 10 completed the final survey (Table 6). Responses showed that the major barrier for rheumatologists not offering deferred visits was that they were already scheduling patients out as far as they were comfortable with. The complexity of rescheduling visits and the need for patients to come in for labs anyway were also notable barriers.
a| Reasons for not offering patients with stable PROsb the option to defer a visitc | Top ranked, nd | In top three, nd | Indicated at least once |
| I am too busy to spend time deciding if the visit could be delayed | 0 | 1 | Yes |
| Most of my patients should come in even if their PROs are stable | 0 | 1 | Yes |
| Most of my patients need to come in for labs anyway | 1 | 5 | Yes |
| Rescheduling visits makes scheduling more complicated for me and my staff | 2 | 6 | Yes |
| Rescheduling visits increases my workload by removing easier visits | 0 | 0 | No |
| I am already scheduling patients as far out as I am comfortable with | 3 | 8 | Yes |
| If the open slots from a delayed visit are not filled, there will be lost revenue | 0 | 0 | Yes |
| I do not trust the PRO data to inform the decision to delay a visit | 0 | 0 | No |
| Delaying a visit could complicate prior authorizations | 0 | 2 | Yes |
| My time needed to determine if the visit could be delayed and not reimbursed | 0 | 1 | Yes |
| Othere | 0 | 0 | Yes |
aN=10 responses; 1 rheumatologist had left the clinic, and all others responded.
bPRO: patient-reported outcome.
cExact wording for questions: You received in-basket messages suggesting delayed visits during the study about patients with stable PROs who had visits scheduled in 2 weeks. The messages suggested that you consider asking your assistant to call the patient and give them the option to delay the visit. (1) Of these in-basket messages, please estimate how many for which you gave the patient the option to delay the visit (choose 1 response): none, some, all. (2) For patients for whom you did not offer a delayed visit, what do you think were the primary reasons? Rank all factors that influenced you (1=most influential and so on). (3) Please add any other thoughts you have about offering patients the option to delay a visit based on their PROs. (Write as much or as little as you’d like.)
d“Top ranked” means the respondent listed the item as their first choice. “In top three” means the respondent listed the item as their first, second, or third choice. Some respondents did not rank their responses numerically but rather selected options using an ‘x’—we did not count those in the “top ranked” and only included them in the “top three” if they selected three or fewer.
eOne rheumatologist wrote, “some patients seem to want to come in anyway, even if they don’t have to come.”
Discussion
Principal Results
Our examination of intervention usage, usability, and barriers and facilitators provides some potential reasons—from both rheumatologist and patient perspectives—for why our prior study did not find an effect of an intervention designed to improve visit follow-up frequency.
From the rheumatologist perspective, although usability of the EHR-embedded dashboard was satisfactory, use of the dashboard varied substantially across rheumatologists, with most using it in a small percentage of visits. Many participating rheumatologists did not offer patients deferred visits because they were not comfortable increasing visit intervals any further, believed rescheduling visits was complicated for them and their staff, or allowed the laboratory testing schedule to determine visit timing.
From the patient perspective, most continued to use the app in the final month of the study and found the app easy to use as measured by the MAUQ_E. The MAUQ_U scores were less favorable, suggesting that patients found the app less useful than easy to use. Relatively few disagreed that it was useful, but many were neutral on its usefulness. While some interviewed patients found the app useful to improve self-awareness of their symptoms, many found the questionnaires too frequent, and one patient found that the lack of questionnaires about nonarticular symptoms was problematic. Some patients were not aware that their rheumatologist would use their reported data to determine visit timing (which was correct in some cases) or that their rheumatologist even had access to these data, which reduced their motivation to complete the questions. However, patients were largely enthusiastic about the idea of reducing the need for visits because of the time it could save them. Although some patients liked the idea of being called for earlier visits, many were skeptical that the data they reported in the app would be useful to their rheumatologists to help inform that decision.
Implications and Comparison With Prior Work
Although we identified several rheumatologist- and patient-facing barriers, the participants who engaged in the intervention provided a proof of concept that this intervention has the potential to improve the efficiency of face-to-face visits by facilitating visit deferrals when patients have stable symptoms. Between-visit symptom monitoring has other benefits, such as helping patients be more aware of their symptoms so they can get help sooner and facilitating discussions of symptoms during visits [,,-]. This work provides early evidence that improving visit efficiency can be added to that list of potential benefits. Qualitative data and the wide range of dashboard usage by rheumatologists suggest that clinician engagement is likely the critical factor that determines the extent to which visit deferrals will be offered and patients find the app useful. Other clinically integrated digital interventions have also encountered this challenge [-].
Clinician-facing enhancements may include generating evidence of the benefits of further spacing out visits to improve buy-in, simplifying the patient rescheduling process, and making patients preferences for longer visit intervals available to them. Patient-facing enhancements may include personalizing the timing and topics of the questionnaires based on patient responses or specified preferences, allowing the patients to add notes to elaborate on their symptoms when the questionnaires do not adequately capture them, assessing patient interest in changing visits, and informing patients within the user interface of how their rheumatologists will use the data. These enhancements require additional design and user testing. This approach may also prove promising in other medical specialties such as pulmonology and cardiology, which will also require additional design effort.
To our knowledge, this is the first attempt to use data collected between visits to inform visit timing in the United States. Prior work in rheumatology has attempted to free up appointments using scheduled nurse phone calls but did not base decisions on longitudinally collected patient data []. A recent rheumatology trial in the Netherlands found that app-supported patient-initiated care achieved comparable clinical results with fewer rheumatology consults for patients with stable low disease activity [].
Limitations
This work has several limitations. First, the intervention was implemented in one academic health center with 11 rheumatologists who lacked direct incentives to be more efficient with visit timing. Other settings with different incentives may have had different experiences with the intervention. Second, the clinical logic for when an EHR inbox message was sent to the rheumatologists suggesting early or deferred visits was determined based on the expert opinion of one rheumatologist (DHS) with review and agreement by the participating study rheumatologists rather than through empirical evaluation. Therefore, the logic may have suggested changes to visit timing that were not appropriate or missed opportunities to change visits that were appropriate. Third, we were unable to comprehensively attribute an actual early or deferred visit to an app-generated message because we did not have data for when a provider initiated a patient outreach to offer an early or deferred visit in response to a study-generated message (that data is not recorded reliably or systematically in the EHR). Finally, responses to the rheumatologist survey may be influenced by social desirability bias []. Specifically, it is notable that none indicated that removing easier visits would be a barrier, and few selected barriers related to revenue and reimbursement, which may nonetheless be important factors to some clinicians.
Conclusions
Informing visit timing systematically based on data collected from patients between specialty visits would, among other benefits, represent a fundamental advance in health care delivery—a move from one-size-fits-all visit timing to data-informed determinations that personalize care to individual patient needs. We examined how a PRO-focused mHealth app for RA might contribute to deferring unneeded follow-up visits. The app suggested many opportunities to defer visits, some of which resulted in rescheduling visits, though many did not. Based on our quantitative and qualitative results, clinicians had inconsistent buy-in, which patients perceived. The experience of the rheumatologists who did engage in the intervention and the patients who received the option to defer a visit provide a proof of concept that this intervention has the potential to succeed in reducing visit frequency if further enhanced based on user-centered design to improve clinician engagement and patient-perceived usefulness. It is also possible that in different health care settings, that is, nonacademic practices, or when used by clinicians practicing under different incentives (eg, capitation and performance measures relevant to access), such an app that allows for rescheduling patients may achieve greater uptake and produce significant improvements in access. Directly addressing clinician buy-in is likely the key factor and requires experimentation with different kinds of implementation strategies, such as stronger leadership support, audit feedback, and direct incentives. If successfully adopted, this type of intervention may reduce per-patient visit frequency, improve patient satisfaction, and improve access.
Acknowledgments
This study was supported by the Rheumatology Research Foundation and VERITY (P30-AR072577). RSR carried out this work independently, outside of his employment at RAND.
Data Availability
The datasets generated and analyzed during this study are not publicly available to protect the privacy of the research participants.
Conflicts of Interest
RSR has received consulting fees from the Assistant Secretary for Technology Policy at the US Department of Health and Human Services unrelated to this work. DHS receives separate research funding from CorEvitas, Janssen Pharmaceuticals, and Novartis. He also receives royalty payments from UpToDate on unrelated content. IG receives consulting fees from FPrime unrelated to this work.
References
- Santo L, Kang K. National hospital ambulatory medical care survey: 2019 national summary tables. Centers for Disease Control and Prevention; Jan 20, 2023. [CrossRef]
- Martin AB, Hartman M, Lassman D, Catlin A, National Health Expenditure Accounts Team. National health care spending in 2019: steady growth for the fourth consecutive year. Health Aff (Millwood). Jan 2021;40(1):14-24. [CrossRef] [Medline]
- Javorsky E, Robinson A, Boer Kimball A. Evidence-based guidelines to determine follow-up intervals: a call for action. Am J Manag Care. 2014;20(1):17-19. [Medline]
- Ganguli I, Wasfy JH, Ferris TG. What is the right number of clinic appointments? Visit frequency and the accountable care organization. JAMA. May 19, 2015;313(19):1905-1906. [CrossRef] [Medline]
- Jiang Y, Rudin RS, Solomon DH. Clinical visit frequencies in rheumatology: a systematic literature review. Arthritis Care Res (Hoboken). Oct 2023;75(10):2054-2062. [CrossRef] [Medline]
- Ganguli I, Chant ED, Orav EJ, Mehrotra A, Ritchie CS. Health care contact days among older adults in traditional Medicare: a cross-sectional study. Ann Intern Med. Feb 2024;177(2):125-133. [CrossRef] [Medline]
- Sequist TD. Ensuring equal access to specialty care. N Engl J Med. Jun 9, 2011;364(23):2258-2259. [CrossRef]
- Solomon DH, Dalal AK, Landman AB, et al. Development and testing of an electronic health record-integrated patient-reported outcome application and intervention to improve efficiency of rheumatoid arthritis care. ACR Open Rheumatol. Nov 2022;4(11):964-973. [CrossRef] [Medline]
- Solomon DH, Altwies H, Santacroce L, et al. A mobile health application integrated in the electronic health record for rheumatoid arthritis patient–reported outcomes: a controlled interrupted time‐series analysis of impact on visit efficiency. Arthritis Rheumatol. May 2024;76(5):677-683. [CrossRef]
- Battafarano DF, Ditmyer M, Bolster MB, et al. 2015 American College of Rheumatology Workforce Study: supply and demand projections of adult rheumatology workforce, 2015–2030. Arthritis Care Res (Hoboken). Apr 2018;70(4):617-626. [CrossRef]
- Hewlett S, Kirwan J, Pollock J, et al. Patient initiated outpatient follow up in rheumatoid arthritis: six year randomised controlled trial. BMJ. Jan 22, 2005;330(7484):171. [CrossRef] [Medline]
- Seppen B, Wiegel J, Ter Wee MM, et al. Smartphone-assisted patient-initiated care versus usual care in patients with rheumatoid arthritis and low disease activity: a randomized controlled trial. Arthritis Rheumatol. Nov 2022;74(11):1737-1745. [CrossRef] [Medline]
- Butt S, Newman E, Smith N. Nurse scheduled telephone visit: the right rheumatology care for the right patient at the right time. Arthritis Rheumatol. 2016;68(Suppl 10). URL: https://acrabstracts.org/abstract/nurse-scheduled-telephone-visit-the-right-rheumatology-care-for-the-right-patient-at-the-right-time/ [Accessed 2024-10-10]
- Colls J, Lee YC, Xu C, et al. Patient adherence with a smartphone app for patient-reported outcomes in rheumatoid arthritis. Rheumatology (Sunnyvale). Jan 5, 2021;60(1):108-112. [CrossRef]
- Lee YC, Lu F, Colls J, et al. Outcomes of a mobile app to monitor patient-reported outcomes in rheumatoid arthritis: a randomized controlled trial. Arthritis Rheumatol. Aug 2021;73(8):1421-1429. [CrossRef] [Medline]
- Zhou L, Bao J, Setiawan IMA, Saptono A, Parmanto B. The mHealth App Usability Questionnaire (MAUQ): development and validation study. JMIR Mhealth Uhealth. Apr 11, 2019;7(4):e11500. [CrossRef] [Medline]
- Bangor A, Kortum PT, Miller JT. An empirical evaluation of the System Usability Scale. Int J Hum Comput Interact. Jul 29, 2008;24(6):574-594. [CrossRef]
- Lewis JR. The System Usability Scale: past, present, and future. Int J Hum Comput Interact. Jul 3, 2018;34(7):577-590. [CrossRef]
- Hsieh HF, Shannon SE. Three approaches to qualitative content analysis. Qual Health Res. Nov 2005;15(9):1277-1288. [CrossRef] [Medline]
- Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol. Sep 18, 2013;13(1):117. [CrossRef] [Medline]
- Hyzy M, Bond R, Mulvenna M, et al. System Usability Scale benchmarking for digital health apps: meta-analysis. JMIR Mhealth Uhealth. Aug 18, 2022;10(8):e37290. [CrossRef] [Medline]
- Rudin RS, Fanta CH, Qureshi N, et al. A clinically integrated mHealth app and practice model for collecting patient-reported outcomes between visits for asthma patients: implementation and feasibility. Appl Clin Inform. Oct 2019;10(5):783-793. [CrossRef] [Medline]
- Rudin RS, Fanta CH, Predmore Z, et al. Core components for a clinically integrated mHealth app for asthma symptom monitoring. Appl Clin Inform. Oct 2017;8(4):1031-1043. [CrossRef] [Medline]
- Sulca Flores JA, Dalal AK, Sousa J, et al. Evaluation of a primary care-integrated mobile health intervention to monitor between-visit asthma symptoms. Appl Clin Inform. Aug 2024;15(4):785-797. [CrossRef] [Medline]
- Meirte J, Hellemans N, Anthonissen M, et al. Benefits and disadvantages of electronic patient-reported outcome measures: systematic review. JMIR Perioper Med. Apr 3, 2020;3(1):e15588. [CrossRef] [Medline]
- Bartlett SJ, De Leon E, Orbai AM, et al. Patient-reported outcomes in RA care improve patient communication, decision-making, satisfaction and confidence: qualitative results. Rheumatology (Oxford). Jul 1, 2020;59(7):1662-1670. [CrossRef] [Medline]
- Rudin RS, Thakore N, Mulligan KL, Ganguli I. Addressing the drivers of medical test overuse and cascades: user-centered design to improve patient-doctor communication. Jt Comm J Qual Patient Saf. Apr 2022;48(4):233-240. [CrossRef] [Medline]
- Ganguli I, Mulligan KL, Chant ED, et al. Effect of a peer comparison and educational intervention on medical test conversation quality: a randomized clinical trial. JAMA Netw Open. Nov 1, 2023;6(11):e2342464. [CrossRef] [Medline]
- Greenhalgh T, Wherton J, Papoutsi C, et al. Analysing the role of complexity in explaining the fortunes of technology programmes: empirical application of the NASSS framework. BMC Med. May 14, 2018;16(1):66. [CrossRef] [Medline]
- Kreuter F, Presser S, Tourangeau R. Social desirability bias in CATI, IVR, and web surveys: the effects of mode and question sensitivity. Public Opin Q. Dec 1, 2008;72(5):847-865. [CrossRef]
Abbreviations
| EHR: electronic health record |
| MAUQ: mHealth App Usability Questionnaire |
| MAUQ_E: mHealth App Usability Questionnaire for Ease of Use |
| MAUQ_U: mHealth App Usability Questionnaire for Usefulness |
| mHealth: mobile health |
| PRO: patient-reported outcome |
| RA: rheumatoid arthritis |
| SUS: System Usability Scale |
Edited by Amaryllis Mavragani; submitted 02.12.24; peer-reviewed by Johannes Knitza, Michael Bass; final revised version received 28.01.25; accepted 28.01.25; published 21.04.25.
Copyright© Robert S Rudin, Leah M Santacroce, Ishani Ganguli, Daniel H Solomon. Originally published in JMIR Formative Research (https://formative.jmir.org), 21.4.2025.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.