Osteoarthritis (OA) is the most common form of arthritis and a leading cause of disability worldwide [1]. Current management strategies for knee OA largely target symptom control with the primary aim of reducing pain, improving function, and delaying the need for surgical intervention [2]. There are currently no disease-modifying therapies for OA, and existing pharmacological treatments for targeting symptoms are largely ineffective [3,4]. Nonpharmacological interventions such as exercise, weight management, and patient education have been shown to be effective in reducing OA-related pain and disability [5,6].

The latest clinical guidelines for OA recommend self-management, including health education and goal setting, as the core treatment alongside nonpharmacological and pharmacological treatments [7,8] when used as a supplement to medical care [9]. In the context of chronic disease management, goal setting and action planning have been found to improve self-efficacy, encourage behavioral change, and improve health outcomes [10-12]. Educational interventions, when delivered alongside exercise and weight loss programs, have also been shown to lead to better treatment adherence, reductions in pain, better self-management, and improved quality of life [13]. OA self-management tools can have an important impact on improving the delivery of interventional programs targeting patient education [14] and behavioral modification [15] and can also influence the effectiveness of patient–health care provider (HCP) communication and shared decision-making [9,16]. As such, acceptance and adoption of mobile health (mHealth) strategies for self-management can help improve health outcomes, reduce costs to the health care system, and encourage patients to take a more active role in improving their health.

mHealth apps have shown promise in supporting patient self-management of health conditions, especially for chronic diseases that require long-term care and maintenance [9]. However, barriers to the adoption and use of mHealth technologies have been identified. First, the lack of user-centered designs integrating patients’ needs and preferences intercepts the adoption of and sustained engagement with such mHealth technologies [9,17]. Second, there has been a lack of formal mHealth evaluations to date [18,19], raising concerns about the safety and effectiveness of mHealth technologies [20,21]. Our understanding of the effectiveness of mHealth apps in supporting self-management and their impact on patient-reported outcomes in adults with knee OA remains in its early stages. Thus, there is both a need to fill the gap in the availability of mHealth apps for knee OA management and provide a framework to evaluate mHealth apps targeting OA self-management driven by the priorities and feedback of target users.

In our previous work, we explored the perspectives of end users on an mHealth app for knee OA self-management, leading to the co-design of an app prototype aimed at facilitating self-management and improving patient–physician communication [22,23]. The resulting app reflected a consensus of patient and HCP priority functional requirements, achieved through co-design. In this study, the overarching objective is to evaluate the app from both patient and HCP perspectives using qualitative and quantitative assessments. The primary aims of the 2-part patient evaluation are to assess the app by its (1) overall usability and quality, (2) ability to improve patient self-management behavior (goal setting and activity completion), and (3) effectiveness in improving quality of life, patient activation, and patient-HCP communication. The primary aim of the 1-time HCP evaluation is to assess the overall usability, quality, and perceived impact on patient-HCP communication.

We used a combination of qualitative and quantitative methods and triangulated data for increased rigor to evaluate the app features and functionalities from patient and HCP perspectives. Details on the main features and functionalities included in the app, which were determined in preceding co-design sessions, are summarized in Textbox 1. The app was assessed at baseline by patients and HCPs using semistructured interviews and surveys, followed by a 6-week usability period and a final follow-up survey evaluation for patients only. This study was reviewed and approved by the University of Calgary research ethics board (REB16-1372). Privacy and confidentiality of data were maintained throughout all phases of the study. All personal identifying information was stripped from interview documents and recruitment materials, and all the data were deidentified and stored in a password-protected electronic file on secure servers at the University of Calgary.

Surveys were administered to patients at baseline and follow-up to evaluate changes in patient-reported outcomes over the course of the 6-week pilot trial. At the baseline patient evaluation, data were captured on demographics, knee symptoms, OA risk factors, quality of life, and patient activation. Quality of life was assessed using the European Quality-of-Life 5-Dimension 5-Level Questionnaire (EQ-5D-5L), a preference-based measure for describing and evaluating health, covering 5 dimensions (mobility, self-care, usual activities, pain or discomfort, and anxiety or depression) scored on a 5-point Likert scale [29]. The 10-item Patient Activation Measure (PAM-10) was administered to patients at week 0 and week 6 to provide a measure of patient activation, defined as a person’s knowledge, skills, and confidence related to managing their own health [30]. The PAM-10 was designed to minimize the response burden from its 13-item successor, the PAM-13, and has been reported to have comparable levels of consistency and reliability with the PAM-13 [31]. The PAM provides a measure of a patient’s engagement in self-management of their disease and thus enables change in patient activation to be monitored between baseline and follow-up evaluations.

At the patient follow-up evaluation, the EQ-5D-5L and PAM tools were re-administered along with additional measures assessing perceived app usability and quality. App usability was measured using the System Usability Scale (SUS), a well-established psychometric tool used worldwide with high levels of reported validity and reliability [32]. App quality was assessed using the App Chronic Disease Checklist (ACDC), an expert opinion–based checklist developed to evaluate the usability of chronic disease apps for monitoring, self-management, and behavioral change [33]. Finally, 2 sections (app subjective quality and app-specific quality) from the Mobile App Rating Scale (MARS) were used to provide an objective rating of perceived app quality [34], with each question measured on a 5-point Likert scale. However, it should be noted that not all sections from the ACDC and MARS tools were included in the assessment, as they included dimensions beyond the scope of the app prototype features (eg, gamification).

During the 1-time evaluation with HCPs, the SUS and MARS measures were used to assess perceived app usability and quality, respectively, from the provider perspective. Further details on each survey instrument used in the pilot testing evaluation are included in Tables 1 and 2.

Descriptive statistics were computed for instruments measured only once at the baseline and follow-up assessments, including demographic characteristics, app usability (SUS), and app quality (MARS and ACDC), reporting the mean and SD or frequency and percentage where appropriate. Because of the small sample size, normality assumptions were not met; thus, nonparametric methods were used for all subsequent analyses. Wilcoxon matched-pairs signed-rank tests were performed to compare outcomes measured at both baseline and follow-up (PAM and EQ-5D-5L), including P values. In addition, app user metadata collected during the 6-week app usability period were exported, including the frequency of goal setting and activity completion. Spearman rank correlations (r_s) were used to evaluate correlations between patient activation (as measured by the PAM), number of goals set, and number of activities completed for knee OA management, with correlations <0.20 considered negligible [37].

For the descriptive thematic analysis, PaCER researchers generated descriptive statements for each app feature that reflected the scope of the responses. Key messages were then extracted and summarized using inductive coding [38] (Multimedia Appendix 3). The notes taken during the HCP interviews were organized according to participants, questions, and app features. Using the audio recordings, the notes were then expanded upon and validated to ensure that they accurately represented the perspectives of participating HCPs. Once all interviews were completed, primary care researchers from ACTT compared similarities and differences in HCP responses and summarized the findings into descriptive categories by each app feature.

We have demonstrated that an app prototype co-designed with end users has the potential to successfully deliver self-management guidance to patients with knee OA. During the 6-week observation period, patients experienced significant improvements in patient-reported quality of life and patient activation and exhibited high levels of engagement with the app, as demonstrated by a high number of activities completed and goals achieved during the usability period. However, usability scores were reported to be in the marginal range for patients and an unacceptable range for HCPs. The overall average app ratings from patients and HCPs demonstrated the potential opportunity for further quality improvement.

The qualitative assessment revealed that, from the patient perspective, the app features, including the dashboard, goals, activities, flags, and resources, were useful and user-friendly but could be expanded to include functions that are more personalized and specific to each patient’s lived experiences, such as the ability to add notes and reminders. Patients indicated that they were most likely to use the resources feature, emphasizing that the knowledge aspect of the tool was important to their self-management. Furthermore, patients largely viewed the app as a catalyst to increased autonomy in self-management. Many elements of the app features were viewed as useful by HCPs, particularly those related to the inclusion of evidence-based information; specific, measurable, attainable, relevant, time-based goal setting, and exercise therapies for OA.

However, there were gaps between patients’ and HCPs’ perceptions of app usability and quality, with HCPs expressing greater concern than patients about the patient’s ability to effectively navigate and use the app features. This is consistent with our previous work [22], where physicians and patients expressed different views on the seriousness of knee OA and their approach to its management and also their perceptions of whether those diagnosed with knee OA can manage their disease using an app or other mHealth tools. Despite patient receptiveness to the prospect of using an mHealth tool such as an app, patients and physicians held diverging views, where physicians were concerned about the technological literacy of the conventionally older OA population. Negotiating consensus during co-design about the app features beneficial to both patients and physicians may help align patient and physician perspectives. Furthermore, improved patient-HCP communication and the discussion of priorities and best-available evidence may help bolster shared decision-making [42].

We recognize several limitations of our study design. The evaluation sample size was small but aligned with the suggestion by Nielsen et al [43] that for projects of medium to large size, 15-20 test users are optimal to balance evaluation costs with the benefits of finding usability problems in testing. Furthermore, our patients were predominantly female and had higher educational attainment. Thus, our patient sample may not be representative of the target early knee OA population, as research has shown that active participants research may have different motivations and priorities compared with those who are less engaged [44], and departures from representativeness may be amplified with increasing age [45]. This may be reflected by our study findings, in which 88% (15/17) and 100% (17/17) of participants reported PAM scores in the highest range of patient activation at baseline and follow-up assessments, respectively, despite the use of multiple recruitment mechanisms intended to capture a range of patient perspectives.

Similarly, the HCPs involved in the design and evaluation of the mHealth app were primarily family physicians. Thus, our findings may not reflect the diversity of perspectives among HCPs that are typically involved in guiding self-management for patients with knee OA. Further studies will expand the focus of the evaluation to assess its applicability in a more diverse group of HCPs (eg, physiotherapists). In addition, the inclusion of patient partners with a wider range of experiences related to education, health literacy, and technological proficiency in future app development and evaluation is essential to address potential health inequities [46].

There were shortcomings in app usability, as demonstrated by the low SUS scores reported by both patients and HCPs. However, usability challenges were of greater concern among HCPs in comparison with patients who reported higher usability scores and identified minimal to no usability challenges in the qualitative assessment. This discrepancy between patient and HCP perceptions of usability is aligned with the findings from our preliminary work [22], where HCPs largely underestimated patients’ receptiveness and ability to use mHealth technologies for OA self-management. It is possible that HCPs evaluated the app usability with a broader OA patient population in mind than that represented in our sample, who were both highly educated and demonstrated high levels of activation in their care.

This study had a relatively short duration of follow-up of 6 weeks. As such, the emphasis on the evaluation of patient-reported outcomes against clinical outcomes was suitable, as it was unlikely that we would observe clinically meaningful changes in OA symptoms during the short observation period. However, although significant improvements in health-related quality of life and patient activation were reported during the study period, the change in EQ-5D-5L scores did not surpass the MID established in the degenerative knee population. Thus, the length of follow-up may have been insufficient to establish meaningful changes in some patient-reported outcomes. Given the chronic nature of OA, using a longer evaluation period for a future iteration of the app would provide more relevant data on clinical outcomes to augment the patient-reported outcomes featured here and additional context for whether long-term engagement with the app could be sustained. Further app development involving an evaluation that is more inclusive of diverse patient and HCP perspectives, with a longer follow-up duration and a wider range of outcomes measured, will be an essential next step in this study. In addition, we are exploring opportunities to integrate self-monitoring data with advanced machine learning analytics to provide an intelligent platform to assist patients with knee OA in initiating and sustaining self-management activities.

To our knowledge, this is the first mHealth app developed specifically to aid in knee OA self-management, although several digital self-management programs have been developed for OA [47] and other chronic diseases, including diabetes [48,49] and heart failure [50]. A 2020 systematic review on digital health interventions for people with OA identified that most (5/8, 63%) studies were primarily focused on health education (n=5), whereas some incorporated additional self-management elements, such as goal setting (n=6), action planning (n=4), physical activity (n=6), weight management (n=5), and pain management (n=6) [47]. A 2017 systematic review focusing specifically on mHealth technologies identified a lack of emphasis on tracking OA symptoms and self-management behavior that could be useful for shared decision-making [9]. The review by Choi et al [9] provided a framework for developing mHealth apps for OA management, describing the need for patient-facing mobile apps with capabilities such as symptom monitoring, activity monitoring, joint function measurement, physical activity guidelines, educational content, and provision of data visualization and summary reports for shared-decision-making. Thus, our app addressed many of the identified market gaps for apps in the OA self-management domain by incorporating capabilities that have been established as integral to facilitating self-management, decision support, and shared decision-making.

Regarding methods for evaluating mHealth tools, a recent scoping review summarizing quality assessment methods for mobile apps in chronic disease management found minimal agreement on the most appropriate criteria for evaluating mobile apps, with only 18% (12/65) of apps including evidence-based health information, 22% (14/65) using a behavioral change framework, and 37% (24/65) applying usability metrics as quality criteria for app assessment [51]. On the basis of the gaps identified in the analysis and practice methods for evaluating mobile apps, the authors proposed 3 primary goals for building quality criteria for app assessment: (1) prioritize existing evidence and knowledge against ease of assessment (eg, using patient-reported outcomes vs app ratings), (2) emphasize principles of behavior change theory, and (3) explicitly incorporate the patient perspective. Our study begins to address these identified gaps in quality assessment methods using a range of evidence-based patient-reported outcomes, incorporating behavioral change principles such as goal setting and activity monitoring, and integrating the patient perspective from early planning to app assessment phases.

This pilot study provides support for the use of evidence-based, patient-centered mHealth apps to improve patient-reported outcomes by encouraging patient self-efficacy and improving patient-HCP communication, ultimately promoting conservative management of knee OA. These findings, along with findings from previous study phases, provide a framework from which app developers and researchers can co-design and evaluate mHealth apps targeting self-management in a way that is inclusive of all stakeholders and reflects diverse user perspectives. Further development of the app to address usability and feasibility in the context of a larger evaluation trial, including a more diverse and representative population sample and a longer period of evaluation, will be instrumental in understanding the impact of the app on self-management and a broader range of clinical outcomes such as pain and disability. In addition, our findings can provide a basis for developing best-practice reporting standards and practices to evaluate evidence-based mHealth technologies that target self-management of chronic conditions. Future app development may involve the integration of machine learning to provide personalized self-management recommendations to patients with knee OA to address individual needs and priorities.