Background: Patient-centered measurement (PCM) aims to improve the overall quality of care through the collection and sharing of patient values, outcomes, and perspectives. However, the use of PCM in care team decisions remains limited. Integrated knowledge translation (IKT) offers a collaborative, adaptive approach to explore best practices for incorporating PCM into primary care practices by involving knowledge users, including patients and providers, in the exploratory process.
Objective: This study aims to test the feasibility of using patient-generated data in team-based care; describe the use of these data for team-based mental health care; and summarize patient and provider care experiences with PCM.
Methods: We conducted a multi-method exploratory study in a rural team-based primary care clinic using IKT to co-design, implement, and evaluate the use of PCM in team-based mental health care. Care pathways, workflows, and quality improvement activities were adjusted iteratively to improve integration efforts. Patient and provider experiences were evaluated using individual interviews relating to the use of PCM and patient portals in practice. All meeting notes, interview summaries, and emails were analyzed to create a narrative evaluation.
Results: During co-design, a care workflow was developed to incorporate electronically collected patient-generated data from the patient portal into the electronic medical record, and customized educational tools and resources were added. During implementation, care pathways and patient workflows for PCM were developed. Patients found portal use easy, educational, and validating, but data entries were not used during care visits. Providers saw the portal as extra work, and the lack of portal and electronic medical record integration was a major barrier. The IKT approach was invaluable for addressing workflow changes and understanding the ongoing barriers to PCM use and quality improvement.
Conclusions: Although the culture toward using PCM is changing, the use of PCM during care has not been successful. Patients felt validated and supported through portal use and could be empowered to bring these data to their visits. Training, modeling, and adaptable PCM methods are required before PCM can be integrated into routine care.
Seminal work in the 1960s and 1970s, supporting the combination of the concepts of a medical home with attributes of patient-centered care , provided the foundation for research that demonstrated the benefits of this care model (eg, comprehensive, coordinated care within a primary care team). Benefits linked to patient-centered medical home models include improved health-related quality of life, self-management, and depression; reduced hospital admissions; and improved clinical measures such as blood pressure and glycated hemoglobin [ ]. However, the concept of patient-centeredness has been poorly theorized and operationalized, although several papers have identified key attributes such as patient and family being respected; given complete health information; being involved in decision-making; and supported in their physical, psychological, and social needs [ , ]. Newly realized in British Columbia (BC), the patient’s medical home is a community practice that operates at an ideal level to provide longitudinal patient-centered, team-based primary care [ ].
To assess these attributes, measures are needed to capture patient-centered care in a form that can be used to rate care quality and quality improvement (QI). Patient-centered measurement (PCM) has the potential to capture data to improve patient care experiences, care quality, communication, and trust. To this end, the BC Ministry of Health and the 7 health authorities established the BC PCM Steering Committee  in 2021 to implement scientifically rigorous approaches to collect and report patient-generated data (PGD). Examples of sources of these data include patient-reported experience measures (PREMs) and patient-reported outcome measures (PROMs).
However, the integration of PCM into regular care visits and decisions presents major conceptual, methodological, and logistical challenges in translating this body of knowledge into routine clinical practice [- ]. A systematic review found that PGD improved patient health awareness and communication with providers but that difficulties arose when patients wanted greater provider involvement with their data during clinic visits [ ]. Lordon et al [ ] found that providers had difficulty accommodating patient requests for engagement with PGD because of the perceived lack of value, time constraints, and lack of workflow integration. Even with access to patient portals for data entry, the ability to incorporate and track these data varies across systems because of organizational, practice, workflow, resource, and technological challenges [ - ].
There is an unprecedented opportunity to develop and test best practices for incorporating PCM into clinical care because of the rapid uptake of virtual care and enhanced digital literacy for both patients and providers during the COVID-19 pandemic. This requires planning to determine which strategies to use, selection considerations about which patient populations to target for PREMs and PROMs, choice of specific measures, and engagement considerations, including how clinical care teams will incorporate and act on the data . Given the complexity of implementing PCM in clinical practice, multilevel implementation science frameworks are effective, with the choice of framework or theory based on fit for purpose [ , ]. Iterative knowledge sharing, or integrated knowledge translation (IKT), was identified for our research, whereby planning, implementing, and evaluating team-based care performance could be optimally developed through the lens of providers, patients, and the research team [ ]. Synergies derived from this IKT approach enhance the understanding of patients’ and providers’ context and needs, thereby enhancing the relevance of the generated research and increasing user knowledge and understanding of the research process, awareness of the research, and appreciation for how and when it can be applied [ ].
For this research, we partnered with the regional district of Kootenay Boundary (KB) Division of Family Practice , whose mission is to help rural practitioners meet patient and practice needs and lead change as part of a province-wide initiative to strengthen health care in BC. The focus on people living in rural and remote communities is critical to address limited access to providers and services, and the resulting health disparities of higher chronic disease multimorbidity and all-cause mortality [ - ]. In this context, digital health solutions have great potential to address these health disparities [ ]. The study team assessed priority areas for rural clinical care and identified mental health care as a high-needs area based on BC Community Health Data [ ]. Mental illness is a common and disabling health problem in Canada, affecting 1 in 5 Canadians, and has become of even greater importance during the COVID-19 pandemic [ ]. Care team–based patient-centered planning strategies have important potential in the treatment of mental illnesses, such as anxiety and depression [ ]. Mental health concerns have worsened during the pandemic; based on the 2019 Community Health Survey, almost 5 million Canadians aged ≥12 years (16%) had seen or spoken to a health care professional in the previous year about their mental health, an increase of 2% since 2015 [ ].
The overall research aim was to develop new methods to incorporate patient-generated mental health and experience data for team-based in-clinic and virtual care. The new methods for PCM that emerged from this study are reported in another publication (M Antonio et al, unpublished data, December 2021). This paper reports on the IKT approach that was used in the multi-method study with the following objectives: (1) to explore the feasibility of integrating PGD using a patient portal in team-based care; (2) to describe the use of these data for team-based mental health care; and (3) to summarize patient and provider care experiences with PCM.
The study took place in a rural team-based care practice in the southern interior of BC, Canada, between February 2020 and April 2021. The private, multi-provider primary care clinic served as the patient’s medical home . The study received institutional ethical approval (protocol number: BC H19-03855). Notably, the first stage began at the onset of the COVID-19 pandemic, and the study was completed during the pandemic. As such, the research was conducted virtually, including regular interactions with providers, patients, and the research team.
The research used a multi-method, IKT approach with interconnected stages of study (). In line with the directive in a scoping review on IKT in health care [ ], we created a protocol, timeline, and IKT plan to guide our efforts, and the analysis of continuous and documented feedback offered the research team the ability to report findings with sufficient detail to reveal how IKT was associated with outcomes. Stage 1 of the study involved co-design during which the research team worked with the care team to identify relevant PCMs and optimize the use of the patient portal in the context of the team’s clinical needs, roles and responsibilities, and workflow. Stage 2 included the implementation of the portal and care workflows with adaptation based on feedback and evaluation of its use and impact on clinical care. Implementation and evaluation occurred concurrently so that feedback could be incorporated and further evaluated.
The research team included patient partners, researchers, subject-matter expert scholars, and our industry partner. The patient partners (PB and MS) and two researchers (MA and SD) met regularly with clinic staff and care providers recruited to the study and with other knowledge users at various points within the study, including local professionals from our study collaborators, KB Division of Family Practice and General Practice Service Committee Practice Support Program . Monthly research meetings were held with the entire research team to discuss and apply what we had learned to date.
Recruitment and Composition of the Provider Practice
A pragmatic approach and convenience sampling were used to recruit a community team-based clinic. The KB Division of Family Practice invited one of the early adopter clinics of a patient medical home care model to participate in both study stages. The enrolled clinic included 2 physicians, 2 medical office assistants, and a newly hired social worker and registered nurse (RN). The practice used an electronic medical record (EMR) but had no experience with the use of a patient portal, and their EMR did not have an embedded portal. The practice had an identified patient panel within the EMR and received summary patient data from the KB Patient Experience Survey (a questionnaire currently in use by the Divisions of Family Practice to understand patient experience with care) and the Canadian Primary Care Sentinel Surveillance Network , a primary care research network that offers a web-based data presentation tool to improve primary health care delivery outcomes across the country. However, these data were not used to practice QI.
One of the initial joint decisions was the identification of the clinical domain of study for this intervention—mental health care. The factors considered in this decision included the limited number of mental health providers in rural primary care, the high prevalence of mental illness , and the providers’ current familiarity with and routine use (including at the point of care) of two patient-generated mental health measures—the Patient Health Questionnaire-9 (PHQ-9) depression measure [ ] and the Generalized Anxiety Disorder 7 item (GAD-7) [ ].
Co-design Stage 1
This stage involved coproducing the implementation and evaluation plans. The components are shown in. Methods for data collection included team mapping, care workflow, portal development, emails, and summary meeting notes. Outputs of this stage included mapping the care team’s roles and responsibilities, identifying relevant PCMs, detailing the office and clinical care workflows to enable PCM use, optimizing the use of the patient portal, and documenting learnings to understand issues and generate solutions.
Team-Based Care Mapping
Team roles and responsibilities were mapped using the team mapping method, a facilitated cocreation workshop designed to help groups explore how to work together in a primary care team  and informed by the circle of care modeling [ ], an analytical technique to develop the PCM adoption methods and create and validate patient use cases. A 2-hour–long team mapping session, using patient personas focused on mental health concerns, was held over Zoom (Zoom Video Communications) with all providers and staff from the clinical practice and members of the research team. The mapping session explored and defined team roles and responsibilities in caring for the simulated patients and how PCMs could support care decisions. This exercise also explored the use of technology, such as patient portals outside of regular visits to engage patients, and workflow associated with monitoring the data. Following the session, facilitators generated a summary report that included images of the patient-centered maps that were created, as well as summaries of the PCM gaps and potential solutions, categorized for discussion with the care team.
Discussions with providers led to the following selected PREM and PROMs: (1) a multipart question from the KB Patient Experience Survey that contained subitems about whether someone from the care team talked with patients about difficulties taking care of their health, main health goals and priorities, stressors, needed support, medication review, and offer of preventive care; (2) Patient-Reported Outcomes Measurement Information System Item Bank (version 1.0)—General Self-Efficacy Short Form 4a—a general population measure with demonstrated good convergent validity, internal consistency reliability, model fit, and sufficient unidimensionality [, ]; (3) PHQ-9 depression measure; (4) GAD-7; and (5) self-action plan for depression [ ]. The PHQ-9 and GAD-7 are validated measures already in use by the practice at the point of care.
A group session, held with the clinic after the team mapping exercise, focused on reviewing the mapping report and how PCM gaps could be addressed through alternative workflows . An electronic patient-reported outcomes toolkit [ ] guided discussions related to improvements in office efficiency and integration of electronic PGD. Iterations of care workflow were brought forward as a flow diagram for feedback at the next meeting. In consultation with patient partners and co-designed with the care providers and clinic staff, an initial workflow was identified for the implementation stage.
The research team selected a commercial vendor portal that had a patient-centered perspective (eg, patients could select who they want to share data within their care team). The web-based portal was used as a stand-alone system during the study (ie, was not interoperable with the clinic’s EMR system) as no current tethered or interoperable patient portal was available for use in this study, and the industry partner was amenable to adding selected PCMs and tailoring to study needs (eg, education and resources). Ongoing adjustments to the user interface were gathered and provided to the industry partner for implementation.
The portal had a user interface for patients and a separate one for the clinic staff and care providers. The researchers had access to both the interfaces. Of the 7 functionalities assessed in a recent Cochrane review on patient access to EMRs , this portal provided for PCM, including tracking, education, and a reminder feature that was sent if a requested patient questionnaire was not completed within 3 months. The portal did not provide the ability to request other information, bidirectional communication and sharing, or the ability of patients to manage their care. The materials provided in the portal included selected PCMs and educational resources.
Preparing for Implementation and Evaluation
Preparing for stage 2 encompassed patient recruitment, change management support, portal deployment, and development of the evaluation.
Recruitment of Patient Participants
Clinic staff and care providers contacted patients who had been living with a mental health diagnosis for many years to inform them about the study. Interested patients were provided with a flyer that included contact information for a researcher (MA) who provided study information and a consent form before enrollment. Four patients agreed to participate in the study. Participation involved using the patient portal over 4 months and interviews to discuss how PCM and educational materials within the portal were used during their care. Owing to the COVID-19 pandemic, most clinical visits between patients and providers were held over phone.
This stage comprised activities to operationalize the intervention and adapt design components based on iterative feedback. The components of this stage are shown in.
Patient Workflow and Portal Use
The use of the portal was presented to patients at the start of the intervention as a part of their workflow. Iterative patient workflow adaptations were made based on patient feedback during the implementation and evaluation stages.
One researcher (MA) provided training in portal use for all patients, clinic staff, and care providers. The patient user interface allowed patients to see messages from their provider, complete the PREM and PROMs, and explore educational materials. The portal was set up to send an email to inform patients that there was an invite in their portal account. Both patient and provider portal interfaces allowed completed PCMs to be viewed on the computer device screen or downloaded as a document that clinic staff could upload into the patient’s record in their EMR. Patients and providers could also view data trends of these measures (eg, scores on PHQ-9) if more than one was completed.
Care Workflow for Providers
The care workflow, finalized in the previous stage, was used as the initial care workflow for the study implementation. Individual discussions with providers and clinic staff during the intervention focused on changes to the care workflow specific to care pathways such as screening, monitoring, and follow-up resulting from the addition of PCM to team-based care processes. For example, the RN took responsibility for sending PROMs and educational materials to patients biweekly to complete through the portal. The PREM was sent at the beginning and end of the portal-use period.
We recognized that because of the lack of interoperability between the portal and EMR, we would have to simulate some embedded portal functions, such as sending reminders and data entry. For example, the portal-completed PROMs were intended to be reviewed during clinic visits. To achieve this, the RN and a researcher (MA) shared responsibility for checking for upcoming appointments, deploying the PROM and checking for completion and transferring the PROM score into the EMR encounter notes section, attaching the document, and entering the score into the existing PROM sections of the EMR. The researchers met periodically with the RN to adjust the implementation efforts.
Evaluation Plan—Stage 2
A general interview guide approach was used for the patients () and providers ( ). Patient interview questions included interactions with the portal, use of questionnaire results and educational materials by themselves and with the provider, and how other tools could be used to complement communication with the portal. Interviews were scheduled within 1-5 days after a virtual or phone visit with their provider. A total of 13 patient interviews with 4 patients were conducted, and the number of interviews with each patient was dependent on the number of care visits (2-4 visits) during the study period. Provider questions included visit information, use of patient questionnaires during the visit, whether educational materials prompted discussion, and whether the questionnaire results/educational materials influenced the visit or team interactions. An individual interview was conducted with all 4 care providers and 2 clinic staff members at a convenient time following a patient visit (timing varied between 1 week and 1 month). To gain immediate insights, a web-based survey was provided to physicians immediately after a patient visit ( ).
All interviews were conducted virtually (phone or video) by two researchers (SD and MA) and lasted 10 to 30 minutes. Both researchers had experience with qualitative research. Notes were taken by both interviewers and combined as summary notes. The summary interview notes were coded by one researcher (MA) using ATLAS.ti (ATLAS.ti Scientific Software Development GmbH), and the coding reports and original data were reviewed by 2 additional team members with experience in the analysis of qualitative data (MS and SD). The final interview coding reports were confirmed by the research team.
PCM Data Analysis
The framework method for the analysis of qualitative PCM data was used. The framework method applies a matrix structure to facilitate the recognition of patterns and has been used effectively under the leadership of experienced qualitative researchers . Data analyses involved looking across interview coding reports, team mapping reports, portal use, and care workflow diagrams to determine areas of convergence or divergence and develop a narrative of the evaluation of the implementation. Our reflective practice comprised the development of new care workflow diagrams, iterative writing, and discussion among the research team at monthly analysis meetings.
The collective results of stage 1 were used to inform stage 2.
Team Mapping to Inform Care Workflow
The team mapping exercise focused on two areas: care of patients with mental health concerns and how PCM could be used in practice. The session produced circle of care maps with defined team roles and responsibilities in caring for the simulated patients (). The session’s discussion focused on PCM integration and how PCM could influence care.
Following the mapping session, co-designing efforts produced an initial care workflow for incorporating PREM and PROMs into routine care, before, during, and in between a patient’s clinic visits, using a patient portal. Five key activities were identified: (1) deploying PCM, (2) collecting electronic patient data, (3) tracking completion, (4) reviewing data, and (5) documentation. Workflow changes were then designed to incorporate these activities, and a final workflow diagram was developed ([ ]). Before the visit, the patient or clinic staff may initiate a visit appointment, the care team may tailor resources and PCM questionnaires in the portal, the patient would receive notifications and review the resources, and then complete the questionnaires. The nurse would track and triage the PCM scores and initiate urgent responses if needed. During the virtual or in-person visit, the physician might review the PCM scores with the patient, document appropriate actions, and refer the patient to other providers as needed. In between visits, the care team would comanage the patient.
Strategies and Concerns About Integrating PCM
On the basis of team mapping and care workflow sessions, we constructedwith learnings and potential solutions to care gaps and team limitations, strategies needed to allow the use of PCM, constraints in the current workflow including restrictions because of the COVID-19 pandemic, and modifications needed to the patient portal to enhance care. Providers commented that the mapping and workflow sessions mirrored clinical practice well. The major concerns expressed included the practice’s limited capacity for incorporating PCM gathered outside the clinic visit (both electronic and paper-based), patient expectations for review of their data, and the provider’s knowledge and clinic capacity that limited engagement in QI. Providers expressed concerns about the rural context with lack of access to high-speed internet, insufficient resources to manage practice workload, and current fee-for-service structures that did not provide an allowance for the collection and use of PCM.
|Co-design step||Learnings||Potential solutions|
|Team roles and responsibilities|
|Strategies (what needs to be in place to use PGD)|
|Patient portal||There was concern regarding limited digital literacy|
aRN: registered nurse.
bQI: quality index.
cPGD: patient-generated data.
dEMR: electronic medical record.
Customization of the Patient Portal
The patient portal was collaboratively prepared with selected mental health questionnaires, patient experience measures, depression self-action planning tool, and educational resources, and comprised a feature for trending data such as the anxiety questionnaire, GAD-7, in. An appraisal guide was created to select the educational and resource material to ensure that the material was evidence-based, used patient-oriented language, and was adaptable to local contexts (eg, local community resources). Examples of the educational materials are provided in .
Before deployment, the portal underwent initial testing by patient partners and an undergraduate student who simulated the role of the patient. Through discussions with the practice and research team, the type, timing of deployment, quantity of educational material, messaging, and notifications within the portal were determined before launching portal use.
Implementation and Evaluation—Stage 2
Care Pathways and Portal Use
On the basis of workflow discussions with care providers and patient partners, a clinical care pathways diagram was produced ([ ]) to identify the opportunities for care team members to integrate PCM (eg, asynchronous screening or follow-up or during an encounter). There are 7 PCM-related clinical care activities performed by different care team members to screen, triage, assess, diagnose, treat, monitor, and follow up with the patient, which can be guided by their PCM data. Nurses and social workers may initiate screening, monitoring, and triage of patients, or when triggered by web-based notifications and reminders from the portal. Physicians may refer to PCM data when assessing, diagnosing, and treating the patient during an in-person or virtual visit. In between visits, the nurse could follow up with the patient to provide resources and PCM questionnaires within the portal, monitor for completed PCM data, and schedule notifications as reminders to complete them.
A patient workflow based on patient feedback was created to incorporate real and potential actions related to PCM ([ ]). PCM-related activities for patients are to review resources in the portal and evaluate their relevance, respond to notifications, complete PCM questionnaires indicated by the provider or on their own initiative, and take actions in response to new learnings. These actions include reflecting on their condition in response to new information, initiating self-care, tracking trends over time, sharing information with family or care network, reviewing additional resources, following up with the care team, and preparing for a visit. Considerations included delivery-method notification, the timing for completion of questionnaires and resources, tailoring efforts of educational materials, and visit type (in-person or virtual). A video was created [ ] to illustrate study results relating to the use and integration of PGD into clinical care from the patient’s perspective.
provides excerpts from the summary notes that demonstrate our key learnings from stage 2. The qualitative synthesis of 13 patient interviews indicated that the portal was easy to use and valued. Two patients noted the inability to use the portal during times of severe depression and fatigue. Providers saw benefits in patients’ use of the portal for information and to give them a sense that care was nearby. The major problems identified with provider use of the portal were lack of portal integration with the EMR and lack of alerts for changes in questionnaire scores that should trigger action. For providers, to limit the extra work across the team, 1 team member (RN) oversaw the PCM data in the portal and transferred it into the EMR. Even with the simulated integration of data transfer into the EMR by the RN, the process of reviewing those data elements in the EMR was often overlooked during a care visit.
All patients read at least some of the educational material provided, with skill-building workbooks being the most appreciated. One patient stated that “knowledge of our illness is vital to taking steps to live with illness and attain victory as often as possible.” Getting information over time rather than all at once was useful to avoid overwhelming the participant. Most patient participants had been living with a mental health diagnosis for many years and reported that the initial educational materials offered would have been helpful earlier in their diagnosis and illness. At their request for more advanced information, patient-friendly abstracts of scientific papers were added to the portal. Patients also wanted the ability to rate articles for their usefulness. Providers appreciated the potential for educational resources to be tailored to the patient’s stage and needs. However, they recognized that they would not have the resources or domain expertise needed to sustain ongoing educational material development or deployment in the portal.
|Use of patient portal (not used during visits)|
|Value of educational resources|
|Use of PCMs||Did not use within portal because of the following reasons:|
|Optimizing completion and use of PROMs|
aPROM: patient-reported outcome measure.
bPREM: patient-reported experience measure.
cEMR: electronic medical record.
PCM Use in Team-Based Mental Health Care
Three patients reported that the process of completing PROMs was extremely validating in that they learned more about themselves and gained confidence in reaching out for support. As one patient noted on reviewing the PHQ-9 with a resident physician, the experience was “unexpectedly rewarding and validating (same page together) and resulted in a treatment change.” Several patients noted that clinic visits in their current form are often used to revisit past appointments or begin with general questions that may not relate to a patient’s current status. As 1 patient, who was hesitant to talk with the physician about anxiety, noted, completing the questionnaire ensured that all pertinent aspects were covered, preventing communication disconnect. In addition, tracking scores were appreciated; 1 patient stated that the weekly trend “painted a picture of where I am.” However, 1 patient noted difficulty in interpreting depression scores in that they did not provide insight into “what created my responses.” Suggested additions by patients for PROMs included functional measures (eg, what is working now for or against you), more sophisticated graphics for tracking scores and more specific questions in the action plan based on depression scores. Patients wanted to be able to complete PROMs on their own and trend data over multiple years.
Patients expressed frustration that their providers did not review PROMs and did not ask about PCM at visits. One patient reported that she thought she was doing something wrong in not having questionnaire data at her visit. Providers noted that although PCM collected before a visit could make visits more efficient, they did not review them for reasons noted in.
As PGD in the portal was a new activity, there was also a lack of clarity when the PROM scores would be viewed and discussed. Patients were used for in-person visits and thus did not consider opening the portal during their virtual visit. Each provider had different preferences for where to view these data but had not established a practice of viewing them. The limited-time for visits meant that providers prioritized the purpose of the visit and planned to look at the PROM scores during the patient’s annual mental health check-up.
Concerning optimizing the use of PCMs, both patients and providers wanted to be able to set timely reminders for PROM completion. Patients sometimes logged into the portal to see an invite and start completing a PROM, and then realized that they wanted to return when they could give more focus or were less fatigued. The lack of reminders meant that patients often forgot to return and complete PROM. Similarly, providers wanted the ability to customize reminders (eg, 24 hours before a visit or a week after the initial invite). Providers also noted that they lacked education and training on PROMs’ limited use, including interpretation of a PCM score and what constituted a significant change.
Although QI activities were never completed, discussions were held to explore opportunities within workflow and care processes to carry it out. Examples of mental health care included PHQ-9 completion, counseling, and medication prescriptions for patients with a diagnosis of depression. The resultant QI process workflow is displayed in[ ]. PCM-related QI activities for the care team are based on the plan-do-study-act cycle and documentation processes that can be informed by PCM. Planning is driven by clinic priorities and capacities and is further guided by community health profiles and the latest clinical practice guidelines. Doing involves running patient panel queries using the clinic EMR and related external databases or applications to identify areas for attention. Studying, acting, and documentation involves a detailed review of the panel query outputs, implementing specific actions, and documenting the actions and results when available. The PCM focus brings awareness to specific groups of patients that may require action depending on their health situation and PCM scores.
To date, most QI efforts have been focused at the regional level, and the clinic does not know how to apply PREM results to clinical processes with multiple providers. An agreement was received around its value, but resistance was observed, and one provider commented on the lack of medical school training in QI. There was also a sense of technology overload, where providers had to remember multiple login codes and how to view data across more than 4 different systems. QI tools and resources available, delivery medium and type of PCM, timing and triggers for QI, and maintenance of patient anonymity when using PREMs were discussed.
Knowledge sharing (or IKT) throughout the study was invaluable to our process and learnings. Specifically, during all interactions and interviews with knowledge users, the researchers received productive feedback to questions like how we are doing and whether we hit the mark with iterations of (1) process descriptions; (2) design planning, implementation, and evaluation actions; and (3) documentation of identified issues and solutions.
Although providers noted that it takes a whole team to care for patients with mental health disorders, clinical resources were overwhelmed and there were limited community mental health services in this rural area. Multiple patients commented on how their physician was overworked, so they were mindful of what could be done at a visit, although they would have preferred mental health visits more frequently than once annually. The addition to the practice of the social worker was relatively recent, and one patient did not know about the availability of the RN or social worker or how they could fit into their care. Another patient stated that they wanted a physiotherapist to be part of their team. Both patients and providers noted that rural living was associated with fewer specialist resources and longer wait times.
Since the COVID-19 pandemic, virtual care visits have become a routine part of practice. One staff member noted that patients were no longer in the waiting room or available to complete questionnaires on-site. Both patients and providers felt that virtual care was less valuable because of lack of personal contact; one provider stated that “the loss of human connection is devastating.” One patient reported that they might have brought up their anxiety if face-to-face and that being virtual missed the seriousness of their illness. However, patients also preferred to answer PROMs at home, as the answers seemed more honest than completion in a public waiting room. One clinic staff member noted a possible benefit of phone visits in a rural (small population) clinic in that patients could have a degree of anonymity by not having a face to recognize when seen out in the community.
IKT offered the research a continuous and rich understanding of patient and provider needs and current challenges in the context of team-based care and PCM, as well as potential best practices for integrating PGD using a patient portal. Providers were unable to incorporate the use of electronically generated mental health PROMs within team-based care during this study for several reasons including workload, need to prioritize issues to address during an encounter, lack of easy access to the data, lack of value placed on the data, and lack of education about and practice in its use. The absence of established provincial standards for interoperability between systems hampered the integration of electronic PCMs in team-based care. In a review of personal health record functionalities and implementation, Harahap et al  identified interoperability as a key implementation issue, as well as security and privacy, usability, data quality, and personalization as other important factors. Digital health transformation at the system level is needed to realize interoperability, providing standard definitions for data exchange and cooperation with the patient, provider, and organizational systems. Using an ontological information model, Plastiras and O’Sullivan [ ] demonstrated the feasibility of transferring PGD using common standards from personal health records to EMRs. Interoperability must be addressed to ensure that electronic PROMs and PREMs from patient systems can be fully integrated within provider systems and readily available at the point of care. In addition to the issue of interoperability, the authors perceived that the limited experience of these newly formed primary care teams, and having to shift to virtual care during the pandemic made it less likely that any member could ensure delegation and use of the data provided.
Solutions to learnings from the co-design step that were acted upon (ie, engaging with the local Division of Family Practice around QI, patient reminders for the completion of questionnaires, research assistant tracking, and providing patient education and resources within the portal) were valued and demonstrative of the way forward. Ultimately, the gap in getting data viewed in the EMR and usable at the time of a visit was not accomplished. For digital tools to be successful in addressing provider workload, authors of a systematic review  identified a need for training, reducing documentation and task time, expanding the care team, and leveraging QI processes in workflows.
Although QI was an intended part of PCM advancement, it was discovered early in the study that providers felt that the practice could not engage in practice improvement using PREMs. An additional knowledge user with subject-matter expertise in QI was added to the team to present information on EMR panel report analysis, community profile, best practices, and clinical practice QI guidelines using PROMs. Although providers expressed interest in the future use of PCM for QI, because of the COVID-19 pandemic, frustrations expressed over prior poor quality QI data provided, and inexperience with use, the study shifted to provide an opportunity to explore QI and educate the practice on ways PCM could be used for QI. The QI activities were not implemented or evaluated.
PREMs are being collected through population surveys in the geographic area of this study, but QI efforts are still nascent, and there is no single best way to collect or use PCM for QI. Gleeson et al  systematically reviewed current QI efforts using PREMs and found that most practices attempted small, incremental changes to services that did not require a change in provider behavior and resulted in unclear impacts. They called for more attention to how PREM data can be used to inform practice changes that have a positive influence on the patient care experience. Translating new ideas into practice among early-career providers requires three considerations—credibility, practicality, and need [ ]. Efforts to incorporate QI into routine practice will likely require attention to messaging and creating digital solutions to address these issues. In addition, simply providing PGD, even if entered into the EMR, has the potential to increase provider burnout. A review of health record-integrated PCM found that technostress (technological complexity, uncertainty, overload, insecurity, and invasion [higher patient expectations]), time pressure, and workflow-related issues need to be addressed to accelerate the integration of PCM into clinical care [ ]. Therefore, future endeavors will need to consider the human and fiscal resources needed for QI in clinical practice and the integration of PGD into the digital health ecosystem.
To create primary care team-based practices that value and use PCM to improve care, a culture change is needed. This process, in the experience of one of the authors (MS) in teaching evidence-based medicine and research training, takes 3 to 5 years, with time for training and practice through targeted, small-scale projects. For example, the introduction of QI training of multidisciplinary teams in local health departments in North Carolina resulted in small but important changes in organizations’ cultures over 5 years, increasing engagement in future QI, and improving overall care and services . Mandating QI may be problematic, with a before-and-after study of Foundation Year 1 doctors in the United Kingdom reporting less motivation to complete QI projects and placing less importance on QI for their professional development despite a significant increase in overall QI knowledge at the end of the year [ ]. It remains to be seen if the BC support system approach to practice facilitation for QI is successful in this effort.
However, patients found the process of completing and tracking the results of mental health-related questionnaires on portal validation useful in capturing relevant aspects of their experiences, a unique feature of this study. The importance of portals as a communication tool and limited use of PGD was noted in an umbrella review , and lack of bidirectional communication was likely an additional barrier to PCM used in this study. Patients commented on how PROMs could provide a way to focus visits on what is of the greatest value for them at the moment. PCM may be particularly relevant for patients with illnesses for which there are no biomedical markers. For people living with mental health concerns, these measures can aid in communication when the illness makes it difficult to communicate the severity of their difficulties. Although evidence supports the use of patient medical home care models that include data-driven quality of care and patient engagement [ ], what seems to be missing is the value placed on PCM by providers and an understanding and appreciation of ways in which the use of these data can advance communication, care quality and QI. To achieve the patient-centered potential of PCM requires a conceptual shift in workflows, where patients and providers are encouraged to bring PGD, particularly around function, into care discussions.
As this was an exploratory study within a single clinic, the transferability of the findings to another context needs to be explored further. The study was carried out in a rural, early adopter patient medical home practice with a small care team and few patient participants, which may not be representative of other primary care models, larger clinics, or urban settings. As the study focused on mental health, the relevance of our findings to other health conditions is limited.
The researchers supported transferability by providing a detailed description of the context and location, and trustworthiness by being transparent about our methods for data collection and analysis. The study was conducted during the pandemic, which made recruitment of care teams and patients particularly challenging. We strove to address the limited sample size by adding richness to our data through multiple study methods and an iterative process of knowledge sharing between researchers, including patient partners and knowledge users. Being a transdisciplinary team of patient partners, providers, and researchers across multiple disciplines enabled us to bring in multiple perspectives during the analysis.
Limitations were introduced during the data collection and analysis. The persona image was selected to reflect how Dan may be dressed in an examination room. Upon reflection, we realize that Dan’s appearance would be quite different during a virtual visit and would have selected a persona image that does not reinforce power differentials or stereotyping . Although we did not record interviews, we used 2 interviewers at each session, and summative interview reports were iteratively compared with all PCM study data by the research team. Memos in the summary notes were used to record when the interviewers had different interpretations. Each researcher established their process for reflection, and we did not have a standard way to record and review individual reflections.
The value of PGD and the need for PCM methods to collect, integrate, and use PGD in team-based care remain challenging. Through collaborative and adaptive efforts, this gap was narrowed by demonstrating ways in which PGD can be incorporated into clinical practice within a Canadian team-based primary care setting. The conceptualization of PCM methods to accomplish this goal is well served by an IKT approach. IKT offers a beneficial technique for addressing our knowledge of users’ needs related to the collection, integration, and use of PGD, bridging the knowledge to action gap and setting the stage for future success.
The study was conducted during the COVID-19 pandemic when the world was impacted by changing health care resources. The authors want to extend their thanks to all the patient and provider participants who shared their insights throughout the study, to their collaborators at the Kootenay Boundary Division of Family Practice, and their industry partner, Cambian Business Services Inc.
Funding for this study was provided by the British Columbia Support for People and Patient-Oriented Research and Trials Unit the Methods section Cluster on Patient-centered Measurement (Patient-centered Measurement 003).
FL proposed the research and drafted the initial grant proposal. All authors have contributed to the revised and final grant proposal. SD led the engagement with the provider participants. MA led the engagement with patient participants and industry and academic partners. PB, MS, and SD tested and selected material for the portal, and MA integrated education materials and measures into the portal. SD and MA collected interview data. MA and MS performed the analysis of interview data and theme development with input from SD, FL, and PB. SD createdand - . SD and MS created the tables. SD and MS drafted the manuscript. All authors critically revised and approved the submitted manuscript and agreed to be accountable for ensuring its integrity.
Conflicts of Interest
A research team member was an industry partner (Cambian Business Services Inc) for this study, who provided data storage and inclusion of the survey tools and educational resources into the patient portal. The system was provided at no cost to the clinic for the duration of the study.
Interview guide and questions for patients.DOCX File , 28 KB
Interview guide and questions for providers.DOCX File , 26 KB
Questions for providers—web-based survey.DOCX File , 34 KB
Examples of educational resources made available on patient portal.DOCX File , 26 KB
- The patient centered medical home. Robert Graham Center. URL: https://www.aafp.org/dam/AAFP/documents/about_us/initiatives/PCMH.pdf [accessed 2020-11-05]
- John J, Jani H, Peters K, Agho K, Tannous W. The effectiveness of patient-centred medical home-based models of care versus standard primary care in chronic disease management: A systematic review and meta-analysis of randomised and non-randomised controlled trials. Int J Environ Res Public Health 2020;17(18):50. [CrossRef]
- Giusti A, Nkhoma K, Petrus R, Petersen I, Gwyther L, Farrant L, et al. The empirical evidence underpinning the concept and practice of person-centred care for serious illness: a systematic review. BMJ Glob Health 2020 Dec 10;5(12):e003330 [FREE Full text] [CrossRef] [Medline]
- Scholl I, Zill JM, Härter M, Dirmaier J. An integrative model of patient-centeredness - a systematic review and concept analysis. PLoS One 2014 Sep 17;9(9):e107828 [FREE Full text] [CrossRef] [Medline]
- System change - patient medical homes. General Practice Service Committee. 2020. URL: https://gpscbc.ca/what-we-do/system-change/patient-medical-homes [accessed 2021-05-01]
- BC Patient-Centred Measurement. URL: https://www.bcpcm.ca/ [accessed 2021-03-07]
- Hsiao C, Dymek C, Kim B, Russell B. Advancing the use of patient-reported outcomes in practice: understanding challenges, opportunities, and the potential of health information technology. Qual Life Res 2019 Jun 25;28(6):1575-1583. [CrossRef] [Medline]
- Reading MJ, Merrill JA. Converging and diverging needs between patients and providers who are collecting and using patient-generated health data: an integrative review. J Am Med Inform Assoc 2018 Jun 01;25(6):759-771 [FREE Full text] [CrossRef] [Medline]
- Cohen DJ, Keller SR, Hayes GR, Dorr DA, Ash JS, Sittig DF. Integrating patient-generated health data into clinical care settings or clinical decision-making: lessons learned from project healthdesign. JMIR Hum Factors 2016 Oct 19;3(2):e26 [FREE Full text] [CrossRef] [Medline]
- Lordon RJ, Mikles SP, Kneale L, Evans HL, Munson SA, Backonja U, et al. How patient-generated health data and patient-reported outcomes affect patient-clinician relationships: a systematic review. Health Informatics J 2020 Dec 20;26(4):2689-2706 [FREE Full text] [CrossRef] [Medline]
- West P, Van Kleek M, Giordano R, Weal M, Shadbolt N. Common barriers to the use of patient-generated data across clinical settings. In: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 2018 Presented at: CHI Conference on Human Factors in Computing Systems; April 21-26, 2018; Montreal, QC, Canada p. 1-13. [CrossRef]
- Chung AE, Basch EM. Incorporating the patient's voice into electronic health records through patient-reported outcomes as the "review of systems". J Am Med Inform Assoc 2015 Jul 21;22(4):914-916 [FREE Full text] [CrossRef] [Medline]
- Lavallee DC, Lee JR, Austin E, Bloch R, Lawrence SO, McCall D, et al. mHealth and patient generated health data: stakeholder perspectives on opportunities and barriers for transforming healthcare. Mhealth 2020;6:8 [FREE Full text] [CrossRef] [Medline]
- Gensheimer SG, Wu AW, Snyder CF, PRO-EHR Users’ Guide Steering Group. Oh, the places we'll go: patient-reported outcomes and electronic health records. Patient 2018 Dec;11(6):591-598. [CrossRef] [Medline]
- Stover AM, Haverman L, van Oers HA, Greenhalgh J, Potter CM, ISOQOL PROMs/PREMs in Clinical Practice Implementation Science Work Group. Using an implementation science approach to implement and evaluate patient-reported outcome measures (PROM) initiatives in routine care settings. Qual Life Res 2021 Nov 10;30(11):3015-3033 [FREE Full text] [CrossRef] [Medline]
- Manalili K, Santana MJ, ISOQOL PROMs/PREMs in Clinical Practice Implementation Science Work Group. Using implementation science to inform the integration of electronic patient-reported experience measures (ePREMs) into healthcare quality improvement: description of a theory-based application in primary care. Qual Life Res 2021 Nov 26;30(11):3073-3084. [CrossRef] [Medline]
- Graham ID, Kothari A, McCutcheon C, Integrated Knowledge Translation Research Network Project Leads. Moving knowledge into action for more effective practice, programmes and policy: protocol for a research programme on integrated knowledge translation. Implement Sci 2018 Feb 02;13(1):22 [FREE Full text] [CrossRef] [Medline]
- Gagliardi AR, Kothari A, Graham ID. Research agenda for integrated knowledge translation (IKT) in healthcare: what we know and do not yet know. J Epidemiol Community Health 2017 Feb 19;71(2):105-106 [FREE Full text] [CrossRef] [Medline]
- Kootenay Boundary Division of Family Practice. URL: https://divisionsbc.ca/kootenay-boundary [accessed 2019-10-18]
- Rural health services in BC: a policy framework to provide a system of quality care. Cross Sector Policy Discussion Paper - BC. 2015. URL: https://www.health.gov.bc.ca/library/publications/year/2015/rural-health-policy-paper.pdf [accessed 2021-12-07]
- Roberts KC, Rao DP, Bennett TL, Loukine L, Jayaraman GC. Prevalence and patterns of chronic disease multimorbidity and associated determinants in Canada. Health Promot Chronic Dis Prev Can 2015 Aug;35(6):87-94 [FREE Full text] [CrossRef] [Medline]
- Bosco CI, Oandasan I. Review of family medicine within rural and remote Canada: education, practice, and policy. College of Family Physicians of Canada, Mississauga, ON. 2016. URL: https://www.cfpc.ca/CFPC/media/Resources/Rural-Practice/ARFM_BackgroundPaper_Eng_WEB_FINAL.pdf [accessed 2021-12-03]
- Zhang X, Hailu B, Tabor D, Gold R, Sayre MH, Sim I, et al. Role of health information technology in addressing health disparities: patient, clinician, and system perspectives. Med Care 2019 Jun;57 Suppl 6 Suppl 2(6):115-120 [FREE Full text] [CrossRef] [Medline]
- BC community health data. Provincial Health Services Authority. URL: http://communityhealth.phsa.ca/ [accessed 2019-12-03]
- Fast facts about mental illness. Canadian Mental Health Association. URL: https://cmha.ca/brochure/fast-facts-about-mental-illness/ [accessed 2021-12-07]
- Pinho LG, Correia T, Lopes MJ, Fonseca C, Marques MD, Sampaio F, et al. Patient-centered care for people with depression and anxiety: an integrative review protocol. J Pers Med 2021 May 13;11(5):411 [FREE Full text] [CrossRef] [Medline]
- Canadian Community Health Survey 2019. Statistics Canada. URL: https://www150.statcan.gc.ca/n1/daily-quotidien/200806/dq200806a-eng.htm [accessed 2021-12-08]
- Gagliardi AR, Berta W, Kothari A, Boyko J, Urquhart R. Integrated knowledge translation (IKT) in health care: a scoping review. Implement Sci 2016 Mar 17;11(1):38 [FREE Full text] [CrossRef] [Medline]
- Practice support program. General Practice Service Committee. URL: https://gpscbc.ca/what-we-do/practice-supports/psp [accessed 2019-09-25]
- Canadian Primary Care Sentinel Surveillance Network (CPCSSN). URL: https://cpcssn.ca/ [accessed 2019-02-09]
- Kroenke K, Spitzer RL, Williams JB. The PHQ-9: validity of a brief depression severity measure. J Gen Intern Med 2001 Sep;16(9):606-613 [FREE Full text] [CrossRef] [Medline]
- Spitzer RL, Kroenke K, Williams JB, Löwe B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med 2006 May 22;166(10):1092-1097. [CrossRef] [Medline]
- Price M, Bellwood P, Hill TT, Fletcher S. Team mapping: a novel method to help community primary healthcare practices transition to team-based care. Healthc Q 2020 Jan;22(4):33-39. [CrossRef] [Medline]
- Price M. Circle of care modelling: an approach to assist in reasoning about healthcare change using a patient-centric system. BMC Health Serv Res 2016 Oct 04;16(1):546 [FREE Full text] [CrossRef] [Medline]
- Salsman JM, Schalet BD, Merluzzi TV, Park CL, Hahn EA, Snyder MA, et al. Calibration and initial validation of a general self-efficacy item bank and short form for the NIH PROMIS. Qual Life Res 2019 Sep 28;28(9):2513-2523 [FREE Full text] [CrossRef] [Medline]
- PROMIS Gender Self-Efficacy - short form. PROMIS Health Organization. 2017. URL: https://www.healthmeasures.net/explore-measurement-systems/promis [accessed 2020-12-03]
- Depression self-action plan. Adapted from Ted Amann, RN C. URL: https://www.aafp.org/dam/AAFP/documents/patient_care/nrn/depression-self-care-action-plan.pdf [accessed 2020-12-02]
- Unertl KM, Weinger MB, Johnson KB, Lorenzi NM. Describing and modeling workflow and information flow in chronic disease care. J Am Med Inform Assoc 2009;16(6):826-836 [FREE Full text] [CrossRef] [Medline]
- Lerouge C, Austin E, Lee J, Segal C, Sangameswaran S, Hartzler A. ePROs in clinical care?: guidelines and tools for health systems. CERTAIN. 2020. URL: http://epros.becertain.org/ [accessed 2020-09-07]
- Ammenwerth E, Neyer S, Hörbst A, Mueller G, Siebert U, Schnell-Inderst P. Adult patient access to electronic health records. Cochrane Database Syst Rev 2021 Feb 26;2(2):CD012707. [CrossRef] [Medline]
- Gale NK, Heath G, Cameron E, Rashid S, Redwood S. Using the framework method for the analysis of qualitative data in multi-disciplinary health research. BMC Med Res Methodol 2013;13:117 [FREE Full text] [CrossRef] [Medline]
- Lau F, Davis S, Antonio M. Team-based methods for patient-centred measurement. BC Patient-Centred Measurement Working Group (BCPCMWG) and the BC SUPPORT Unit. URL: https://onlineacademiccommunity.uvic.ca/patientcentredmeasure/ [accessed 2022-02-01]
- Davis S, Antonio M, Ritland L. Video: Uses and Integration of PGD Into Clinical Care - The Patient Perspective. 2021. URL: https://youtu.be/dREwtq_fkXA [accessed 2022-02-01]
- Harahap NC, Handayani PW, Hidayanto AN. Functionalities and issues in the implementation of personal health records: systematic review. J Med Internet Res 2021 Jul 21;23(7):e26236 [FREE Full text] [CrossRef] [Medline]
- Plastiras P, O'Sullivan D. Exchanging personal health data with electronic health records: a standardized information model for patient generated health data and observations of daily living. Int J Med Inform 2018 Dec;120:116-125. [CrossRef] [Medline]
- Gleeson H, Calderon A, Swami V, Deighton J, Wolpert M, Edbrooke-Childs J. Systematic review of approaches to using patient experience data for quality improvement in healthcare settings. BMJ Open 2016 Aug 16;6(8):e011907 [FREE Full text] [CrossRef] [Medline]
- Yelon S, Sheppard L, Sleight D, Ford J. Intention to transfer: how do autonomous professionals become motivated to use new ideas? Perform Improv Q 2008;17(2):103. [CrossRef]
- Ye J. The impact of electronic health record-integrated patient-generated health data on clinician burnout. J Am Med Inform Assoc 2021 Apr 23;28(5):1051-1056. [CrossRef] [Medline]
- Schaaf EB, Cornett A, Randolph G. Improving culture, one quality improvement project at a time. J Public Health Manag Pract 2018;24(1):57-62. [CrossRef] [Medline]
- Bartlett T, Hewertson E, Vassallo M. Culture change and mandating quality improvement. Clin Teach 2019 Feb 10;16(1):30-35. [CrossRef] [Medline]
- Antonio MG, Petrovskaya O, Lau F. The State of Evidence in Patient Portals: Umbrella Review. J Med Internet Res 2020 Nov 11;22(11):e23851. [CrossRef]
- Cutting K, Hedenborg E. Can personas speak? Biopolitics in design processes. In: Proceedings of the Companion Publication of the Designing Interactive Systems Conference. 2019 Presented at: DIS '19: Designing Interactive Systems Conference 2019; June 23 - 28, 2019; San Diego CA USA p. 153-157 URL: https://dl.acm.org/doi/abs/10.1145/3301019.3323911 [CrossRef]
|BC: British Columbia|
|EMR: electronic medical record|
|GAD-7: Generalized Anxiety Disorder 7-item|
|IKT: integrated knowledge translation|
|KB: Kootenay Boundary|
|PCM: patient-centered measurement|
|PGD: patient-generated data|
|PHQ-9: Patient Health Questionnaire-9|
|PREM: patient-reported experience measure|
|PROM: patient-reported outcome measure|
|QI: quality improvement|
|RN: registered nurse|
Edited by A Mavragani; submitted 24.09.21; peer-reviewed by M Menear, N Harahap, JH Koh; comments to author 18.11.21; revised version received 21.12.21; accepted 30.12.21; published 18.03.22Copyright
©Selena Davis, Marcy Antonio, Mindy Smith, Paul Burgener, Danielle C Lavallee, Morgan Price, Sarah C Fletcher, Francis Lau. Originally published in JMIR Formative Research (https://formative.jmir.org), 18.03.2022.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.