This stage involved coproducing the implementation and evaluation plans. The components are shown in Figure 1. Methods for data collection included team mapping, care workflow, portal development, emails, and summary meeting notes. Outputs of this stage included mapping the care team’s roles and responsibilities, identifying relevant PCMs, detailing the office and clinical care workflows to enable PCM use, optimizing the use of the patient portal, and documenting learnings to understand issues and generate solutions.

Team roles and responsibilities were mapped using the team mapping method, a facilitated cocreation workshop designed to help groups explore how to work together in a primary care team [33] and informed by the circle of care modeling [34], an analytical technique to develop the PCM adoption methods and create and validate patient use cases. A 2-hour–long team mapping session, using patient personas focused on mental health concerns, was held over Zoom (Zoom Video Communications) with all providers and staff from the clinical practice and members of the research team. The mapping session explored and defined team roles and responsibilities in caring for the simulated patients and how PCMs could support care decisions. This exercise also explored the use of technology, such as patient portals outside of regular visits to engage patients, and workflow associated with monitoring the data. Following the session, facilitators generated a summary report that included images of the patient-centered maps that were created, as well as summaries of the PCM gaps and potential solutions, categorized for discussion with the care team.

Discussions with providers led to the following selected PREM and PROMs: (1) a multipart question from the KB Patient Experience Survey that contained subitems about whether someone from the care team talked with patients about difficulties taking care of their health, main health goals and priorities, stressors, needed support, medication review, and offer of preventive care; (2) Patient-Reported Outcomes Measurement Information System Item Bank (version 1.0)—General Self-Efficacy Short Form 4a—a general population measure with demonstrated good convergent validity, internal consistency reliability, model fit, and sufficient unidimensionality [35,36]; (3) PHQ-9 depression measure; (4) GAD-7; and (5) self-action plan for depression [37]. The PHQ-9 and GAD-7 are validated measures already in use by the practice at the point of care.

A group session, held with the clinic after the team mapping exercise, focused on reviewing the mapping report and how PCM gaps could be addressed through alternative workflows [38]. An electronic patient-reported outcomes toolkit [39] guided discussions related to improvements in office efficiency and integration of electronic PGD. Iterations of care workflow were brought forward as a flow diagram for feedback at the next meeting. In consultation with patient partners and co-designed with the care providers and clinic staff, an initial workflow was identified for the implementation stage.

The research team selected a commercial vendor portal that had a patient-centered perspective (eg, patients could select who they want to share data within their care team). The web-based portal was used as a stand-alone system during the study (ie, was not interoperable with the clinic’s EMR system) as no current tethered or interoperable patient portal was available for use in this study, and the industry partner was amenable to adding selected PCMs and tailoring to study needs (eg, education and resources). Ongoing adjustments to the user interface were gathered and provided to the industry partner for implementation.

The portal had a user interface for patients and a separate one for the clinic staff and care providers. The researchers had access to both the interfaces. Of the 7 functionalities assessed in a recent Cochrane review on patient access to EMRs [40], this portal provided for PCM, including tracking, education, and a reminder feature that was sent if a requested patient questionnaire was not completed within 3 months. The portal did not provide the ability to request other information, bidirectional communication and sharing, or the ability of patients to manage their care. The materials provided in the portal included selected PCMs and educational resources.

The use of the portal was presented to patients at the start of the intervention as a part of their workflow. Iterative patient workflow adaptations were made based on patient feedback during the implementation and evaluation stages.

One researcher (MA) provided training in portal use for all patients, clinic staff, and care providers. The patient user interface allowed patients to see messages from their provider, complete the PREM and PROMs, and explore educational materials. The portal was set up to send an email to inform patients that there was an invite in their portal account. Both patient and provider portal interfaces allowed completed PCMs to be viewed on the computer device screen or downloaded as a document that clinic staff could upload into the patient’s record in their EMR. Patients and providers could also view data trends of these measures (eg, scores on PHQ-9) if more than one was completed.

The care workflow, finalized in the previous stage, was used as the initial care workflow for the study implementation. Individual discussions with providers and clinic staff during the intervention focused on changes to the care workflow specific to care pathways such as screening, monitoring, and follow-up resulting from the addition of PCM to team-based care processes. For example, the RN took responsibility for sending PROMs and educational materials to patients biweekly to complete through the portal. The PREM was sent at the beginning and end of the portal-use period.

We recognized that because of the lack of interoperability between the portal and EMR, we would have to simulate some embedded portal functions, such as sending reminders and data entry. For example, the portal-completed PROMs were intended to be reviewed during clinic visits. To achieve this, the RN and a researcher (MA) shared responsibility for checking for upcoming appointments, deploying the PROM and checking for completion and transferring the PROM score into the EMR encounter notes section, attaching the document, and entering the score into the existing PROM sections of the EMR. The researchers met periodically with the RN to adjust the implementation efforts.

A general interview guide approach was used for the patients (Multimedia Appendix 1) and providers (Multimedia Appendix 2). Patient interview questions included interactions with the portal, use of questionnaire results and educational materials by themselves and with the provider, and how other tools could be used to complement communication with the portal. Interviews were scheduled within 1-5 days after a virtual or phone visit with their provider. A total of 13 patient interviews with 4 patients were conducted, and the number of interviews with each patient was dependent on the number of care visits (2-4 visits) during the study period. Provider questions included visit information, use of patient questionnaires during the visit, whether educational materials prompted discussion, and whether the questionnaire results/educational materials influenced the visit or team interactions. An individual interview was conducted with all 4 care providers and 2 clinic staff members at a convenient time following a patient visit (timing varied between 1 week and 1 month). To gain immediate insights, a web-based survey was provided to physicians immediately after a patient visit (Multimedia Appendix 3).

All interviews were conducted virtually (phone or video) by two researchers (SD and MA) and lasted 10 to 30 minutes. Both researchers had experience with qualitative research. Notes were taken by both interviewers and combined as summary notes. The summary interview notes were coded by one researcher (MA) using ATLAS.ti (ATLAS.ti Scientific Software Development GmbH), and the coding reports and original data were reviewed by 2 additional team members with experience in the analysis of qualitative data (MS and SD). The final interview coding reports were confirmed by the research team.

The framework method for the analysis of qualitative PCM data was used. The framework method applies a matrix structure to facilitate the recognition of patterns and has been used effectively under the leadership of experienced qualitative researchers [41]. Data analyses involved looking across interview coding reports, team mapping reports, portal use, and care workflow diagrams to determine areas of convergence or divergence and develop a narrative of the evaluation of the implementation. Our reflective practice comprised the development of new care workflow diagrams, iterative writing, and discussion among the research team at monthly analysis meetings.

The team mapping exercise focused on two areas: care of patients with mental health concerns and how PCM could be used in practice. The session produced circle of care maps with defined team roles and responsibilities in caring for the simulated patients (Figure 2). The session’s discussion focused on PCM integration and how PCM could influence care.

Following the mapping session, co-designing efforts produced an initial care workflow for incorporating PREM and PROMs into routine care, before, during, and in between a patient’s clinic visits, using a patient portal. Five key activities were identified: (1) deploying PCM, (2) collecting electronic patient data, (3) tracking completion, (4) reviewing data, and (5) documentation. Workflow changes were then designed to incorporate these activities, and a final workflow diagram was developed (Figure 3 [42]). Before the visit, the patient or clinic staff may initiate a visit appointment, the care team may tailor resources and PCM questionnaires in the portal, the patient would receive notifications and review the resources, and then complete the questionnaires. The nurse would track and triage the PCM scores and initiate urgent responses if needed. During the virtual or in-person visit, the physician might review the PCM scores with the patient, document appropriate actions, and refer the patient to other providers as needed. In between visits, the care team would comanage the patient.

On the basis of team mapping and care workflow sessions, we constructed Table 1 with learnings and potential solutions to care gaps and team limitations, strategies needed to allow the use of PCM, constraints in the current workflow including restrictions because of the COVID-19 pandemic, and modifications needed to the patient portal to enhance care. Providers commented that the mapping and workflow sessions mirrored clinical practice well. The major concerns expressed included the practice’s limited capacity for incorporating PCM gathered outside the clinic visit (both electronic and paper-based), patient expectations for review of their data, and the provider’s knowledge and clinic capacity that limited engagement in QI. Providers expressed concerns about the rural context with lack of access to high-speed internet, insufficient resources to manage practice workload, and current fee-for-service structures that did not provide an allowance for the collection and use of PCM.

The patient portal was collaboratively prepared with selected mental health questionnaires, patient experience measures, depression self-action planning tool, and educational resources, and comprised a feature for trending data such as the anxiety questionnaire, GAD-7, in Figure 4. An appraisal guide was created to select the educational and resource material to ensure that the material was evidence-based, used patient-oriented language, and was adaptable to local contexts (eg, local community resources). Examples of the educational materials are provided in Multimedia Appendix 4.

Before deployment, the portal underwent initial testing by patient partners and an undergraduate student who simulated the role of the patient. Through discussions with the practice and research team, the type, timing of deployment, quantity of educational material, messaging, and notifications within the portal were determined before launching portal use.

On the basis of workflow discussions with care providers and patient partners, a clinical care pathways diagram was produced (Figure 5 [42]) to identify the opportunities for care team members to integrate PCM (eg, asynchronous screening or follow-up or during an encounter). There are 7 PCM-related clinical care activities performed by different care team members to screen, triage, assess, diagnose, treat, monitor, and follow up with the patient, which can be guided by their PCM data. Nurses and social workers may initiate screening, monitoring, and triage of patients, or when triggered by web-based notifications and reminders from the portal. Physicians may refer to PCM data when assessing, diagnosing, and treating the patient during an in-person or virtual visit. In between visits, the nurse could follow up with the patient to provide resources and PCM questionnaires within the portal, monitor for completed PCM data, and schedule notifications as reminders to complete them.

A patient workflow based on patient feedback was created to incorporate real and potential actions related to PCM (Figure 6 [42]). PCM-related activities for patients are to review resources in the portal and evaluate their relevance, respond to notifications, complete PCM questionnaires indicated by the provider or on their own initiative, and take actions in response to new learnings. These actions include reflecting on their condition in response to new information, initiating self-care, tracking trends over time, sharing information with family or care network, reviewing additional resources, following up with the care team, and preparing for a visit. Considerations included delivery-method notification, the timing for completion of questionnaires and resources, tailoring efforts of educational materials, and visit type (in-person or virtual). A video was created [43] to illustrate study results relating to the use and integration of PGD into clinical care from the patient’s perspective.

Table 2 provides excerpts from the summary notes that demonstrate our key learnings from stage 2. The qualitative synthesis of 13 patient interviews indicated that the portal was easy to use and valued. Two patients noted the inability to use the portal during times of severe depression and fatigue. Providers saw benefits in patients’ use of the portal for information and to give them a sense that care was nearby. The major problems identified with provider use of the portal were lack of portal integration with the EMR and lack of alerts for changes in questionnaire scores that should trigger action. For providers, to limit the extra work across the team, 1 team member (RN) oversaw the PCM data in the portal and transferred it into the EMR. Even with the simulated integration of data transfer into the EMR by the RN, the process of reviewing those data elements in the EMR was often overlooked during a care visit.

All patients read at least some of the educational material provided, with skill-building workbooks being the most appreciated. One patient stated that “knowledge of our illness is vital to taking steps to live with illness and attain victory as often as possible.” Getting information over time rather than all at once was useful to avoid overwhelming the participant. Most patient participants had been living with a mental health diagnosis for many years and reported that the initial educational materials offered would have been helpful earlier in their diagnosis and illness. At their request for more advanced information, patient-friendly abstracts of scientific papers were added to the portal. Patients also wanted the ability to rate articles for their usefulness. Providers appreciated the potential for educational resources to be tailored to the patient’s stage and needs. However, they recognized that they would not have the resources or domain expertise needed to sustain ongoing educational material development or deployment in the portal.

Three patients reported that the process of completing PROMs was extremely validating in that they learned more about themselves and gained confidence in reaching out for support. As one patient noted on reviewing the PHQ-9 with a resident physician, the experience was “unexpectedly rewarding and validating (same page together) and resulted in a treatment change.” Several patients noted that clinic visits in their current form are often used to revisit past appointments or begin with general questions that may not relate to a patient’s current status. As 1 patient, who was hesitant to talk with the physician about anxiety, noted, completing the questionnaire ensured that all pertinent aspects were covered, preventing communication disconnect. In addition, tracking scores were appreciated; 1 patient stated that the weekly trend “painted a picture of where I am.” However, 1 patient noted difficulty in interpreting depression scores in that they did not provide insight into “what created my responses.” Suggested additions by patients for PROMs included functional measures (eg, what is working now for or against you), more sophisticated graphics for tracking scores and more specific questions in the action plan based on depression scores. Patients wanted to be able to complete PROMs on their own and trend data over multiple years.

Patients expressed frustration that their providers did not review PROMs and did not ask about PCM at visits. One patient reported that she thought she was doing something wrong in not having questionnaire data at her visit. Providers noted that although PCM collected before a visit could make visits more efficient, they did not review them for reasons noted in Table 2.

As PGD in the portal was a new activity, there was also a lack of clarity when the PROM scores would be viewed and discussed. Patients were used for in-person visits and thus did not consider opening the portal during their virtual visit. Each provider had different preferences for where to view these data but had not established a practice of viewing them. The limited-time for visits meant that providers prioritized the purpose of the visit and planned to look at the PROM scores during the patient’s annual mental health check-up.

Concerning optimizing the use of PCMs, both patients and providers wanted to be able to set timely reminders for PROM completion. Patients sometimes logged into the portal to see an invite and start completing a PROM, and then realized that they wanted to return when they could give more focus or were less fatigued. The lack of reminders meant that patients often forgot to return and complete PROM. Similarly, providers wanted the ability to customize reminders (eg, 24 hours before a visit or a week after the initial invite). Providers also noted that they lacked education and training on PROMs’ limited use, including interpretation of a PCM score and what constituted a significant change.

Although QI activities were never completed, discussions were held to explore opportunities within workflow and care processes to carry it out. Examples of mental health care included PHQ-9 completion, counseling, and medication prescriptions for patients with a diagnosis of depression. The resultant QI process workflow is displayed in Figure 7 [42]. PCM-related QI activities for the care team are based on the plan-do-study-act cycle and documentation processes that can be informed by PCM. Planning is driven by clinic priorities and capacities and is further guided by community health profiles and the latest clinical practice guidelines. Doing involves running patient panel queries using the clinic EMR and related external databases or applications to identify areas for attention. Studying, acting, and documentation involves a detailed review of the panel query outputs, implementing specific actions, and documenting the actions and results when available. The PCM focus brings awareness to specific groups of patients that may require action depending on their health situation and PCM scores.

To date, most QI efforts have been focused at the regional level, and the clinic does not know how to apply PREM results to clinical processes with multiple providers. An agreement was received around its value, but resistance was observed, and one provider commented on the lack of medical school training in QI. There was also a sense of technology overload, where providers had to remember multiple login codes and how to view data across more than 4 different systems. QI tools and resources available, delivery medium and type of PCM, timing and triggers for QI, and maintenance of patient anonymity when using PREMs were discussed.

Knowledge sharing (or IKT) throughout the study was invaluable to our process and learnings. Specifically, during all interactions and interviews with knowledge users, the researchers received productive feedback to questions like how we are doing and whether we hit the mark with iterations of (1) process descriptions; (2) design planning, implementation, and evaluation actions; and (3) documentation of identified issues and solutions.

Although providers noted that it takes a whole team to care for patients with mental health disorders, clinical resources were overwhelmed and there were limited community mental health services in this rural area. Multiple patients commented on how their physician was overworked, so they were mindful of what could be done at a visit, although they would have preferred mental health visits more frequently than once annually. The addition to the practice of the social worker was relatively recent, and one patient did not know about the availability of the RN or social worker or how they could fit into their care. Another patient stated that they wanted a physiotherapist to be part of their team. Both patients and providers noted that rural living was associated with fewer specialist resources and longer wait times.

Since the COVID-19 pandemic, virtual care visits have become a routine part of practice. One staff member noted that patients were no longer in the waiting room or available to complete questionnaires on-site. Both patients and providers felt that virtual care was less valuable because of lack of personal contact; one provider stated that “the loss of human connection is devastating.” One patient reported that they might have brought up their anxiety if face-to-face and that being virtual missed the seriousness of their illness. However, patients also preferred to answer PROMs at home, as the answers seemed more honest than completion in a public waiting room. One clinic staff member noted a possible benefit of phone visits in a rural (small population) clinic in that patients could have a degree of anonymity by not having a face to recognize when seen out in the community.