Background: Patient-reported outcomes (PROs) are increasingly being used in the management of type 2 diabetes (T2D) to integrate data from patients’ perspective into clinical care. To date, the majority of PRO tools have lacked patient and provider involvement in their development, thus failing to meet the unique needs of end users, and lack the technical infrastructure to be integrated into the clinic workflow.
Objective: This study aims to apply a systematic, user-centered design approach to develop i-Matter (investigating a mobile health [mHealth] texting tool for embedding patient-reported data into diabetes management), a theory-driven, mobile PRO system for patients with T2D and their primary care providers.
Methods: i-Matter combines text messaging with dynamic data visualizations that can be integrated into electronic health records (EHRs) and personalized patient reports. To build i-Matter, we conducted semistructured group and individual interviews with patients with T2D and providers, a design thinking workshop to refine initial ideas and design the prototype, and user testing sessions of prototypes using a rapid-cycle design (ie, design-test-modify-retest).
Results: Using an iterative user-centered process resulted in the identification of 6 PRO messages that were relevant to patients and providers: medication adherence, dietary behaviors, physical activity, sleep quality, quality of life, and healthy living goals. In user testing, patients recommended improvements to the wording and timing of the PRO text messages to increase clarity and response rates. Patients also recommended including motivational text messages to help sustain engagement with the program. The personalized report was regarded as a key tool for diabetes self-management by patients and providers because it aided in the identification of longitudinal patterns in the PRO data, which increased patient awareness of their need to adopt healthier behaviors. Patients recommended adding individualized tips to the journal on how they can improve their behaviors. Providers preferred having a separate tab built into the EHR that included the personalized report and highlighted key trends in patients’ PRO data over the past 3 months.
Conclusions: PRO tools that capture patients’ well-being and the behavioral aspects of T2D management are important to patients and providers. A clinical trial will test the efficacy of i-Matter in 282 patients with uncontrolled T2D.
Trial Registration: ClinicalTrials.gov NCT03652389; https://clinicaltrials.gov/ct2/show/NCT03652389
Uncontrolled type 2 diabetes (T2D) is a significant public health problem in the United States, particularly among vulnerable populations (eg, low-income and racial and ethnic minorities) [, ]. Annually, T2D incurs about US $250 billion in health care costs and lost productivity, representing a significant social and economic burden [ ]. Despite recent improvements in the proportion of adults with T2D achieving hemoglobin A1c (HbA1c) targets <7%, only 50.9% achieved this level of control [ ]. The number of patients who fail to meet these goals is even higher in resource-limited primary care practices—a place where most vulnerable populations receive their care [ , ].
Recognizing the central role patients play in the management of T2D (eg, being aware of its signs and symptoms and engaging in daily self-care behaviors), several national and local organizations have forged initiatives to support the development and use of patient-reported outcomes (PROs) in the evaluation of health and well-being of patients with T2D [- ]. Measures of PROs are a standardized and quantifiable approach that allows for the collection and integration of data on patients’ perspective of their chronic disease into its clinical management [ ].
Much of the existing research that incorporates PROs in T2D has been limited to clinical drug trials examining patient tolerance to new treatments . The few practice-based studies conducted in T2D have used long batteries of PRO measures, and patients report PROs only on a single occasion, most often immediately before clinic visits [ , ]. Such reporting introduces a recall bias because patients are asked to approximate changes in their symptoms and behaviors over several months. To address these challenges, a growing number of studies are utilizing mobile health (mHealth) platforms that enable real-time data collection to facilitate patient self-monitoring outside the clinic environment, enhance patient engagement in their care, and inform provider decision making [ - ].
Systematic reviews of mHealth interventions in patients with T2D have demonstrated positive, short-term benefits on HbA1c levels and self-care behaviors [- ]. However, these studies have several methodological shortcomings that limit their impact, including small sample sizes (24-180 patients), short study duration (mean 24 weeks), low patient compliance, limited integration with clinical practice, and exclusion of vulnerable populations that would benefit most from mHealth interventions [ ]. More importantly, the PROs collected in the mHealth tools are researcher-driven and lack patient and provider involvement in the conceptualization of the intervention. As a result, the tools are not customized to address the complex and unique needs and preferences of patients and lack the technical infrastructure to support integration into the clinic workflow.
The i-Matter (investigating an mHealth texting tool for embedding patient-reported data into diabetes management) trial aims to address this gap in the literature by evaluating the efficacy of an innovative mobile PRO system that incorporates patients’ perspective of their disease into the management of T2D in primary care practices. The i-Matter intervention uses text messaging to capture patients’ self-reported PROs in real time, enhances patient engagement through data-driven feedback and motivational messages, and creates dynamic visualizations of the PROs that can be shared in personalized reports and integrated into the clinical workflow. A future randomized controlled trial (RCT) will evaluate the efficacy of the i-Matter intervention versus usual care on reduction in HbA1c and adherence to self-care behaviors at 12 months among 282 patients with uncontrolled T2D who receive care in resource-limited primary care practices. This paper discusses the iterative process of developing, integrating, and user testing the i-Matter intervention in the formative phase of the trial.
The i-Matter intervention is a blend of 2 frameworks: technology acceptance model (TAM) and capability, opportunity, and motivation model of behavior (COM-B). The TAM is based on the theory of reasoned action and asserts that perceptions of usefulness and ease of use directly influence the intention to use a new technology, leading in turn to its adoption . The TAM is considered a gold standard for characterizing the adoption and use of new health information technology [ , ]. COM-B is a parsimonious amalgamation of existing theories of behavior change [ ], which states that interaction among 3 key components is necessary for successful behavior change: the person needs to feel capable (ie, the ability to engage in necessary physical and thought processes) of changing, needs to have the opportunity (ie, social and environmental factors) to change, and needs to feel motivated (ie, confidence and self-efficacy) to change [ ]. The COM-B model has been proven effective for designing programs that help patients with T2D improve adherence to health behaviors [ , ]. The core components of the COM-B model are integrated into the design features of the i-Matter intervention to create a theoretically grounded technology solution ( ) [ - ].
|i-Mattera components||COM-Bb constructs||Mechanisms of action||Design features|
|Feedback messages (insights)||Data-driven insights based on PRO ratings, such as:|
|Motivational messages||Text messages that encourage journaling, such as:|
ai-Matter: investigating an mHealth texting tool for embedding patient-reported data into diabetes management.
bCOM-B: capability, opportunity, and motivation model of behavior.
cPRO: patient-reported outcome.
dEHR: electronic health record.
Overview of the Study Design
We used the evidence-based user-centered design (UCD) approach to conduct the formative phase of the trial [- ]. The aims of this phase were to (1) systematically gather and incorporate feedback from patients and providers to develop and refine the i-Matter intervention and (2) optimize the design of the personalized report for patients and providers [ , ]. The formative phase consisted of 3 steps: (1) focus groups and semistructured interviews to adapt i-Matter to diverse patient and provider needs, including those of Spanish-speaking patients; (2) a design workshop to understand a day in the life of patients with T2D and provider workflow processes to integrate i-Matter into clinical practice; and (3) user testing to evaluate the usability and acceptability of i-Matter in patients with T2D and optimize the tool’s performance and display of the personalized reports. The primary outcome of this phase was a refined, integrated, and well-tested mobile PRO system for T2D whose efficacy will be evaluated in the clinical trial.
Study Setting and Population
This study was conducted in a network of primary care practices of New York University Langone Health (NYULH). The practices comprised >1500 ambulatory physicians, nurse practitioners, and physician assistants who care for >800,000 patients in 235 facilities in New York City’s 5 boroughs: Long Island, New Jersey, Westchester County, Putnam County, and Dutchess County. The participating sites include academic practices, many community-based practices, and federally qualified health centers, serving an ethnically diverse population. All primary care practice sites share a single, integrated electronic health record (EHR; Epic).
The target enrollment for the formative phase was 36 patients and 14 providers. To be eligible, patients must (1) have had a diagnosis of T2D for ≥6 months; (2) have had uncontrolled T2D, defined as HbA1c >7%, documented in the EHR at least twice in the past year; (3) be fluent in English or Spanish; (4) be willing to send and receive text messages; and (5) be aged ≥18 years. Patients were excluded if they (1) refused or were unable to provide informed consent; (2) had acute renal failure, end-stage renal disease (ESRD) or evidence of dialysis, renal transplantation, or other ESRD-related services documented in the EHR; (3) were participating in another T2D study; (4) had significant psychiatric comorbidity or reports of substance abuse (as documented in the EHR); (5) were pregnant or planning to become pregnant within 12 months; or (6) planned to discontinue care at the practice within the next 12 months. Providers were eligible if they (1) were a primary care provider (ie, medical doctor, nurse practitioner) practicing at the participating practices and (2) provided care to at least five patients with T2D. The NYULH Institutional Review Board approved this study.
We used 2 approaches to recruit patients and providers into the formative phase. First, potentially eligible patients were identified through a review of the EHR using the diagnosis-related group codes indicating the presence of T2D and receiving care from a primary care provider at one of NYULH practices. After retrieving a list of potentially eligible patients, research assistants (RAs) reviewed patients’ EHR to determine if the patient met the eligibility criteria. Patients that met these criteria were contacted via telephone to confirm eligibility. During the telephone call, the RA gave patients a description of the study, including their role as participants in the study. If the patient remained interested in participating, they were given the option to either complete the focus groups or interviews in-person in a private room or via a remote session using the secure Webex conferencing platform. Providers were sent emails from the study principal investigators inviting them to provide feedback on the development of an interactive mHealth tool that could help enable patients with T2D to take a more active role in their diabetes management. All patients and providers provided written informed consent before participation in the study.
Development of the i-Matter Intervention
provides an overview of the UCD process used to develop the PROs for i-Matter. A description of each step is also included below.
|1. Adapt||Patient focus groups and provider interviews|
|2. Integrate||Design workshop|
aPRO: patient-reported outcome.
bi-Matter: investigating an mHealth texting tool for embedding patient-reported data into diabetes management.
cEHR: electronic health record.
Step 1. Focus Groups and Interviews to Adapt i-Matter to Diverse T2D Patients and Primary Care Physician Needs
The goal of the focus groups was to select the PROs that would be integrated into the i-Matter intervention as it relates to patients’ experiences living with T2D. A trained moderator used a semistructured guide to explore (1) patients’ daily experiences living with T2D, (2) the barriers and facilitators to achieving their diabetes-specific goals, (3) descriptions of patient-provider conversations about T2D and goals for HbA1c, and (4) interest in sharing PRO data with their provider to support treatment of T2D. A trained bilingual moderator also conducted separate focus groups with Spanish-speaking patients to inform the cultural and linguistic adaptation of i-Matter. Before starting each focus group, all patients completed questions about their comfort with using technology and their interest in using mHealth tools for diabetes care.
A trained moderator conducted semistructured individual interviews with primary care providers at the participating practices. The goal of the interviews was to elicit provider feedback on the clinical relevance of the PROs discussed in the patient focus groups for the management of T2D. The interview guide also explored (1) providers’ level of comfort with PRO data, (2) descriptions of patient-provider discussions about diabetes management, and (3) other important PROs not identified in the patient groups.
Results from the thematic analysis of the focus groups and interviews were used to develop a preliminary list of PROs for inclusion in i-Matter . The PROs were individual items taken from existing validated PRO measures that assess the impact of T2D and its treatments on patients’ psychosocial, physical, and behavioral functioning (eg, emotional distress, treatment and disease burden, adherence to medications, and lifestyle behaviors) [ ]. The measures included the Problem Areas in Diabetes Questionnaire, Diabetes Treatment Satisfaction Questionnaire, Treatment Related Impact Measure-Diabetes, Audit of Diabetes-Dependent Quality of Life, Diabetes Impact Management Scale, and Diabetes Distress Screening Scale [ - ]. General items from the National Institutes of Health Patient-Reported Outcome Measurement Information System Global 10 measure, which assesses patients’ physical, social, and emotional functioning [ ], were also included on the list of candidate PROs.
The study team then recontacted patients from the focus groups to get their feedback on the candidate list and have them rank the perceived importance of each PRO for management of T2D on a 1 (least important) to 6 (most important) scale. The study team used patients’ ratings in concert with the thematic analysis to narrow the list of PROs that would be presented to participants in the design workshop.
Step 2. Integrate i-Matter Into Provider Workflows and Patient Daily Lives
The design workshop comprised patients, providers, academic researchers with expertise in T2D and PROs; the digital health company Rip Road; and staff from the NYULH Medical Center Information Technology (MCIT) department. The design workshop used a UCD protocol adapted from the Agency for Healthcare Research and Quality  that sequentially led the group through a variety of activities (eg, story mapping and workflow or patient journey analysis) designed to further refine the i-Matter PRO content, stimulate ideas for the content and layout of the personalized report, discuss ideal workflow integration, and identify potential problems and opportunities to improve i-Matter for patients and providers.
Following steps 1 and 2, the study team collaborated with Rip Road to develop a prototype of i-Matter (ie, the beta version of the text message program and personalized report).
We wrote 2 variations of each PRO question to evaluate the wording and response formats that would yield the highest patient response rates and data quality. On the basis of our previous experiences and best practices for data collection via text message , all PRO questions were written so they require short, simple answer choices (eg, 0-10 rating or yes or no response), thereby minimizing the likelihood of missing and/or unanalyzable data that is common with open-ended (free text) response options. We also created 2 versions of the personalized report: a 1-month view and a 3-month view. All text messages and report content were translated to Spanish before user testing.
In addition to prototype development, we created decision rules that would drive the delivery of the text messages and report content. The rules, which were iteratively refined throughout the formative phase, outline the timing and order of the messages, the duration of time patients had to respond to each message (ie, response window), and the conditions that triggered specific motivational text messages and individualized insights displayed on the personalized report (). As shown in , patients receive 3 types of motivational text messages over the course of the study: (1) response-based, (2) activity-based, and (3) completion-based. The personalized report displays 2 types of insights: (1) correlational, which compares associations between 2 different PROs, and (2) individual, which identify trends in patients’ responses to the PROs over the past month (see for example messages).
Step 3. User Testing of the i-Matter Prototype
User testing was conducted in a purposive sample of patients drawn from the focus groups and those who were naive to the tool (ie, did not participate in previous steps). A rapid-cycle design (ie, design-test-modify-retest in short intervals of time) was used to allow for iterative refinement of the i-Matter prototype between each user test. Patients participated in the user testing sessions for 2 weeks, during which time they received and responded to the PRO questions sent via text message. At the end of the 2-week period, the study team sent patients a copy of the personalized report () and conducted an interview about their experiences. The interviews used a combination of think-aloud techniques and semistructured questions to collect patient feedback on the perceived ease of receiving and responding to the PRO questions; the clarity, timing, and frequency of the messages; and the perceived usefulness of systematically tracking the selected PROs for diabetes self-management. Patients also provided feedback on the personalized report, including the clarity of the data visualizations and data-driven feedback messages (herein called insights), content and layout of the report, and utility of the report for diabetes self-management. In addition to the interview, patients responded to questions derived from the TAM version 3 (TAM3) survey.
We also conducted interviews with providers to elicit their feedback on preferences for visual displays and placement of the report in the EHR and perceived barriers and facilitators to viewing the reports in clinical practice. The primary outcome of this step was the fully functional i-Matter intervention for testing in the RCT.
Participant demographics: a self-report instrument was used to collect patient sociodemographic data including gender, race or ethnicity, age, annual household income, education level, marital status, employment status, and current HbA1c level.
Patient use of mobile technology: before the focus groups, patients completed a survey created for this study that assessed the frequency of mobile phone use, capabilities of their mobile phones (eg, Wi-Fi connection, Bluetooth, and mobile data plan), the most commonly used functions (eg, text messaging, phone calls, email, and apps), comfort with using their mobile phone to manage T2D, interest in enrolling in a text messaging diabetes program, and challenges to using their mobile phone for diabetes self-management.
Use behavior: these data were extracted from the i-Matter platform at the end of the user testing sessions and included the following metrics (described in the analysis section): number of mobile phone inputs, time-on-task, task success, number of missed responses to PRO questions, and number of responses by patients outside the response window.
TAM3 survey: following the 2-week user testing period, patients completed questions derived from the well-validated TAM3 survey that assessed the perceived ease of use, usefulness, and quality of i-Matter; the likelihood of using i-Matter in the future and recommending it to others (ie, behavioral intention); and perceived benefits of discussing i-Matter data with providers to help manage their diabetes (ie, communication). The internal consistency of this scale ranged from 0.86 (communication) to 0.94 (perceived usefulness).
Sample size estimates for the formative phase were based on best practices for maximizing the information power of qualitative research, which recommends beginning with 6 to 8 participants per qualitative method and adding to the sample, as needed . As with previous studies, user testing sessions were scheduled until data saturation was reached [ ]. Our previous studies suggested that we would need 2 to 3 cycles of user testing to reach saturation [ ].
Focus groups and interviews were audiotaped, translated where necessary, and transcribed verbatim. Both data sources were analyzed using the constant comparative method, in which text was categorized into themes with the use of codes developed iteratively to reflect the data [, ]. The coding scheme was developed by the study investigators to focus on key themes identified both a priori (eg, from the interview protocols) and those that emerged during the interviews or focus group discussions. A trained qualitative researcher coded the transcripts independently, after which the research team met to discuss the coding and resolve any discrepancies.
After each round of user testing, the study team employed the best practices for instant data analysis of usability data for each PRO [, ]. Task success was calculated as the percentage of PRO questions that were answered correctly without errors. Time-on-task was calculated as the average amount of time in minutes and seconds that patients took to respond to each PRO question. Mobile phone inputs were calculated as raw counts of PRO questions sent by the i-Matter platform and the number of responses received by patients. Missing data were calculated as the percentage of PRO questions that had no response by patients, and late responses were calculated as the percentage of messages sent by patients that was outside a 1-hour window. In addition, we calculated frequencies for the TAM3 survey questions.
Following the analysis of use data, the research team categorized each issue with usability as either critical (abandon or remove), severe (significant delay or frustration in task completion requiring revision), or cosmetic (minor issue). Each of these issues were mapped onto the interview transcripts and survey responses to provide specific and detailed recommendations for refining i-Matter before proceeding to the next testing session.
Step 1. Patient Focus Groups, Provider Interviews, and Ranking of Candidate PROs
We invited 55 patients with T2D (22 male and 33 female) to participate in the focus groups, of which 35 (64%) declined participation, leaving 20 potential participants. Reasons for declining participation included being too busy, limitations owing to other comorbid conditions, personal or family constraints, and lack of interest in participating in the research. Of the 20 people who agreed to participate, 12 (60%) attended one of the focus groups, 1 did not attend owing to a scheduling conflict with work, and 7 stopped responding to the RA’s outreach calls. We held 4 focus groups: 2 for English-speaking patients (n=6) and 2 for Spanish-speaking patients (n=6).describes the sociodemographic characteristics of the focus group participants and their comfort with technology.
Analysis of the focus groups identified 4 core themes: (1) patients felt as though their lives were controlled by their blood sugar values; (2) patients’ greatest fear of having T2D were vision loss, kidney failure, or risk of amputation, and avoiding these consequences served as motivators for behavior change; (3) important goals for patients were being in control of their T2D, feeling well, living a long healthy life, and eventually not needing medications for T2D (owing to concerns about the negative long-term effects); and (4) forgetfulness, poor dietary adherence, physical inactivity, tiredness or fatigue, and poor emotional health were viewed as major barriers to keeping blood sugar in control. Patients in the Spanish-speaking focus groups also spoke about God being an important source of strength and motivation to improve their health.
We conducted 6 provider interviews (3/6, 50% female; 4 primary care providers, 1 endocrinologist, and 1 general surgeon and weight management specialist). Analysis of the interviews identified the central theme that providers want PRO data that are specific and actionable and can help them focus the clinic visit on what is most important for their T2D patients’ care. All providers felt that an asset of a program like i-Matter would be having patients systematically track data such as dietary intake and medication adherence that they cannot reliably assess within the time constraints of a clinic visit. All providers liked the idea of showing correlations between PROs being tracked in i-Matter and clinical data that are already stored in the EHR, such as HbA1c values. Providers varied on the importance of tracking patient functional status, quality of life, and psychosocial health, with two-thirds of the providers commenting that it was central to understanding patients’ behaviors, whereas the remaining one-third felt they were soft symptoms that may be important for the patient but not for clinical management.
Next, the study team selected individual items from existing PRO measures that best represented themes derived from the focus groups and interviews. This resulted in the selection of items representing 8 categories of PROs: diabetes-related symptoms, quality of life, emotional health (eg, depression, mood, and distress), treatment-related symptoms, treatment satisfaction, diabetes-related functional status, medication adherence, and lifestyle behaviors. Patient ranking of the items further reduced the number of PRO categories to 5: diabetes-related symptoms, quality of life, emotional health, medication adherence, and lifestyle behaviors ().
|Age (years), mean (SD)||62.5 (5.6)|
|HbA1ca, mean (SD)||7.95 (0.8)|
|Female, n (%)||8 (67)|
|Employed, n (%)||4 (33)|
|Retired, n (%)||4 (33)|
|Annual income <US $25,000, n (%)||7 (58)|
|Hispanic, n (%)||7 (58)|
|Race, n (%)|
|Education, n (%)|
|Less than high school||1 (8)|
|High school degree||4 (3)|
|Some college||2 (17)|
|College degree||5 (42)|
|Technology survey, n (%)|
|Currently uses text messaging||7 (58)|
|Has an unlimited text messaging plan||12 (100)|
|Always has mobile phone with them||9 (75)|
|Comfortable downloading apps on their mobile phone||7 (58)|
|Comfortable receiving and responding to text messages about T2Db||8 (67)|
|Interested in using mobile phone to help keep track of T2D||7 (58)|
|Challenges to tracking T2D on mobile phone, n (%)|
|Cost of receiving messages||2 (17)|
|Unreliable internet access||1 (8)|
|Do not use mobile phone regularly||3 (25)|
|Unsure of benefit||4 (33)|
|Concerns about privacy and security||2 (17)|
aHbA1c: hemoglobin A1c.
bT2D: type 2 diabetes.
|PROa categories||PRO statements||Mean importance score, range 1 (low) to 6 (high)|
|Symptom||Over the past week, did you experience tingling or prickling sensations in hands or feet owing to your diabetes?||1.8|
|Symptom||Over the past week, did you experience dry mouth owing to your diabetes?||3.0|
|Symptom||Over the past week, how often were you bothered by blurred vision?||3.6|
|Symptom||Over the past week, how would you rate your level of fatigue owing to your diabetes?||4.3|
|Symptom||Over the past week, how often did you experience increased thirst and frequent urination owing to your diabetes?||4.1|
|Emotional health||Over the past week, how often were you been bothered by emotional problems such as feeling anxious, depressed, or irritable owing to your diabetes?||4.2|
|Emotional health||How often over the past month, did you feel overwhelmed by the demands of living with diabetes?||2.75|
|Lifestyle behavior||On average, how many days did you participate in at least 30 min of physical activity over the past 7 days?||3.13b|
|Lifestyle behavior||Over the past week, how often did you eat (favorite unhealthy food)?||4.6|
|Lifestyle behavior||Over the past week, how often did you eat (favorite healthy food)?||2.25|
|Lifestyle behavior||How would you rate your sleep quality over the past 7 days?||4.8|
|Quality of life||I was able to keep my blood sugar in good control today.||4.6|
|Medication adherence||Over the past week, how often were you able to take your diabetes medication on time?||4.9|
|Medication adherence||How many days did you miss or skip at least one dose of your diabetes medication over the past 7 days?||2.9|
aPRO: patient-reported outcome.
bDespite the lower score, physical activity was added as a PRO after review of transcripts and notes from patient and provider interviews.
Step 2. Design Workshop
A total of 17 stakeholders participated in the design workshop. The following themes emerged when the group discussed the candidate list of PROs: (1) PROs should show variability in patients’ responses over time and be actionable by both patients and providers, (2) PROs should be taken from validated questionnaires to increase provider confidence in the data patients report and be comparable with HbA1c levels, (3) choosing fewer PROs would help increase patient response rates and reduce the burden on providers to view the data, (4) tracking PROs that focus on adherence to lifestyle behaviors were most appealing to patients, and (5) PRO content should be general (eg, “how are you feeling today?”) as opposed to diabetes-specific (“how much does diabetes interfere with your life?”). The group reasoned that questions that were too specific may not be relevant to all patients and could lead to disengagement or missing data. Alternatively, a broader question could be used as a way to show care for patients’ overall well-being and as an entry point for more diabetes-specific questions that may uncover new or different concerns the provider should be aware of.
On the basis of these discussions, the group generated several ideas for potential visualizations of PRO data. These included defining a threshold that patients’ data can fall above or below and depicting it in a way that makes it easily detectible and actionable, using bar graphs to show directionality, including icons or coloring schemes in addition to PRO labels that enhance the readability of the report, and including summary data in percentages or raw numbers to show the patient’s progress over time.
Applying the findings from steps 1 and 2, the study team reduced the number of PRO categories to 4. Diabetes-related fatigue (symptom category) was removed from the list because providers viewed it as too nebulous and not actionable, whereas patients felt sleep quality was a more meaningful PRO for their diabetes management. In addition, physical activity was added to the lifestyle category because many patients felt that physical inactivity was a major contributor to weight gain and poor diabetes control.
Step 3. User Testing of the i-Matter Prototype
Patients: Text Messages
We completed 2 rounds of user testing with patients: 7 patients completed the first round of testing (1 Spanish-speaking), and 3 patients completed the second round.presents the results of the use behavior data for both rounds of user testing. The i-Matter platform sent 325 messages, and patients sent 256 responses (78.7%). The most common reason for invalid messages was the response being sent in the wrong format (eg, sending free-text responses instead of a numerical response). The most common reasons for missing messages included problems with message filtering by the mobile carrier (which has been resolved by changing to short code messages), being busy during the response window, and not having their phone during the time the messages were sent. For the Spanish-speaking patient, the average response rate was 67.3 min (range 0-209.1 min). Overall, 59.7% (153/256) of the messages were answered within an hour, of which all (256/256, 100%) were answered within 1 min.
In the second round of user testing, the message protocol was modified to address the suboptimal percentage of missed responses. For example, to address the wide range of response times seen in the first round of testing (range 0-661.6 min), we restricted the patients’ ability to respond to the morning and evening PRO questions to a 1-hour window (based on the median response time). Overall, the i-Matter platform sent 222 messages and received 188 responses (84.6%) from patients. The most frequently missed message was sleep quality (77/188, 40.9% of missed messages). The most common reason for an invalid message was the patient responding to a question outside the 1-hour response window.
In qualitative interviews, patients in both rounds of user testing described the program as easy to use, not intrusive to their daily life, and helpful for managing their T2D. Similar findings were seen in the TAM3 survey responses (). Patients also liked the consistency in message timing because it helped them build a habit to respond (“it becomes second nature”). Several patients commented that they felt as though a person was sending the messages to check up on them. Patients also felt that the number of messages sent was adequate, with 2 people commenting, “No number is too many because they want to get better.” There were no differences in qualitative feedback or TAM3 responses between the English- and Spanish-speaking patients.
|User behaviors||User testing round 1 (n=7)||User testing round 2 (n=3)|
|Time-on-task||44 min (range 0-661.6)||20 min (range 0.08-30)|
|Task success (messages), n (%)||232 (90.6)||175 (93.1)|
|Missed responses, n (%)||100 (39.2)||28 (15.0)|
|Late responses, n (%)||49 (19.3)||14 (7.5)|
|Invalid responses, n (%)||24 (9.4)||13 (6.9)|
|Questions||Proportion of patients agreeing with statement, n (%)|
|I would definitely use the i-Matter program in the future||5 (71)|
|The PRO questions are very helpful for managing T2Db||6 (86)|
|Receiving and responding to PRO questions was easy||7 (100)|
|I responded to the PRO questions all the time||5 (71)|
|I would recommend i-Matter to friends and family||7 (100)|
|My provider would be more effective managing T2D with my PRO data||5 (71)|
|Overall, the i-Matter program is great or excellent||6 (86)|
|Personalized report (n=9)|
|I would definitely use the personalized report in the future||8 (89)|
|The personalized report is very helpful for managing T2D||7 (78)|
|The personalized report is easy to use||5 (56)|
|I would share the personalized report with friends or family||5 (56)|
|Showing my provider the personalized journal would help make clinic visits more effective||7 (78)|
|The charts and images are great||6 (67)|
|Overall, the personalized report is great||6 (67)|
aPRO: patient-reported outcome.
bT2D: type 2 diabetes.
As shown in, patients recommended improvements to the wording and timing of several of the PROs (eg, sending the sleep message at 9 AM rather than 7 AM), which is reflective of the use data. Patients also recommended including motivational messages to help sustain engagement with the program. After examining the data, the study team decided to remove the emotional health PRO (ie, labeled as critical). This was owing to the lack of variability in the use data (206/256, 80.6% of responses were 0-1 on a 10-point scale) and feedback from patients in the interviews that the PRO was not relevant to the management of their T2D.
|PROa categories and original messages||Original timing||Revised message||Revised timing|
|Have you taken all of your diabetes medications as prescribed today?||Daily at 7 AM||Retain as is||Allow patients to decide if they want the message in the AM or PM, or both (11 AM and 9 PM)|
|Reply with 1-4 to track ONE healthy living goal:||Daily at 7 AM||Retain for all patients. Separate less carbs and sweets to 2 separate goals||Changes so patients choose healthy goal at baseline visit (with option to change goal every 3 months)|
|How successful were you in achieving your goal to (custom text healthy goal) yesterday? Response: 0 (not at all) to 10 (very successful)||Daily at 2 PM||How successful were you in achieving your goal to (custom text healthy goal) this past week?b||Change timing to weekly at 2 PM|
|In general, how healthy your overall diet was today?||Daily at 7 AM||Retain message but change timing to assessing overall diet yesterdayb||Change to daily at 10 AM|
|Rate your sleep quality last night. Think how easily you fell asleep, how often you woke up and if sleep was refreshing. Response: 0 (poor) to 10 (excellent)||Daily at 7 AM||Reply with the number that best describes how well you slept last night||Change to daily at 9 AM|
|How many days in the past week did you do any physical activities like brisk walking where you breathed harder than normal?||Weekly at noon||Other than your regular job, did you do any physical activities like brisk walking for at least 30 min today?||Change to daily at 8 PM|
|Diabetes quality of life|
|Reply with the number that best describes how much control you felt you had over your diabetes over the past 2 weeks||Biweekly at noon||Retain as is change timing to weekly||Change to weekly at 3 PM|
|Reply with the number that best describes how irritable or moody you felt today owing to your diabetes||Daily at 7 PM||Remove||N/Ac|
aPRO: patient-reported outcome.
bText in italics show the changes made to the PRO timing across user testing sessions.
cN/A: not applicable.
Patients: Personalized Reports
A total of 9 patients provided feedback on the 1-month and 3-month versions of the personalized report: 4 of these patients participated in the user testing (of which 2 were recruited from the focus groups), whereas 5 were naive to the program. Overall, the majority of patients (8/9, 89%) felt the report was easy to read, eye-catching, and comprehensive. There was a strong preference for the 1-month version of the report owing to the larger font size. Patients also felt that receiving the report more frequently would help sustain motivation. Patients preferred layouts that used darker fonts and lighter background colors to help make the text easier to read. All patients viewed the color-coded schema favorably because it helped draw attention to the most important aspects of the report and made the data easy to interpret.
Several patients had difficulty reading the bar graphs of PROs that were collected biweekly (eg, quality of life) and recommended changing the items to weekly measures to be consistent with other PROs. Finally, email was the preferred delivery method, and most patients would share the report with their family and friends ().
Benefits of using the personalized report for diabetes self-management included being able to analyze how well one is adhering to recommended diabetes behaviors (“being honest with yourself”), providing visual cues to take responsibility for one’s health (“a visual reminder of things I need to do but don’t do and how I can be better”), and providing support to stay on track to be successful with diabetes. In the first round of user testing, 3 critiques of the report included being of greater use to providers than patients, concerns about confidentiality, and being too limited because it did not include tips on how patients can improve unhealthy behaviors. From this feedback, we incorporated motivational text messages and insights into the i-Matter prototype and created a study website that included additional resources. Patients in the second round of testing regarded the inclusion of insight messages as a source of motivation to change their behaviors and to continue responding to text messages to monitor changes in behavior over time.
Overall, all (n=6) providers thought the report was a good tool to help patients manage their T2D. Similar to patients, they felt that the insight messages were helpful for interpreting the data and prompting behavioral changes. When reviewing the PRO content, providers cautioned that before starting the program, patients would need to be educated on the recommended dietary and physical activity guidelines for diabetic patients and the medications they are currently taking for their T2D to ensure they are reliably answering the questions. On the basis of this feedback, at the baseline visit, trained study staff provide a brief overview of evidence-based guidelines for healthy eating and physical activity for T2D using low-literacy and language-congruent patient education handouts from the American Diabetes Association and review the patient’s current diabetes medication regimen.
To integrate the report into clinical practice, providers preferred having a separate tab built into the EHR, which included a summary of the personalized report and highlighted key trends in patients’ PRO data over the past 3 months. All providers found value in discussing the report with patients during the clinic visit because the data complemented the questions that they had already asked about diabetes self-management. Finally, although they found value in the longitudinal trends displayed in the graphs, owing to time constraints, they felt that patients should bring up anything important that stood out in the detailed view. On the basis of this feedback, the study team is working in collaboration with NYULH MCIT to integrate the personalized report into Epic. This includes the development of security protocols that will link patients’ encrypted research ID to their medical record number and integrate the report image into an Epic web integration record. Web integration records are used to visually integrate external apps with Epic. Providers will be able to access the i-Matter report via a button located within the patient’s chart at the top of the Office Visit toolbar (). The button will only be visible for patients randomized to the intervention arm.
Although achieving glycemic control is of clinical importance, it is the daily experiences of living with T2D that drive patients’ decisions to adhere to treatment recommendations and become engaged in their care . Even with the most efficacious treatments, failure to incorporate patients’ perspective of their disease into clinical decision making will make achieving the outcomes desired by patients and providers unattainable. The i-Matter trial will assess whether a theory-driven mobile PRO system that incorporates a set of PROs that are meaningful to both patients and providers can lead to reductions in HbA1c and improvements in patient adherence to self-care behaviors. Unlike existing programs, i-Matter is designed to collect real-time PRO data in the form of data-driven feedback, motivational messages, and dynamic data visualizations that are displayed in personalized reports for patients and providers.
This paper describes the design and refinement of i-Matter through an iterative user-centered approach that actively involved patients and providers throughout the process. Active involvement of end users in the development of the intervention can help to address the difficulties with protocol compliance, lack of clinical integration in the EHR, and provider skepticism about the utility of PROs in practice, which are hallmarks of previous trials, thus increasing the likelihood of developing a sustainable approach . Findings from our formative phase resulted in several insights regarding issues with the design, usability, and workflow of i-Matter, which led to key changes in the content and delivery of the text messages and personalized report and the technical infrastructure to support the integration of i-Matter into the EHR to improve patient and provider acceptability. In addition to evaluating the clinical benefit of i-Matter, the RCT will provide much needed evidence on the conditions under which mHealth interventions work in primary care settings and in patients’ daily lives and the organizational, individual, and technical factors that are required to support their use.
Although there are many strengths of our intervention approach, we note limitations that can be considered for future research. First, although our intervention is designed to target patients with T2D, it is more common for patients to have 2 or more chronic diseases (ie, multimorbidity) than 1 disease in isolation (89.3% vs 8.5%, respectively) . In fact, recent research demonstrates the negative impact of multimorbidity on PROs such as quality life, psychosocial health, self-efficacy, physical function, and self-management behaviors (eg, physical activity and medication adherence) [ ]. Thus, future research should examine whether adapting i-Matter for a multimorbid population would improve the integration and coordination of patient and provider management of co-occurring chronic diseases rather than using a single disease focus that can cause inefficiencies and fragmentation in care. Second, we did not perform psychometric testing of the final PROs before they were deployed in our intervention. We will use data collected in this study to assess the psychometric properties of our PRO questions and test their validity in subsequent research.
Finally, 2 (out of the 6) providers interviewed during the development of i-Matter indicated that they found less value in PROs that were not immediately actionable in primary care practice (eg, depression and quality of life). A key strength of the i-Matter study is the full EHR integration of the PROs with the health care team. Many previous PRO initiatives share the patient PRO data back with the providers in a workflow disruptive manner—asking providers to change their normal activities and make a special effort to review the PRO data. i-Matter overcomes these challenges by delivering the patient PRO data directly into the patient’s chart in the EHR—presented as just another commonly viewed data visualization by the provider such as patients’ lab and test results. Thus, our intervention will test the hypothesis that if actionable diabetes PRO data are delivered in the right context, it will influence patient-provider interactions. Early adopters of our intervention will also help to provide important data on the potential effectiveness and (time) efficiency of using PROs in clinical care. Sharing the outcomes of this work could provide providers who are hesitant to adopt such innovations with much needed information about the benefits of using these tools.
The authors would like to thank Aditya Verma and Sara Chokshi, DrPH, for their assistance with this project. This work was supported by a grant from Merck & Co, Inc, (principal investigator: AS) and the Agency for Healthcare Research and Quality, R01HS026522 (principal investigators: AS and DM).
AS analyzed, interpreted data, and drafted the manuscript. JC, LP, KL and MR acquired data and critically reviewed the manuscript. CJ and JG oversaw the EHR integration and critically reviewed the manuscript. MP, SP, and EL developed the mHealth platform and contributed to all critical revisions of the manuscript. DM interpreted data and critically reviewed the manuscript.
Conflicts of Interest
DM, JC, LP, MR, KL, CJ, and JG have no competing interests or financial disclosures to declare. AS is a consultant for Rip Road, Inc. MP, SP, and EL were paid as consultants to develop the mHealth intervention for this project.
Example motivational and feedback messages.DOCX File , 16 KB
- Fox CS, Golden SH, Anderson C, Bray GA, Burke LE, de Boer IH, American Heart Association Diabetes Committee of the Council on Lifestyle and Cardiometabolic Health, Council on Clinical Cardiology‚ Council on Cardiovascular and Stroke Nursing‚ Council on Cardiovascular SurgeryAnesthesia‚ Council on Quality of Care and Outcomes Research, American Diabetes Association. Update on prevention of cardiovascular disease in adults with type 2 diabetes mellitus in light of recent evidence: a scientific statement from the American heart association and the American diabetes association. Diabetes Care 2015 Sep;38(9):1777-1803 [FREE Full text] [CrossRef] [Medline]
- Geiss LS, Wang J, Cheng YJ, Thompson TJ, Barker L, Li Y, et al. Prevalence and incidence trends for diagnosed diabetes among adults aged 20 to 79 years, United States, 1980-2012. J Am Med Assoc 2014 Sep 24;312(12):1218-1226. [CrossRef] [Medline]
- National Diabetes Statistics Report: Estimates of Diabetes and Its Burden in the United States. Centers for Disease Control and Prevention. 2014. URL: https://www.cdc.gov/diabetes/data/statistics-report/index.html?CDC_AA_refVal=https%3A%2F%2Fwww.cdc.gov%2Fdiabetes%2Fdata%2Fstatistics%2Fstatistics-report.html [accessed 2020-08-12]
- Carls G, Huynh J, Tuttle E, Yee J, Edelman SV. Achievement of glycated hemoglobin goals in the US remains unchanged through 2014. Diabetes Ther 2017 Aug;8(4):863-873 [FREE Full text] [CrossRef] [Medline]
- Boltri JM, Okosun IS, Davis-Smith M, Vogel RL. Hemoglobin A1c levels in diagnosed and undiagnosed black, Hispanic, and white persons with diabetes: results from NHANES 1999-2000. Ethn Dis 2005;15(4):562-567. [Medline]
- Ali MK, McKeever Bullard K, Imperatore G, Barker L, Gregg EW, Centers for Disease Control and Prevention (CDC). Characteristics associated with poor glycemic control among adults with self-reported diagnosed diabetes--national health and nutrition examination survey, United States, 2007-2010. MMWR Suppl 2012 Jun 15;61(2):32-37. [Medline]
- Cella D, Yount S, Rothrock N, Gershon R, Cook K, Reeve B, PROMIS Cooperative Group. The patient-reported outcomes measurement information system (PROMIS): progress of an NIH roadmap cooperative group during its first two years. Med Care 2007 May;45(5 Suppl 1):S3-11 [FREE Full text] [CrossRef] [Medline]
- Snyder CF, Jensen RE, Segal JB, Wu AW. Patient-reported outcomes (PROs): putting the patient perspective in patient-centered outcomes research. Med Care 2013 Aug;51(8 Suppl 3):S73-S79 [FREE Full text] [CrossRef] [Medline]
- Speight J, Barendse SM. FDA guidance on patient reported outcomes. Br Med J 2010 Jun 21;340:c2921. [CrossRef] [Medline]
- Funnell MM, Brown TL, Childs BP, Haas LB, Hosey GM, Jensen B, et al. National standards for diabetes self-management education. Diabetes Care 2007 Jun;30(6):1630-1637. [CrossRef] [Medline]
- Cho JH, Ha SJ, Kao LR, Megraw TL, Chae C. A novel DNA-binding protein bound to the mitochondrial inner membrane restores the null mutation of mitochondrial histone Abf2p in Saccharomyces cerevisiae. Mol Cell Biol 1998 Oct;18(10):5712-5723 [FREE Full text] [CrossRef] [Medline]
- Deutsch S, Smith L, Gage B, Kelleher C, Garfinkel D. Patient-Reported Outcomes in Performance Measurement. New York, USA: RTI Press; 2013.
- Barsdorf A, Rubinstein E, Jaksa A. Patient-reported outcomes (Pros) in diabetes clinical trials. Value Health 2013 May;16(3):A168-A169. [CrossRef]
- Ridgeway JL, Beebe TJ, Chute CG, Eton DT, Hart LA, Frost MH, et al. A brief patient-reported outcomes quality of life (PROQOL) instrument to improve patient care. PLoS Med 2013 Nov;10(11):e1001548 [FREE Full text] [CrossRef] [Medline]
- Marshall S, Haywood K, Fitzpatrick R. Impact of patient-reported outcome measures on routine practice: a structured review. J Eval Clin Pract 2006 Oct;12(5):559-568. [CrossRef] [Medline]
- Vieta A, Badia X, Sacristán JA. A systematic review of patient-reported and economic outcomes: value to stakeholders in the decision-making process in patients with type 2 diabetes mellitus. Clin Ther 2011 Sep;33(9):1225-1245. [CrossRef] [Medline]
- Peters M, Crocker H, Jenkinson C, Doll H, Fitzpatrick R. The routine collection of patient-reported outcome measures (PROMs) for long-term conditions in primary care: a cohort survey. BMJ Open 2014 Feb 21;4(2):e003968 [FREE Full text] [CrossRef] [Medline]
- Valderas JM, Kotzeva A, Espallargues M, Guyatt G, Ferrans CE, Halyard MY, et al. The impact of measuring patient-reported outcomes in clinical practice: a systematic review of the literature. Qual Life Res 2008 Mar;17(2):179-193. [CrossRef] [Medline]
- Maddigan SL, Majumdar SR, Guirguis LM, Lewanczuk RZ, Lee TK, Toth EL, et al. Improvements in patient-reported outcomes associated with an intervention to enhance quality of care for rural patients with type 2 diabetes: results of a controlled trial. Diabetes Care 2004 Jun;27(6):1306-1312. [CrossRef] [Medline]
- Baron JS, Hirani SP, Newman SP. Investigating the behavioural effects of a mobile-phone based home telehealth intervention in people with insulin-requiring diabetes: results of a randomized controlled trial with patient interviews. J Telemed Telecare 2017 Jun;23(5):503-512. [CrossRef] [Medline]
- Årsand E, Frøisland DH, Skrøvseth SO, Chomutare T, Tatara N, Hartvigsen G, et al. Mobile health applications to assist patients with diabetes: lessons learned and design implications. J Diabetes Sci Technol 2012 Sep 1;6(5):1197-1206 [FREE Full text] [CrossRef] [Medline]
- Kitsiou S, Paré G, Jaana M, Gerber B. Effectiveness of mHealth interventions for patients with diabetes: an overview of systematic reviews. PLoS One 2017;12(3):e0173160 [FREE Full text] [CrossRef] [Medline]
- Liang X, Wang Q, Yang X, Cao J, Chen J, Mo X, et al. Effect of mobile phone intervention for diabetes on glycaemic control: a meta-analysis. Diabet Med 2011 Apr;28(4):455-463. [CrossRef] [Medline]
- Holtz B, Lauckner C. Diabetes management via mobile phones: a systematic review. Telemed J E Health 2012 Apr;18(3):175-184. [CrossRef] [Medline]
- Baron JS, Hirani S, Newman SP. A randomised, controlled trial of the effects of a mobile telehealth intervention on clinical and patient-reported outcomes in people with poorly controlled diabetes. J Telemed Telecare 2017 Feb;23(2):207-216. [CrossRef] [Medline]
- Davis FD. Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Q 1989 Sep;13(3):319. [CrossRef]
- Holden RJ, Karsh B. The technology acceptance model: its past and its future in health care. J Biomed Inform 2010 Feb;43(1):159-172 [FREE Full text] [CrossRef] [Medline]
- Yarbrough AK, Smith TB. Technology acceptance among physicians: a new take on TAM. Med Care Res Rev 2007 Dec;64(6):650-672. [CrossRef] [Medline]
- Michie S, van Stralen MM, West R. The behaviour change wheel: a new method for characterising and designing behaviour change interventions. Implement Sci 2011 Apr 23;6:42 [FREE Full text] [CrossRef] [Medline]
- Presseau J, Ivers NM, Newham JJ, Knittle K, Danko KJ, Grimshaw JM. Using a behaviour change techniques taxonomy to identify active ingredients within trials of implementation interventions for diabetes care. Implement Sci 2015 Apr 23;10:55 [FREE Full text] [CrossRef] [Medline]
- Michie S, Johnston M, Abraham C, Lawton R, Parker D, Walker A. Making psychological theory useful for implementing evidence based practice: a consensus approach. Qual Saf Health Care 2005 Feb;14(1):26-33 [FREE Full text] [CrossRef] [Medline]
- Riley WT, Rivera DE, Atienza AA, Nilsen W, Allison SM, Mermelstein R. Health behavior models in the age of mobile interventions: are our theories up to the task? Transl Behav Med 2011 Mar;1(1):53-71 [FREE Full text] [CrossRef] [Medline]
- Pal K, Eastwood SV, Michie S, Farmer AJ, Barnard ML, Peacock R, et al. Computer-based diabetes self-management interventions for adults with type 2 diabetes mellitus. Cochrane Database Syst Rev 2013 Mar 28(3):CD008776 [FREE Full text] [CrossRef] [Medline]
- Mann DM, Kannry JL, Edonyabo D, Li AC, Arciniega J, Stulman J, et al. Rationale, design, and implementation protocol of an electronic health record integrated clinical prediction rule (iCPR) randomized trial in primary care. Implement Sci 2011 Sep 19;6:109 [FREE Full text] [CrossRef] [Medline]
- Mann DM, Lin JJ. Increasing efficacy of primary care-based counseling for diabetes prevention: rationale and design of the ADAPT (avoiding diabetes thru action plan targeting) trial. Implement Sci 2012 Jan 23;7:6 [FREE Full text] [CrossRef] [Medline]
- Li AC, Kannry JL, Kushniruk A, Chrimes D, McGinn TG, Edonyabo D, et al. Integrating usability testing and think-aloud protocol analysis with 'near-live' clinical simulations in evaluating clinical decision support. Int J Med Inform 2012 Nov;81(11):761-772. [CrossRef] [Medline]
- Chokshi SK, Mann DM. Innovating from within: a process model for user-centered digital development in academic medical centers. JMIR Hum Factors 2018 Dec 19;5(4):e11048 [FREE Full text] [CrossRef] [Medline]
- Arsand E, Demiris G. User-centered methods for designing patient-centric self-help tools. Inform Health Soc Care 2008 Sep;33(3):158-169. [CrossRef] [Medline]
- Hekler EB, Klasnja P, Riley WT, Buman MP, Huberty J, Rivera DE, et al. Agile science: creating useful products for behavior change in the real world. Transl Behav Med 2016 Jun;6(2):317-328 [FREE Full text] [CrossRef] [Medline]
- Reaney M, Black P, Gwaltney C. A systematic method for selecting patient-reported outcome measures in diabetes research. Diabetes Spectr 2014 Nov;27(4):229-232 [FREE Full text] [CrossRef] [Medline]
- Meadows KA, Abrams C, Sandbaek A. Adaptation of the diabetes health profile (DHP-1) for use with patients with type 2 diabetes mellitus: psychometric evaluation and cross-cultural comparison. Diabet Med 2000 Aug;17(8):572-580. [CrossRef] [Medline]
- Brod M, Hammer M, Christensen T, Lessard S, Bushnell DM. Understanding and assessing the impact of treatment in diabetes: the treatment-related impact measures for diabetes and devices (TRIM-diabetes and TRIM-diabetes device). Health Qual Life Outcomes 2009 Sep 9;7:83 [FREE Full text] [CrossRef] [Medline]
- Chawla A, Saha C, Marrero DG. A novel application of the problem areas in diabetes (PAID) instrument to improve glycemic control and patient satisfaction. Diabetes Educ 2010;36(2):337-344. [CrossRef] [Medline]
- Costa FA, Guerreiro JP, Duggan C. An audit of diabetes dependent quality of life (ADDQoL) for Portugal: exploring validity and reliability. Pharm Pract (Granada) 2006;4(3):123-128 [FREE Full text] [Medline]
- Hammond GS, Aoki TT. Measurement of health status in diabetic patients. Diabetes impact measurement scales. Diabetes Care 1992 Apr;15(4):469-477. [CrossRef] [Medline]
- Polonsky WH, Fisher L, Earles J, Dudl RJ, Lees J, Mullan J, et al. Assessing psychosocial distress in diabetes: development of the diabetes distress scale. Diabetes Care 2005 Mar;28(3):626-631. [CrossRef] [Medline]
- Saisho Y. Use of diabetes treatment satisfaction questionnaire in diabetes care: importance of patient-reported outcomes. Int J Environ Res Public Health 2018 May 9;15(5):947 [FREE Full text] [CrossRef] [Medline]
- CMS Quality Measure Development Plan Supporting the Transition to the Merit-Based Incentive Payment System (MIPS) and Alternative Payment Models (APMs). CMS. 2016. URL: https://www.cms.gov/blog/cms-quality-measure-development-plan-supporting-transition-merit-based-incentive-payment-system-mips [accessed 2020-08-12]
- What is Workflow? Agency for Healthcare Research and Quality. 2016. URL: https://healthit.ahrq.gov/health-it-tools-and-resources/workflow-assessment-health-it-toolkit/workflow [accessed 2016-09-30]
- Shimoni N, Nippita S, Castaño PM. Best practices for collecting repeated measures data using text messages. BMC Med Res Methodol 2020 Jan 3;20(1):2 [FREE Full text] [CrossRef] [Medline]
- Malterud K, Siersma VD, Guassora AD. Sample size in qualitative interview studies: guided by information power. Qual Health Res 2016 Nov;26(13):1753-1760. [CrossRef] [Medline]
- Kolb S. Grounded theory and the constant comparitive method: valid research strategies for educators. J Emerg Trends Educ Res Policy Stud 2012;3:83-86 [FREE Full text]
- Strauss A, Corbin J. Grounded theory methodology. In: Senzin N, Lincon YS, editors. The Landscape of Qualitative Research. Thousand Oaks, CA: Sage Publications; 1994.
- Kjeldskov JS, Stage J. Instant data analysis: conducting usability evaluations in a day. Proc Third Nord Conf Human-Comput Interact 2004:- [FREE Full text] [CrossRef]
- Joe J, Chaudhuri S, Le T, Thompson H, Demiris G. The use of think-aloud and instant data analysis in evaluation research: exemplar and lessons learned. J Biomed Inform 2015 Aug;56:284-291 [FREE Full text] [CrossRef] [Medline]
- Engström MS, Leksell J, Johansson U, Gudbjörnsdottir S. What is important for you? A qualitative interview study of living with diabetes and experiences of diabetes care to establish a basis for a tailored patient-reported outcome measure for the Swedish national diabetes register. BMJ Open 2016 Mar 24;6(3):e010249 [FREE Full text] [CrossRef] [Medline]
- McCurdie T, Taneva S, Casselman M, Yeung M, McDaniel C, Ho W, et al. mHealth consumer apps: the case for user-centered design. Biomed Instrum Technol 2012;Suppl:49-56. [CrossRef] [Medline]
- Fortin M, Bravo G, Hudon C, Vanasse A, Lapointe L. Prevalence of multimorbidity among adults seen in family practice. Ann Fam Med 2005;3(3):223-228 [FREE Full text] [CrossRef] [Medline]
- Sasseville M, Chouinard M, Fortin M. Patient-reported outcomes in multimorbidity intervention research: a scoping review. Int J Nurs Stud 2018 Jan;77:145-153. [CrossRef] [Medline]
|COM-B: capability, opportunity, and motivation model of behavior|
|EHR: electronic health record|
|ESRD: end-stage renal disease|
|HbA1c: hemoglobin A1c|
|i-Matter: investigating an mHealth texting tool for embedding patient-reported data into diabetes management|
|MCIT: Medical Center Information Technology|
|mHealth: mobile health|
|NYULH: New York University Langone Health|
|PRO: patient-reported outcome|
|RA: research assistant|
|RCT: randomized controlled trial|
|T2D: type 2 diabetes|
|TAM: technology acceptance model|
|UCD: user-centered design|
Edited by G Eysenbach; submitted 04.03.20; peer-reviewed by H Barahimi, T Wieringa; comments to author 12.06.20; revised version received 09.07.20; accepted 26.07.20; published 31.08.20Copyright
©Antoinette Schoenthaler, Jocelyn Cruz, Leydi Payano, Marina Rosado, Kristen Labbe, Chrystal Johnson, Javier Gonzalez, Melissa Patxot, Smit Patel, Eric Leven, Devin Mann. Originally published in JMIR Formative Research (http://formative.jmir.org), 31.08.2020.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on http://formative.jmir.org, as well as this copyright and license information must be included.