Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Monday, March 11, 2019 at 4:00 PM to 4:30 PM EST. We apologize in advance for any inconvenience this may cause you.
Who will be affected?
Readers: No access to all 28 journals. We recommend accessing our articles via PubMed Central
Authors: No access to the submission form or your user account.
Reviewers: No access to your user account. Please download manuscripts you are reviewing for offline reading before Monday, March 11, 2019 at 4:00 PM.
Editors: No access to your user account to assign reviewers or make decisions.
Copyeditors: No access to user account. Please download manuscripts you are copyediting before Monday, March 11, 2019 at 4:00 PM.
JMIR Formative Research
Process evaluations, early results, and feasibility/pilot studies of digital and non-digital interventions
Background: Providing patients access to their medical notes has been demonstrated to offer many benefits for patients and providers. This has led to a rapidly expanding national movement, OpenNotes, which provides resources to clinicians who desire to share medical notes with their patients. However, a significant barrier to the widespread adoption of OpenNotes is clinicians' concern that the medical terminology in such notes may confuse patients. Objective: Artificial intelligence (AI) software may provide the opportunity to rapidly simplify medical notes to plain language through natural language processing. This offers the potential to resolve concerns over medical terminology and patient confusion. This pilot study assesses patient’s perception of AI-simplified plain language medical notes. Methods: Patient’s perception of notes was studied through comprehension questionnaires and guided interviews with subsequent thematic analysis. Study participants were recruited from patients hospitalized at the University of Colorado Hospital. A standardized cardiology patient’s note was generated using a synthetic patient generator. AI software applied natural language processing to produce a simplified version of this note. Patients were randomly assigned to first read either the original template note or the simplified version, then completed a set of seven comprehension assessment questions. Subsequently, patients reviewed the opposite version of the note and participated in a guided interview to discuss their thoughts on these notes. Statistical analysis of comprehension questionnaire scores was performed to assess for differences in comprehension between original (unsimplified) and simplified notes. Guided interview responses were thematically analyzed. Results: Twenty patients agreed to participate. The study was found to be underpowered to detect statistical significance of the impact of simplified notes on participant comprehension. Though the mean number of comprehension assessment questions answered correctly was found to be higher in the simplified note group at 4.7 as compared to 3.9 in the unsimplified note group, this was found to be non-significant (p=0.32).
Guided interviews found that AI simplified open notes were perceived as desirable and beneficial by participants. Thematic analysis identified that simplified medical notes may (1) be more useable than unsimplified notes, (2) improve the patient-provider relationship, and (3) empower patients through an enhanced understanding of their conditions and management. Participant’s recommendations highlighted the need to reduce lengthy plain-language phrases and to target the level of simplification to each patient’s health literacy. Conclusions: Simplified notes were well received by participants, who expressed a desire to have access to such notes for their own medical conditions. This study illustrates the potential for artificial intelligence software to quickly generate plain language medical notes that are useful for patients and their providers. Feedback from participants in this study should be used to improve the simplification of notes. Larger studies should be conducted with heed to the insight gained from this pilot study.
(Toronto, Nov 29th, 2018) We are pleased to report that JMIR Formative Res has been accepted for indexing in PubMed Central and PubMed. The staff at PubMed Central is currently finalizing the setup and we expect all articles published since Vol 1 / Iss 1 (2017) to be in PubMed Central and PubMed within the next couple of weeks. The acceptance is the result of a rigorous scientific and technical evaluation by the US National Library's (NLM’s) Library Operations Division, which decided that the scientific and editorial character and quality of a JMIR Formative Research merits its inclusion in PMC. In making this decision NLM has considered the suitability of the journal for the NLM collection as well as the opinions of expert consultants.
JMIR Formative Research (JFR) (a PubMed-indexed sister journal of J Med Internet Res (JMIR) and JMIR mHealth & uHealth, the leading eHealth and mHealth journals by impact factor) publishes peer-reviewed, openly accessible papers containing results from process evaluations, feasibility/pilot studies and other kinds of formative research and preliminary results. While the original focus was on the design of medical and health-related research and technology innovations, JFR publishes studies from all areas of medical and health research.
Formative research is research that occurs before a program is designed and implemented, or while a program is being conducted. Formative research can help
define and understand populations in need of a intervention or public health program
create programs that are specific to the needs of those populations
ensure programs are acceptable and feasible to users before launching
improve the relationship between users and agencies/research groups
demonstrate the feasibility, use, satisfaction with, or problems with a program before large-scale summative evaluation (looking at health outcomes)
Many funding agencies will expect some sort of pilot/feasibility/process evaluation before funding a larger study such as a Randomized Controlled Trial (RCT).
Formative research should be an integral part of developing programs or adapting programs, and should be used while the program is on-going to help refine and improve program activities. Thus, formative evaluation can and should also occur in form of a process evaluation alongside a summative evaluation such as a RCT.
This journal fills an important gap in the academic journals landscape, as it publishes sound and peer-reviewed formative research that is criticial for investigators to apply for further funding, but that is usually not published in outcomes-focussed medical journals aiming for impact and generalizability.
Summative evaluations of programs and apps/software that have undergone a thorough formative evaluation before launch have a better chance to be published in our high-impact flagship journals (JMIR mHealth, J Med Internet Res, JMIR Serious Games), thus we encourage authors to submit - as a first step - their formative evaluations in JMIR Formative Res (and their evaluation protocols in JMIR Res Protoc).
Background: Clinical guidelines recommend screening for depression in the general adult population but recognizes that the optimum interval for screening is unknown. Ideal screening intervals should match the patient risk profiles. Objective: This study describes a predictive analytics approach for mining clinical and patient-reported data from a large clinical study for the identification of primary care patients at high risk for depression to match depression screening intervals with patient risk profiles. Methods: This paper analyzed data from a large safety-net primary care study for diabetes and depression. A regression-based data mining technique was used to examine 53 demographics, clinical variables, and patient-reported variables to develop three prediction models for major depression at 6, 12, and 18 months from baseline. Predictors with the strongest predictive power that require low information collection efforts were selected to develop the prediction models. Predictive accuracy was measured by the area under the receiver operating curve (AUROC) and was evaluated by 10-fold cross-validation. The effectiveness of the prediction algorithms in supporting clinical decision making for six “typical” types of patients was demonstrated. Results: The analysis included 923 patients who were nondepressed at the study baseline. Five patient-reported variables were selected in the prediction models to predict major depression at 6, 12, and 18 months: (1) Patient Health Questionnaire 2-item score; (2) the Sheehan Disability Scale; (3) previous problems with depression; (4) the diabetes symptoms scale; and (5) emotional burden of diabetes. All three depression prediction models had an AUROC>0.80, comparable with published depression prediction studies. Among the 6 “typical” types of patients, the algorithms suggest that patients who reported impaired daily functioning by health status are at an elevated risk for depression in all three periods. Conclusions: This study demonstrated that leveraging patient-reported data and prediction models can help improve identification of high-risk patients and clinical decisions about the depression screening interval for diabetes patients. Implementation of this approach can be coupled with application of modern technologies such as telehealth and mobile health assessment for collecting patient-reported data to improve privacy, reducing stigma and costs, and promoting a personalized depression screening that matches screening intervals with patient risk profiles.
Background: Borderline personality disorder (BPD) is a disorder characterized by difficulties with regulating emotions and impulsive behavior. Long-term monitoring of progress during BPD psychotherapy constitutes a challenge using paper and pencil registration. Hence, a mobile app assessing emotions and progress in treatment may be useful. Objective: The aim of this study was to examine the feasibility of using the mDiary app as an adjunct to dialectical behavior therapy (DBT) for the treatment of BPD. Methods: A total of 9 focus group interviews were conducted and analyzed according to the grounded theory approach. Furthermore, the usability of the mDiary app was examined using the System Usability Scale (SUS). The app was implemented in a standard DBT program as an adjunct to DBT. In total, 16 patients (age range 19-41 years) and 23 therapists (age range 25-64 years) from 5 Danish public outpatient psychiatric treatment facilities participated in the study. Results: Overall, patients were satisfied with the mDiary app, as it was “easy to use” and “always there.” Inside-out innovation, meaning new work tasks generated during implementation and communication of modifications needed in the app, was found to influence the perceived usability negatively among the interviewed therapists. The patients rated the usability as high (mean SUS score 81.2, SD 9.9), whereas therapists rated the mDiary app at an average level (mean 68.3, SD 14.3). Older age of the users correlated with lower usability ratings on the SUS score (Pearson r=−0.60). Conclusions: The mDiary app was considered as an acceptable and relevant way of registering DBT diary data for both patients and therapists generating increased long-term overview. Older users were overall more reluctant to accept this new technology in clinical practice. Time to align expectations among involved parties needs to be set aside when implementing this new approach to patient monitoring. Here, the focus should be realistic use of resources and expected impact on present clinical work.
Background: In the past years, a mobile health (mHealth) app called the Dutch Talking Touch Screen Questionnaire (DTTSQ) was developed in The Netherlands. The aim of development was to enable Dutch physical therapy patients to autonomously complete a health-related questionnaire regardless of their level of literacy and digital skills. Objective: The aim of this study was to evaluate the usability (defined as the effectiveness, efficiency, and satisfaction) of the prototype of the DTTSQ for Dutch physical therapy patients with diverse levels of experience in using mobile technology. Methods: The qualitative Three-Step Test-Interview method, including both think-aloud and retrospective probing techniques, was used to gain insight into the usability of the DTTSQ. A total of 24 physical therapy patients were included. The interview data were analyzed using a thematic content analysis approach aimed at analyzing the accuracy and completeness with which participants completed the questionnaire (effectiveness), the time it took the participants to complete the questionnaire (efficiency), and the extent to which the participants were satisfied with the ease of use of the questionnaire (satisfaction). The problems encountered by the participants in this study were given a severity rating that was used to provide a rough estimate of the need for additional usability efforts. Results: All participants within this study were very satisfied with the ease of use of the DTTSQ. Overall, 9 participants stated that the usability of the app exceeded their expectations. The group of 4 average-/high-experienced participants encountered only 1 problem in total, whereas the 11 little-experienced participants encountered an average of 2 problems per person and the 9 inexperienced participants an average of 3 problems per person. A total of 13 different kind of problems were found during this study. Of these problems, 4 need to be addressed before the DTTSQ will be released because they have the potential to negatively influence future usage of the tool. The other 9 problems were less likely to influence future usage of the tool substantially. Conclusions: The usability of the DTTSQ needs to be improved before it can be released. No problems were found with satisfaction or efficiency during the usability test. The effectiveness needs to be improved by (1) making it easier to navigate through screens without the possibility of accidentally skipping one, (2) enabling the possibility to insert an answer by tapping on the text underneath a photograph instead of just touching the photograph itself, and (3) making it easier to correct wrong answers. This study shows the importance of including less skilled participants in a usability study when striving for inclusive design and the importance of measuring not just satisfaction but also efficiency and effectiveness during such studies.
Background: The impact of mental ill-health on every aspect of the lives of a large number of Australian Defence Force (ADF) personnel, their partners, and their families is widely recognized. Recent Senate inquiries have highlighted gaps in service delivery as well as the need for service reform to ensure appropriate care options for individuals who are currently engaged with mental health and support services as well as for those who, for a variety of reasons, have not sought help. To that end, successive Australian governments generally and the Department of Veterans’ Affairs specifically have prioritized veteran-centric reform. Open Arms is an Australia-wide service that provides counseling and support to current and former ADF personnel, and their family members, for mental health conditions. Objective: The aim of this study was to develop and configure a prototypic Web-based platform for Open Arms—Veterans & Families Counselling (formerly Veterans and Veterans Families Counselling Service) with the Open Arms community to enhance the quality of mental health services provided by Open Arms. Methods: The study aimed to recruit up to 100 people from the Open Arms community (current and former ADF personnel and their families, health professionals, service managers, and administrators) in regions of New South Wales, including Sydney, Canberra, Maitland, Singleton, and Port Stephens. Participants were invited to participate in 4-hour participatory design workshops. A variety of methods were used within the workshops, including prompted discussion, review of working prototypes, creation of descriptive artifacts, and group-based development of user journeys. Results: Seven participatory design workshops were held, including a total of 49 participants. Participants highlighted that the prototype has the potential to (1) provide the opportunity for greater and better-informed personal choice in relation to options for care based on the level of need and personal preferences; (2) ensure transparency in care by providing the individual with access to all of their personal health information; and (3) improve collaborative care and care continuity by allowing information to be shared securely with current and future providers. Conclusions: Our findings highlight the value of actively engaging stakeholders in participatory design processes for the development and configuration of new technologies.
Background: Psychiatry research has begun to leverage data collected from patients’ social media and smartphone use. However, information regarding the feasibility of utilizing such data in an outpatient setting and the acceptability of such data in research and practice is limited. Objective: This study aimed at understanding the outpatients’ willingness to have information from their social media posts and their smartphones used for clinical or research purposes. Methods: In this survey study, we surveyed patients (N=238) in an outpatient clinic waiting room. Willingness to share social media and passive smartphone data was summarized for the sample as a whole and broken down by sex, age, and race. Results: Most patients who had a social media account and who were receiving talk therapy treatment (74.4%, 99/133) indicated that they would be willing to share their social media posts with their therapists. The percentage of patients willing to share passive smartphone data with researchers varied from 40.8% (82/201) to 60.7% (122/201) depending on the parameter, with sleep duration being the parameter with the highest percentage of patients willing to share. A total of 30.4% of patients indicated that media stories of social media privacy breaches made them more hesitant about sharing passive smartphone data with researchers. Sex and race were associated with willingness to share smartphone data, with men and whites being the most willing to share. Conclusions: Our results indicate that most patients in a psychiatric outpatient setting would share social media and passive smartphone data and that further research elucidating patterns of willingness to share passive data is needed.
Background: Persons living with dementia represent a significant and growing segment of the older adult (aged 65 years and older) population. They are often challenged expressively and may experience difficulties with sharing their feelings or moods. Availability of, and easy access to, tablets facilitates the use of information and communication technologies (ICTs) as a delivery mechanism for nonpharmacological interventions, especially for persons living with dementia. Evidence of the impact of ICTs in different community settings on mood with older adults and the impact of engagement on their caregivers is needed to promote broader adoption and sustainment of these technologies in the United States. Objective: This study aimed to determine the extent of the effects of tablets on positive mood change and examine the effects of study variables on care recipients’ mood changes and caregivers’ daily interactions. Methods: The tablet intervention was developed and evaluated in five programs. The primary outcome was caregivers’ assessment of care recipients’ mood (n=1089) before and after a tablet engagement session using an eight-point mood visual analog scale. Session influence on caregivers’ daily activities was captured for a subsample of participants (n=542). Frequency distributions were computed for each study variables. Chi-square tests of association were calculated to determine the association of the variables on mood changes for all care recipients, as well as those being treated in skilled nursing facilities and in-home, and then for those that affected caregivers’ daily activities. Results: The study sample comprised 1089 care recipient and caregiver engagement sessions. Cumulatively, 50.78% (553/1089) of care recipients showed a transition from negative to positive moods, whereas another 41.78% (455/1089) maintained an already-positive mood after the caregiver engagement session. Chi-square analyses demonstrated that positive mood changes resulted from using music (χ210=72.9; P<.001), using YouTube as the sole app (χ212=64.5; P<.001), using multiple engagement strategies (χ22=42.8; P<.001), and when cared for in a skilled nursing facility (χ24=236.8; P<.001) across the entire care recipient sample. In addition, although many features of the engagement session positively influenced the caregivers’ day, the largest effect was observed when care recipients’ mood was considered to have improved following the session (χ24=234.7; P<.001). Conclusions: The study is one of the first in the United States to explore the impact of ICTs, in particular managed tablets and Web-based video services that can be used on a tablet through an app, on improving mood in persons living with dementia, and enhancing caregivers’ perceptions about their care recipient interactions. Importantly, these pilot data substantiate ICTs as part of a personalized engagement approach, as beneficial alternatives to pharmaceutical interventions for mood enhancement. However, a more comprehensive study that explores the ICT’s impact on additional clinical outcomes is needed to confirm these preliminary findings.
Background:
Dengue is a mosquito-borne viral disease that has increasingly affected Sri Lanka in recent years. To address this issue, dengue surveillance through increasingly prevalent digital surveillance applications has been suggested for use by health authorities and the general public. Epihack Sri Lanka was a 5-day hackathon event organized to develop a digital dengue surveillance tool.
Methods:
Ethnographic observation and qualitative informal interviews were conducted with 58 attendees from 11 countries over the 5-day Epihack to identify the main factors that influence a collaborative hackathon. Interviews were transcribed and coded based on grounded theory.
Results:
Three major themes were identified during the Epihack Sri Lanka event: engagement, communication, and current disease environment. Unlike other hackathons, Epihack had no winners or prizes and was collaborative rather than competitive, which worked well in formulating a variety of ideas and bringing together volunteers with a sense of civic duty to improve public health. Having health and IT experts work together concurrently was received positively and considered highly beneficial to the development of the product. Participants were overall very satisfied with the event, although they thought it could have been longer. Communication issues and cultural differences were observed but continued to decrease as the event progressed. This was found to be extremely important to the efficiency of the event, which highlighted the benefit of team-bonding exercises. Bringing expert knowledge and examples of systems from around the world benefited the creation of new ideas. However, developing a system that can adapt and cater to the local disease environment is important in successfully developing the concepts.
Conclusions:
Epihack Sri Lanka was successful in bringing together health and IT experts to develop a digital solution for dengue surveillance. The collaborative format achieved a variety of fruitful ideas and may lead to more hackathons working in this way in the future. Good communication, participant engagement, and stakeholder interest with adaptation of ideas to complement the current environment are vital to achieve the goals of the event.
Background: Whiplash is a health and economic burden worldwide. Contributing to this burden is poor guideline adherence and variable management by health care professionals (HCPs). Web-based tools that facilitate clinical pathways of care are an innovative solution to improve management. Objective: The study aimed to develop, implement, and evaluate a Web-based tool to support whiplash management following a robust process. Methods: This study followed the first 3 processes of a research translation framework (idea generation, feasibility, and efficacy) to inform the development, implementation, and evaluation of a website that supports HCPs in whiplash management. Development followed the idea generation and feasibility processes to inform the content, design, features, and functionality of the website. This involved stakeholder (eg, industry partners, website developers, and HCPs) consultations through face-to-face meetings, surveys, and focus group discussions. Implementation followed the feasibility process to determine the practicality of the website for clinical use and the most effective strategy to promote wider uptake. Implementation strategies included classroom education, educational meetings, educational outreach, reminders, and direct phone contact. The analysis of website use and practicality of implementation involved collection of website metrics. Evaluation followed the feasibility and efficacy processes to investigate the acceptability and extent to which the website assisted HCPs in gaining knowledge about whiplash management. Surveys were conducted among student, primary, and specialist HCPs to explore ease of access, use, and satisfaction with the website, as well as self-rated improvements in knowledge of risk assessment, management, and communication between HCPs. Website logs of specialist management decisions (eg, shared care, specialist care, and referred care) were also obtained to determine actual practice. Results: The development process delivered an interactive, user-friendly, and acceptable website, My Whiplash Navigator, tailored to the needs of HCPs. A total of 260 registrations were recorded from June 2016 to March 2018, including 175 student, 65 primary, and 20 specialist HCPs. The most effective implementation strategies were classroom education for students (81% uptake, 175/215) and educational meetings for primary HCPs (43% uptake, 47/110). Popular pages visited included advice and exercises and risk assessment. Most HCPs agreed that their knowledge about risk management (79/97, 81%) and exercises (85/97, 88%) improved. The specialists’ most common management decision was shared care, an improvement from a previous cohort. Areas to improve were navigation and access to outcome measures. Conclusions: A robust process resulted in an innovative, interactive, user-friendly, and acceptable website, the My Whiplash Navigator. Implementation with HCPs was best achieved through classroom education and educational meetings. Evaluation of the website showed improved knowledge and practice to be more consistent with a risk-based clinical care pathway for whiplash. The positive results provide sufficient evidence to scale implementation nationally and involve other target markets such as people with whiplash, insurers, and insurance regulators.
Background: Providing patients access to their medical notes has been demonstrated to offer many benefits for patients and providers. This has led to a rapidly expanding national movement, OpenNotes,...
Background: Providing patients access to their medical notes has been demonstrated to offer many benefits for patients and providers. This has led to a rapidly expanding national movement, OpenNotes, which provides resources to clinicians who desire to share medical notes with their patients. However, a significant barrier to the widespread adoption of OpenNotes is clinicians' concern that the medical terminology in such notes may confuse patients. Objective: Artificial intelligence (AI) software may provide the opportunity to rapidly simplify medical notes to plain language through natural language processing. This offers the potential to resolve concerns over medical terminology and patient confusion. This pilot study assesses patient’s perception of AI-simplified plain language medical notes. Methods: Patient’s perception of notes was studied through comprehension questionnaires and guided interviews with subsequent thematic analysis. Study participants were recruited from patients hospitalized at the University of Colorado Hospital. A standardized cardiology patient’s note was generated using a synthetic patient generator. AI software applied natural language processing to produce a simplified version of this note. Patients were randomly assigned to first read either the original template note or the simplified version, then completed a set of seven comprehension assessment questions. Subsequently, patients reviewed the opposite version of the note and participated in a guided interview to discuss their thoughts on these notes. Statistical analysis of comprehension questionnaire scores was performed to assess for differences in comprehension between original (unsimplified) and simplified notes. Guided interview responses were thematically analyzed. Results: Twenty patients agreed to participate. The study was found to be underpowered to detect statistical significance of the impact of simplified notes on participant comprehension. Though the mean number of comprehension assessment questions answered correctly was found to be higher in the simplified note group at 4.7 as compared to 3.9 in the unsimplified note group, this was found to be non-significant (p=0.32).
Guided interviews found that AI simplified open notes were perceived as desirable and beneficial by participants. Thematic analysis identified that simplified medical notes may (1) be more useable than unsimplified notes, (2) improve the patient-provider relationship, and (3) empower patients through an enhanced understanding of their conditions and management. Participant’s recommendations highlighted the need to reduce lengthy plain-language phrases and to target the level of simplification to each patient’s health literacy. Conclusions: Simplified notes were well received by participants, who expressed a desire to have access to such notes for their own medical conditions. This study illustrates the potential for artificial intelligence software to quickly generate plain language medical notes that are useful for patients and their providers. Feedback from participants in this study should be used to improve the simplification of notes. Larger studies should be conducted with heed to the insight gained from this pilot study.
Background: Mobile communication has been found to improve ART-adherence among people living with HIV (PLHIV). In our ongoing randomized clinical trial, we used two mobile communication strategies i...
Background: Mobile communication has been found to improve ART-adherence among people living with HIV (PLHIV). In our ongoing randomized clinical trial, we used two mobile communication strategies i.e., sending SMS and real-time medication monitoring (RTMM), to monitor adherence to ART and to intervene on adherence if indicated among PLHIV in Tanzania. We noticed a remarkable discrepancy between self-reported adherence and adherence recorded by SMS or RTMM among some of the first trial participants. Objective: Our objective was to explore reasons for those discrepancies. Methods: Adult PLHIV suspected of low levels of adherence were randomized to receiving adherence monitoring and reminders to take ART using 1) SMS versus 2) RTMM versus 3) a control group in two HIV treatment centers in Tanzania. During bi-monthly study visits PLHIV self-reported their level of adherence, received feedback about their level of adherence based on SMS or RTMM monitoring, and discussed strategies to overcome problems with adherence with HIV nurses. For this report, we selected PLHIV who had completed all follow-up visits and consistently self-reported a markedly different level of adherence to the nurse than was recorded by SMS or RTMM. These PLHIV were invited to a face-to-face interview to explore reasons for this discrepancy. Results: Twenty-six participants had completed follow-up. Six reported discrepancies and had an average of 46% adherence based on SMS/RTMM monitoring , while self-report recorded good adherence . Five participants insisted that their adherence to ART was good but four of them admitted that their adherence to properly using the monitoring device was low. Three participants mentioned concerns about involuntary disclosure of HIV status as main reason for low adherence to ART and/or device use. Two participants were still depending on other reminder cues despite receiving SMS or RTMM reminders. Poor network coverage caused apparently low adherence in one participant. Conclusions: Concerns about involuntary disclosure of HIV status was a main reason for low adherence to ART and/or proper adherence monitoring device use. Some participants still depended on reminders such as alarms despite receiving SMS or RTMM reminders illustrating the need for education and close monitoring when using new digital technologies in resource limited settings.
