Maintenance Notice

Due to necessary scheduled maintenance, the JMIR Publications website will be unavailable from Monday, March 11, 2019 at 4:00 PM to 4:30 PM EST. We apologize in advance for any inconvenience this may cause you.

Who will be affected?

Advertisement

Citing this Article

Right click to copy or hit: ctrl+c (cmd+c on mac)

Published on 18.04.19 in Vol 3, No 2 (2019): Apr-Jun

Preprints (earlier versions) of this paper are available at http://preprints.jmir.org/preprint/10688, first published Apr 04, 2018.

This paper is in the following e-collection/theme issue:

    Original Paper

    Implementing Web-Based Interventions in HIV Primary Care Clinics: Pilot Implementation Evaluation of Positive Health Check

    1Division of HIV/AIDS Prevention, Centers for Disease Control and Prevention, Atlanta, GA, United States

    2Center for Communication Science, RTI International, Durham, NC, United States

    3Digital Health and Clinical Informatics, RTI International, Durham, NC, United States

    4Behavioral and Urban Health Program, RTI International, Durham, NC, United States

    Corresponding Author:

    Camilla Harshbarger, PhD

    Division of HIV/AIDS Prevention

    Centers for Disease Control and Prevention

    1600 Clifton Rd, MSE-37

    Atlanta, GA,

    United States

    Phone: 1 404 639 4267

    Fax:1 404 639 0944

    Email:


    ABSTRACT

    Background: Web-based interventions can help people living with HIV achieve better clinical outcomes and behaviors, but integrating them into clinical practice remains challenging. There is a gap in understanding the feasibility of implementing these interventions in HIV clinic settings from the clinicians’ perspective.

    Objective: The goal of the research was to determine whether Positive Health Check (PHC)—a Web-based, tailored video counseling tool focused on increasing patient adherence and retention in care and reducing HIV risk among HIV-positive patients—was acceptable, appropriate, and feasible for HIV primary care clinic staff to implement in clinic workflows.

    Methods: A multiple-case study design was used to evaluate the pilot implementation. Four primary care clinics located in the southeastern United States implemented PHC over a 1-month period. Nine clinic staff across the clinics participated in structured interviews before, during, and after the implementation. In total, 54 interviews were conducted. We used a framework analysis approach to code the data and identify themes related to implementation outcomes, including acceptability, appropriateness, and feasibility. We also analyzed patient intervention use metrics (n=104) to quantify patient intervention completion rates (n=68).

    Results: Overall, clinicians viewed PHC as acceptable and appropriate. Themes that emerged related to these implementation outcomes include the ability for PHC to increase provider-patient communication and its ability to engage patients due to the tailored and interactive design. While generally feasible to implement, challenges to the clinic workflow and physical environment were areas that clinics needed to manage to make PHC work in their clinics.

    Conclusions: Findings from this pilot implementation suggest that clinical staff viewed PHC as acceptable and appropriate, especially as more patients used the intervention over the pilot period. Feasibility of implementation was challenging in some cases, and lessons learned from this pilot implementation can provide information for larger scale tests of the intervention that include assessment of both implementation outcomes and clinical outcomes.

    JMIR Form Res 2019;3(2):e10688

    doi:10.2196/10688

    KEYWORDS



    Introduction

    Interventions are being developed for clinical settings to retain and support patients in HIV care [1,2]. However, the clinical context presents barriers to implementing new HIV retention and adherence interventions [3-7]. These barriers need to be addressed before widespread adoption of clinic-based interventions can occur [8].

    Computer-based HIV adherence interventions appear to be feasible and acceptable from the patient perspective [9], but they have not been studied from the perspective of the clinic stakeholder implementing such interventions. The benefit of understanding implementation is that strategies can be designed to facilitate integration of evidenced-based interventions into care. Positive Health Check (PHC) is a brief, interactive Web-based video counseling tool to reduce HIV transmission and improve health outcomes for people living with HIV (PLH). The video tool was developed based on evidence that computer-based counseling tools can reduce sexual risk behaviors and improve antiretroviral therapy adherence [1,2,10-13] and viral load suppression [12]. PHC is grounded in the information-motivation-behavior model [14], assuming that providing information and building motivation and skills for medication adherence, appointment keeping, and other behaviors will result in PLH correctly practicing behaviors needed to manage HIV and improve health outcomes. We used principles of motivational interviewing [15] to guide the scripting of empathic language. The transtheoretical model [16] informed tailoring scripts, for example, around the extent to which individual HIV patients choose to interact with PHC intervention modules, ask questions of their provider, or select and practice behavioral strategies.

    The making of PHC was a dynamic 3-year endeavor involving many stakeholders. The goal was to build a Web-based intervention that would support and be adopted by HIV clinics, be easily updated and scaled up, and improve patient health outcomes. We engaged a user-centered approach and gathered detailed and iterative feedback on the many steps taken to design, develop, and implement PHC from prospective HIV-patient users and HIV providers, including both implementers and gatekeepers. This process is described in Harshbarger et al [17]. To film, manage, program, use feedback, and fine-tune PHC from the database of 700 video clips, the PHC team employed an agile development process [18]. We worked with many contributors, including infectious disease researchers, app developer, videographer, graphic artist, and closed captioning specialist.

    The purpose of the pilot implementation evaluation of PHC we describe here was to understand the barriers and facilitators to implementing the intervention in busy clinic settings. Specifically, we were interested in how clinic stakeholders managed the implementation. We examined their perceptions of implementation outcomes including (1) appropriateness (ie, the perceived fit, relevance, or compatibility of PHC to support clinic efforts to improve patient viral load suppression and well-being), (2) acceptability (ie, the perception among implementation stakeholders that PHC is agreeable, palatable, or satisfactory and supports the mission of the HIV clinics), and (3) feasibility (ie, the extent to which clinic staff can successfully administer or use PHC within the busy workflow of HIV clinic settings) [19,20]. Relying on data from the tool itself, we also aimed to determine if staff could provide patients with sufficient time in waiting rooms to complete the tool. This paper aims to address a gap in the literature to better understand staffer implementation of Web-based interventions in their complex clinical settings and workflows [21].


    Methods

    Positive Health Check Intervention and Pilot Implementation Evaluation Procedures

    An overview of PHC is shown in Figure 1. PHC is introduced to patients by a designated clinic staffer, referred to as an onboarder, who offers an eligible patient in the waiting room the opportunity to use the Web-based intervention. The onboarder accesses the PHC clinic Web application (CWA) to generate a user ID and password. Based on responses to questions about clinic attendance, medication use, and HIV risk behaviors, each patient watches individually tailored videos addressing HIV treatment readiness, antiretroviral therapy adherence, retention in HIV medical care, sexual risk reduction, prevention of mother-to-child transmission, and safer injection drug use practices. Patients can select questions to ask their clinic doctor during their scheduled appointment, and they are provided behavioral strategies, called tips, to practice. A patient handout featuring this information is automatically printed and delivered to the patient, and a truncated version is delivered to the provider at the request of the patient. At the end of the intervention, patients can also click on “Extra Info” to view supporting resource materials. The onboarder uses the CWA to track process data, including the number of patients who logged on, completed the intervention, or requested that a link to PHC be sent to their private email. PHC does not collect any personal identifying information or patient data, and no email addresses are stored.

    Study Design

    This implementation evaluation pilot was conducted from May to July 2015, and each of the four participating clinics implemented PHC for 1 month during that period. A multiple case-study design [22] was used to gather process evaluation data from clinic staff to examine and describe the contextual and implementation issues that might help other clinics adopting PHC prepare for implementation [23]. We also analyzed de-identified tool use data to understand patient navigation, use, and completion of the intervention.

    Figure 1. Positive Health Check patient experience flow diagram. IDU: injection drug use; MTCT: mother-to-child transmission.
    View this figure

    Clinic Eligibility

    Four HIV clinics were located in rural, urban, and suburban areas across the southeastern United States. All clinics were required to (1) provide primary HIV care to at least 200 HIV-positive patients annually; (2) use the PHC CWA on a secure, networked Windows desktop, workstation, or server; and (3) have broadband internet access that supports wireless access for iPad and Android devices. The characteristics of the four participating clinics are summarized in Table 1.

    Patient Eligibility

    HIV-positive English-speaking patients who were at least 18 years old were eligible to use the intervention during the pilot implementation evaluation. Onboarders invited patients to use PHC after they checked in to see their provider. One clinic contacted patients by telephone before appointments and issued invitations.

    Preparing Clinics for Implementation

    We provided each clinic with three tablets equipped with high-impact protective cases, privacy screens, and headphones for individual patient use. Additionally, we trained clinic staff how to use PHC and the CWA to onboard patients and generate summary reports. Staff also received user guides covering all aspects of intervention implementation.

    Data Collection

    We collected feedback from clinical staff (N=9), including the intervention onboarder and one primary care provider, at each clinic. At one site, an additional HIV primary care provider participated. We also collected data from the CWA to understand patient use of the intervention (eg, whether the patient completed the intervention; Table 2).

    Pairs of interviewers conducted a series of semistructured interviews with each key informant. Each informant participated in 6 interviews: 1 face-to-face preimplementation interview, 4 weekly telephone interviews, and 1 telephone wrap-up interview, for a total of 54 interviews across all 4 clinics. Preimplementation questions focused primarily on perceptions of clinic preparedness and satisfaction with trainings and training materials. During the weekly interviews, we asked participants questions related to the implementation of PHC, including barriers and facilitators; contextual factors; and acceptability, appropriateness, and feasibility of the intervention. For example, we asked, “What types of patients do you think will most benefit from the intervention?” (appropriateness); “As a provider, what do you like the most/least about Positive Health Check?” (acceptability); “Describe to me how the tool is typically incorporated into your workflow, from the beginning to the end of a patient’s appointment” (appropriateness); “What aspects of your clinic environment do you think were an issue for how PHC was implemented? Why?” (feasibility); and “How, if at all, has implementing Positive Health Check affected the workflow of the doctors at your clinic?” (acceptability/feasibility). Wrap-up interviews touched on these same topics and asked for suggested changes to the intervention and perceptions of sustainability. Development and pilot implementation of the tool was approved as nonresearch by the Office of the Associate Director for Science in the Division of HIV/AIDS Prevention at the US Centers for Disease Control and Prevention (CDC).

    Analysis

    Interview notes were entered into NVivo 10.0 (QSR International Pty Ltd). Notes were tagged by clinic, key informant type, and implementation phase (preimplementation, implementation weeks 1 through 4, and postimplementation). Notes were checked against audio recordings as needed for clarification and to ensure the accuracy of direct quotes used in reporting. A team of four trained staff coded the interviews using a topical codebook that operationalized concepts from the interviews. Concepts included barriers and facilitators, contextual factors, acceptability, appropriateness, feasibility, and sustainability.

    Table 1. Characteristics of HIV primary care clinics in the pilot implementation evaluation (percentages may not add to 100% because of rounding).
    View this table
    Table 2. Use information generated from the clinic Web application for Positive Health Check.
    View this table

    We conducted two rounds of test coding to establish consistency between coders.

    We then used the framework analysis method [24] to identify themes within and across sites, generating theme-based charts to organize the data by code, clinic staff position, site, and time point. To complete the framework analysis, team members completed one round of thematic analysis on the same topical code to ensure consistent interpretation of the data. Discrepancies were reconciled, and team members then worked independently to identify themes in each code. A theme was deemed important when multiple interviewees echoed the same idea within the same site, across sites, and over time. Quotes from interview participants illustrating themes are included in Results. We summarize descriptive statistics about tool use generated from the CWA in Table 2.


    Results

    Patient Consent and Completion Rates

    A total of 145 patients were approached to use PHC across the 4 clinics, and 104 (71.7%) patients agreed to participate. Although 68 (65.4%) of those 104 patients completed PHC, 29 (27.9%) did not complete the intervention, with Clinics C and D having particularly high patient incompletion rates of 39% (22/57) and 31% (5/16), respectively.

    Acceptability and Appropriateness

    Facilitating Provider-Patient Communication

    We asked key informants to what extent clinicians perceived that PHC was acceptable and appropriate and what factors shaped these perceptions. Overall, clinicians reported that the intervention was acceptable and expressed support for its use and enthusiasm about its potential to support patients and providers.

    This tool gave me an opportunity to understand that I probably need to do a better job of communicating with patients. In retrospect, the tool is good for the provider and if they look at what their patient’s concerns are and think it may change or may enhance the conversation they have with the patient.
    [Provider, Clinic D]
    Typically patients are so overwhelmed with their diagnosis that the tool could really help them break things down and see the information in a different way.
    [Provider, Clinic C]

    Clinicians at each of the four sites indicated that providers also would benefit from PHC because it supports interactions during the clinic visit, as facilitated largely by the tailored handouts. For example, several providers said that the handouts empowered patients to ask questions and identify information gaps that needed to be addressed. The handouts also gave providers a starting point for discussions with patients. In one case, the handouts led a patient to reveal an undisclosed sexually transmitted infection. A provider in Clinic A said, “I did use the handouts to see what their concerns were or what they wanted to know more about, and that changed my conversation.”

    Providers at two clinics expanded their notions of who would benefit from the intervention over the course of the implementation period. One saw the potential benefits expand by adapting the intervention for loved ones and by conducting group sessions. Initially, developers perceived PHC as a way to support the patient when interacting with their provider. Yet over the course of implementation, providers in three clinics increasingly viewed PHC as helping them understand patient concerns. One provider learned she had been inadequately addressing medication adherence with her patients. Before implementation, this provider stated the following:

    It is a great tool for patients, and also helps providers who don’t necessarily take the time to have conversations with patients that they should have. This may be the way for patients to get the information to start the conversation.
    [Provider, Clinic A]

    After implementation concluded, this provider was asked if her impression had changed.

    I still think the tool is great for patients to initiate a conversation. I still think this is a good way for patients to open up. Now, I think it’s actually an opportunity for providers to be a little more interactive and to dig a little deeper in conversations with the patient.
    [Provider, Clinic A]
    Engaging Functionality and Interactive Design

    At each of the four sites, clinicians and administrators reported PHC was an appropriate and acceptable intervention for helping PLH. In particular, the clinics liked the interactive components, such as patient selection of doctor and presentation of information in audio and video formats; the tailored messages based on patient responses; and information being presented in a clear and concise manner.

    What I really liked the most about the tool was the fact the patient could select who they wanted to hear... If they wanted to have a female provider, if they wanted to have a provider of color, if they wanted to have a male versus a female...
    [Provider, Clinic A]

    These are important intervention features because electronic, tailored, and interactive interventions have been shown to be effective as they provide more appropriate information to patients, compared with interventions that are not tailored [25]. However, we also noted concerns about the length of PHC (at two of the clinics); ability to deliver the patient and provider handouts; and ease of navigation, with the exception of password generation.

    Challenges to Acceptability

    Relatively low computer literacy diminished the extent to which PHC was deemed to be appropriate for the population served by one rural clinic. Because of provider concerns about patient literacy in general and computer literacy specifically, during the first 2 weeks of implementation in this clinic, PHC was offered only to patients who were able to use the intervention without assistance. PHC was then offered to all patients during the last 2 weeks of implementation. At this point, it became clear that patients with lower computer literacy required more assistance, primarily due to the complexity of password generation, which required more of the onboarder’s time:

    This [the tool] was very easy to use, but in rural areas like these many patients have no experience with the computer, at all. Many homes here don’t have Internet access...I think for people who don’t have any experience with computers, that they may not have really understood how the tool can be used...
    [Provider, Clinic A]

    Feasibility

    We also asked clinicians whether it was feasible to implement PHC as intended and what factors affected feasibility. Their responses revealed two main themes: clinic workflow and physical environment.

    Clinic Workflow

    Several factors related to clinic workflow presented implementation barriers. Respondents reported the challenge of scheduling patients to complete PHC without compromising tightly managed clinic workflows. Onboarders at Clinics A and B mentioned that PHC added 15 to 20 minutes to the time that patients spent in the clinic before seeing the provider, which caused delays. Additionally, onboarded patients were often interrupted to attend their provider appointments. Clinics C and D reported that clinic workflow processes interrupted patients engaging with the intervention. At Clinic C, patients were often called back to their appointment before the onboarder could deliver the handouts to the patient or provider. The onboarder from Clinic B described the experience fitting PHC into clinic workflow: “Possibly to try to get them [the patients] when they first come in the building as opposed to waiting for them to come into...the waiting room so they can view it [the tool] out there.”

    Three of the clinics onboarded patients into the intervention before their appointments. Two of these clinics had patients complete the intervention in exam rooms, and the third clinic had patients complete it in the waiting room, with the option of finishing in the exam room. Conversely, the fourth clinic onboarded patients after their appointments, in a room designated specifically for intervention use, which key informants said was arranged because of possible workflow disruptions if delivered before the visit.

    Sometimes patients can do the tool before they go back, but that’s if they are pretty early, not if they are right on time or late.
    [Onboarder, Clinic A]
    It would be much better if the clinic could have gotten the patient to use the tool before the visit... But the patients would need to come in early for their appointments.
    [Onboarder, Clinic C]
    Physical Environment

    The clinic physical environments also posed barriers that affected implementation feasibility. Two clinics were challenged by finding private space for patients to complete the intervention. At one clinic where patients were completing the tool in exam rooms, there were not enough rooms on particularly busy days.

    There were some cases during implementation where there was no space to complete the tool. If there was anything that hindered use of the tool, it would be the fact that it was very busy and there was overflow of patients.
    [Onboarder, Clinic C]

    Another challenge for three clinics was inconvenient locations for picking up handouts from the clinic fax printers. The location of the fax machine at the clinics, coupled with printing delays, led to handouts not being delivered consistently to patients or providers. To address some of these issues during implementation, we substituted the fax machines with wireless printers.

    Handout delivery methods and success rates differed across clinics. Three clinics delivered handouts to patients in exam rooms, and this approach worked relatively well for two of the clinics, where they delivered 91% (10/11) and 63% (5/8) of handouts; however, it posed challenges for the third, where they delivered only 50% (5/10) of handouts. In the fourth clinic, the printer was in the same room where patients completed the tool; consequently, they typically would retrieve their own handouts (8/8, 100%) when generated. These findings suggest that patient handout delivery is feasible, but delivery methods call for refinement.

    They would finish it [PHC] right before the provider came in, and once the provider is in there, I can’t give either patient or provider the handouts.
    [Onboarder, Clinic A]

    Discussion

    Principal Findings

    This pilot implementation evaluation demonstrated that in four clinics, HIV providers reported that the Web-based video tool PHC is an acceptable and appropriate intervention to supplement the support clinicians currently provide patients. They reported that PHC presents information in new ways that could strengthen patient-provider communication and help patients manage HIV. Clinicians indicated that provider handouts were useful because they increased their ability to address patients’ highlighted issues.

    Feasibility, or the ability to embed PHC in clinic workflow without disruption, proved to be more complex. PHC onboarders had to gain the support of other clinic staffers, manage scheduling patient tool use and the performance of digital tablets and Wi-Fi printers, assign patients passwords to log into the tool, and deliver PHC-generated handouts to patients and providers prior to scheduled appointments. Two clinics (A and B) facilitated patients’ completion of the tool at relatively high rates (approximately 93%), but they reported that implementation increased patient wait time by 15 to 20 minutes and users were sometimes interrupted in order to minimize delays in workflow. Two clinics (C and D) reported more struggles with implementation, as reflected in the lower tool completion rates (51% and 63%, respectively). These clinics reported that PHC users were often interrupted in order to attend their provider appointment. To solve this problem, Clinic C started onboarding patients after their provider appointment, which undermined timely use of the patient and provider handouts.

    PHC onboarders experienced difficulty in delivering patient and provider handouts as demonstrated by inconsistent delivery rates across clinics. The delivery of the provider handout was especially problematic due to the short amount of time between PHC completion and the start of the appointment. Onboarders had more success with the delivery of the patient handout due to the physical accessibility of the patient. We addressed two of the unanticipated barriers that slowed implementation efforts. The first was simplifying patients’ overly complex passwords requirements. The second was finding the correct technology to print patient handouts; consequently, during the pilot we provided wireless printers instead of fax printers.

    Clinic preparation to implement digital interventions requires extensive strategizing, trial and error, and coordination across many clinic staff. We believe that the 1-month time frame was an insufficient period of time for clinics to practice and finely hone these preimplementation strategies. In addition, there was insufficient time to pilot offsite PHC features that can alleviate some of the described barriers to implementation, where patients can finish the tool at home or request that their handout be emailed to them.

    Importantly, qualitative interview data show that clinic staffers presented solutions to many of the noted implementation barriers. These suggestions that can inform future PHC implementation include simplifying overly complex password requirements, meeting patients early in hallways by waiting rooms or asking patients to come to clinic early in order to engage with PHC. One suggestion requires more flexibility than was practical for the pilot—administering PHC after provider appointments where the handouts could be delivered at the next clinic appointment or sent to the user’s email address.

    Aspects of the intervention design that resonated with the clinic staff included the interactive video format presenting information tailored to each user. These are important intervention features: electronic, tailored, and interactive interventions have been shown to be effective because they provide more appropriate information to patients compared with interventions that are not tailored [25].

    This study underscores the concrete challenges of implementing digital interventions in complex and dynamic clinic environments, even when clinic stakeholders describe potential advantages and endorse the intervention. Pilot testing is critical to generate feedback from clinical stakeholders and produce implementation outcomes that will inform future implementation strategies in clinics.

    Limitations

    Several limitations pertain to this implementation pilot. First, we were able to include only the viewpoints of select key clinic staff at each site. Future assessments that rely on clinic staff should include a larger number over a longer period to better understand their viewpoints on implementation. Second, patients were engaged in piloting the intervention; however, we did not obtain user feedback on experience and satisfaction, which would be critical to inform future efforts to implement Web-based interventions. Finally, although the four participating clinics varied in their type of service area and population base, PHC acceptability, appropriateness, and feasibility in other large urban clinics or those with extremely low resources require further study.

    Despite these limitations, this pilot shows promise for the implementation of Web-based interventions like PHC. This work contributes to our understanding of clinic environments and strategies that support intervention implementation. Clinical settings are governed by complex workflow procedures and the need to follow regulatory guidelines and professional association best practices [26,27]. To facilitate the implementation of Web-based interventions that improve patient outcomes in clinical settings, clinicians need to address organizational workflow issues [28-32] and determine how these interventions can become a best practice. To close the gap in the HIV continuum of care for vulnerable populations, it is vital to understand and systematically study the implementation of Web-based interventions in clinical settings.

    Acknowledgments

    This work was supported by the CDC contract 200-2007-20016, task order 0025, to RTI International. PHC was developed and piloted by RTI International in partnership with the CDC under this contract. During the time of this contract, all authors were employees of either RTI or the CDC. The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the CDC. We would like to thank the clinical sites that participated in this pilot implementation.

    Conflicts of Interest

    None declared.

    References

    1. Fisher JD, Amico KR, Fisher WA, Cornman DH, Shuper PA, Trayling C, et al. Computer-based intervention in HIV clinical care setting improves antiretroviral adherence: the LifeWindows Project. AIDS Behav 2011 Nov;15(8):1635-1646. [CrossRef] [Medline]
    2. Gilbert P, Ciccarone D, Gansky SA, Bangsberg DR, Clanon K, McPhee SJ, et al. Interactive "Video Doctor" counseling reduces drug and sexual risk behaviors among HIV-positive patients in diverse outpatient settings. PLoS One 2008 Apr 23;3(4):e1988 [FREE Full text] [CrossRef] [Medline]
    3. Collins CB, Hearn KD, Whittier DN, Freeman A, Stallworth JD, Phields M. Implementing packaged HIV-prevention interventions for HIV-positive individuals: considerations for clinic-based and community-based interventions. Public Health Rep 2010;125 Suppl 1:55-63. [CrossRef] [Medline]
    4. Govindasamy D, Ford N, Kranzer K. Risk factors, barriers and facilitators for linkage to antiretroviral therapy care: a systematic review. AIDS 2012 Oct 23;26(16):2059-2067. [CrossRef] [Medline]
    5. Kempf M, McLeod J, Boehme AK, Walcott MW, Wright L, Seal P, et al. A qualitative study of the barriers and facilitators to retention-in-care Among HIV-positive women in the rural southeastern united states: implications for targeted interventions. AIDS Patient Care STDs 2010 Aug;24(8):515-520. [CrossRef] [Medline]
    6. Higa DH, Marks G, Crepaz N, Liau A, Lyles CM. Interventions to improve retention in HIV primary care: a systematic review of U.S. studies. Curr HIV/AIDS Rep 2012 Dec;9(4):313-325. [CrossRef] [Medline]
    7. Quanbeck A, Gustafson DH, Marsch LA, Chih M, Kornfield R, McTavish F, et al. Implementing a mobile health system to integrate the treatment of addiction into primary care: a hybrid implementation-effectiveness study. J Med Internet Res 2018 Jan 30;20(1):e37 [FREE Full text] [CrossRef] [Medline]
    8. Kilbourne AM, Neumann MS, Pincus HA, Bauer MS, Stall R. Implementing evidence-based interventions in health care: application of the replicating effective programs framework. Implement Sci 2007 Dec 09;2:42 [FREE Full text] [CrossRef] [Medline]
    9. Claborn KR, Fernandez A, Wray T, Ramsey S. Computer-based HIV adherence promotion interventions: a systematic review. Transl Behav Med 2015 Sep;5(3):294-306 [FREE Full text] [CrossRef] [Medline]
    10. Bachmann LH, Grimley DM, Gao H, Aban I, Chen H, Raper JL, et al. Impact of a computer-assisted, provider-delivered intervention on sexual risk behaviors in HIV-positive men who have sex with men (MSM) in a primary care setting. AIDS Educ Prev 2013 Apr;25(2):87-101. [CrossRef] [Medline]
    11. Hersch RK, Cook RF, Billings DW, Kaplan S, Murray D, Safren S, et al. Test of a web-based program to improve adherence to HIV medications. AIDS Behav 2013 Nov;17(9):2963-2976 [FREE Full text] [CrossRef] [Medline]
    12. Kurth AE, Spielberg F, Cleland CM, Lambdin B, Bangsberg DR, Frick PA, et al. Computerized counseling reduces HIV-1 viral load and sexual transmission risk: findings from a randomized controlled trial. J Acquir Immune Defic Syndr 2014 Apr 15;65(5):611-620 [FREE Full text] [CrossRef] [Medline]
    13. Noar SM, Black HG, Pierce LB. Efficacy of computer technology-based HIV prevention interventions: a meta-analysis. AIDS 2009 Jan 2;23(1):107-115. [CrossRef] [Medline]
    14. Fisher JD, Fisher WA. Changing AIDS-risk behavior. Psychol Bull 1992 May;111(3):455-474. [Medline]
    15. Miller W, Rollnick S. Motivational Interviewing. 3rd Edition. New York: Guilford Press; 2012.
    16. Prochaska JO, Velicer WF. The transtheoretical model of health behavior change. Am J Health Promot 1997;12(1):38-48. [Medline]
    17. Harshbarger C, Taylor O, Uhrig J, Lewis M. Positive health check: developing a web-based video counseling tool for HIV primary care clinics. J Comm Healthc 2017 Jul;10(2):70-77. [CrossRef]
    18. Hekler EB, Klasnja P, Riley WT, Buman MP, Huberty J, Rivera DE, et al. Agile science: creating useful products for behavior change in the real world. Transl Behav Med 2016 Jun;6(2):317-328 [FREE Full text] [CrossRef] [Medline]
    19. Proctor E, Silmere H, Raghavan R, Hovmand P, Aarons G, Bunger A, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Adm Policy Ment Health 2011 Mar;38(2):65-76 [FREE Full text] [CrossRef] [Medline]
    20. Proctor EK, Landsverk J, Aarons G, Chambers D, Glisson C, Mittman B. Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges. Adm Policy Ment Health 2009 Jan;36(1):24-34 [FREE Full text] [CrossRef] [Medline]
    21. Sawesi S, Rashrash M, Phalakornkule K, Carpenter JS, Jones JF. The impact of information technology on patient engagement and health behavior change: a systematic review of the literature. JMIR Med Inform 2016;4(1):e1 [FREE Full text] [CrossRef] [Medline]
    22. Yin R. Case Study Research: Design and Methods. Thousand Oaks: Sage Publications; 2014.
    23. Thomas G. How to Do Your Case Study: A Guide for Students and Researchers. Thousand Oaks: Sage Publications; 2011.
    24. Ritchie J, Spencer L. Qualitative data analysis for applied policy research. In: Bryman A, Burgess R, editors. Analysing Qualitative Data. London: Routledge; 1994.
    25. Noar S, Harrington N. eHealth Applications: Promising Strategies for Behavior Change. New York: Routledge; 2012.
    26. Aberg JA, Gallant JE, Ghanem KG, Emmanuel P, Zingman BS, Horberg MA, Infectious Diseases Society of America. Primary care guidelines for the management of persons infected with HIV: 2013 update by the HIV medicine association of the Infectious Diseases Society of America. Clin Infect Dis 2014 Jan;58(1):e1-e34. [CrossRef] [Medline]
    27. AIDSinfo: Clinical Guidelines Portal.   URL: https://aidsinfo.nih.gov/guidelines [accessed 2019-02-22] [WebCite Cache]
    28. Chen HT. The bottom-up approach to integrative validity: a new perspective for program evaluation. Eval Program Plann 2010 Aug;33(3):205-214. [CrossRef] [Medline]
    29. Glasgow RE, Lichtenstein E, Marcus AC. Why don't we see more translation of health promotion research to practice? Rethinking the efficacy-to-effectiveness transition. Am J Public Health 2003 Aug;93(8):1261-1267. [Medline]
    30. Harshbarger CL, O'Donnell LN, Warner L, Margolis AD, Richardson DB, Novey SR, et al. Safe in the city: effective prevention interventions for human immunodeficiency virus and sexually transmitted infections. Am J Prev Med 2012 May;42(5):468-472. [CrossRef] [Medline]
    31. Marley J. Efficacy, effectiveness, efficiency. Aust Prescr 2000 Dec 01;23(6):114-115. [CrossRef]
    32. Neumann MS, O'Donnell L, Doval AS, Schillinger J, Blank S, Ortiz-Rios E, et al. Effectiveness of the VOICES/VOCES sexually transmitted disease/human immunodeficiency virus prevention intervention when administered by health department staff: does it work in the "real world?". Sex Transm Dis 2011 Feb;38(2):133-139. [CrossRef] [Medline]


    Abbreviations

    CDC: US Centers for Disease Control and Prevention
    CWA: clinic Web application
    PHC: Positive Health Check
    PLH: people living with HIV


    Edited by G Eysenbach; submitted 04.04.18; peer-reviewed by C Lelutiu-Weinberger; comments to author 09.10.18; revised version received 25.01.19; accepted 26.01.19; published 18.04.19

    ©Camilla Harshbarger, Olivia Burrus, Brittany A Zulkiewicz, Alexa M Ortiz, Carla A Galindo, Bryan R Garner, Robert D Furberg, Megan A Lewis. Originally published in JMIR Formative Research (http://formative.jmir.org), 18.04.2019.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on http://formative.jmir.org, as well as this copyright and license information must be included.