Minor ailments are defined as nonserious health conditions that may be cared for by patients (eg, back pain, respiratory tract infections [RTIs], headache, and stomach upsets) [1]. However, an estimated 57 million unnecessary visits to general practitioners (GPs) in the National Health Service (NHS) occur every year in the United Kingdom as patients seek advice for managing these conditions [2]. The strain this places on primary care resources is well documented [3,4]; however, there is also a cost to the patient through increased anxiety, lowered confidence, and inconvenience [1]. Promoting self-care for these ailments would help alleviate the stress on both primary care and patients by helping patients understand and feel more enabled to cope with their health [3].

Many patients already use web-based resources for guidance with health issues [5]. However, credible, evidence-based interventions are needed to ensure that potentially serious infections are identified, and users are advised to consult a health care professional (HCP) when necessary. Interventions aimed solely at increasing users’ knowledge regarding their illness have shown only limited effects on increasing self-care [4]. Instead, calls have been made for interventions that address barriers to self-care, such as patients feeling distressed regarding their symptoms and not knowing how to treat them [4,6]. Theoretically-based digital health interventions have the potential to address these barriers and offer the advantage of providing ongoing support at a time and place that is convenient to the user. Internet Dr is a digital intervention that supports appropriate self-management of RTIs [7]. The intervention content is theoretically underpinned and contains tailored advice on self-caring for RTI symptoms, as well as a symptom checker to identify serious illnesses, including meningitis and sepsis [6] (see the Intervention subsection in the Methods section for more details). The content was designed to address previously identified barriers to self-care: (1) uncertainty regarding the need for medical treatment and (2) distress caused by the symptoms [4,6].

A randomized controlled trial (RCT) of Internet Dr was conducted over the winters of 2012 and 2013, with 3044 participants recruited randomly from lists of all patients registered at a selection of general practices in southern England. Users who had access to the intervention had fewer GP consultations for an RTI compared with those in the control group (239/1574, 15.18% vs 304/1664, 18.26%; multivariate risk ratio 0.71, 95% CI 0.52-0.98; P=.04), despite both groups having an equivalent occurrence of illnesses [7]. This means that more users in the intervention group decided to self-care for their symptoms. In addition to GP visits, self-reported scores for the patient enablement index (PEI) were also collected as an outcome measure at follow-up to capture the psychological benefits for patients using the intervention [8]. The PEI items asked users to reflect on perceived changes that occurred as a result of having participated in the study; for example, “thinking about the kinds of symptoms we have asked about in this study, compared with before you took part in this study, do you feel you are able to help yourself: same or less; better; much better?” Having previously focused on the intervention’s impact on health service use [7], a process evaluation of the RCT data is required to understand the psychological changes and behavioral engagement with the theoretically underpinned content of the intervention that led to the intervention group’s increased ability to self-care.

Process evaluations aim to provide insight into the parts of an intervention that work, for whom, and under what conditions [9,10]. This may be achieved by examining the underlying intervention mechanisms that are anticipated to lead to positive outcomes and the impact of context on the implementation of an intervention [9,10]. Logic models are often used to map the intervention content, theoretical underpinning, anticipated mechanisms of action, and outcomes [11], thereby identifying core research questions or hypotheses to address within a process analysis [12]. By explaining the mechanisms and effects of context, process evaluations have the potential to inform future intervention development and dissemination and advance our understanding of intervention theory [9,10].

Quantitative usage data collected automatically during interactions with a digital intervention (ie, log data) have the ability to provide a rich source of metrics for usage analyses [13-16]. Although widely used, broad, summative measures of usage, such as time spent or number of pages or components viewed in an intervention, have been criticized for their inability to explain how usage leads to positive outcomes [15,17-20]. In addition, the breadth of potential usage metrics available means that there is a danger that inferential analyses that examine all of these variables will produce results that do not relate meaningfully to the theoretically designed intervention architecture and are therefore unable to offer specific practical and actionable recommendations to optimize future intervention designs [18]. A clear rationale for choosing usage metrics is necessary to understand what is being measured and what can be inferred from analyses [19,21,22]. On that basis, arguments have been made to identify usage metrics that are meaningful to the intervention rather than data dredging [15]. This means determining types or patterns of usage that are specific to an intervention’s structure and the target behavior and are able to examine usage of theory-based content. For example, by isolating usage of a specific component [15,18]. For example, by isolating the use of a specific component or set of pages aimed at improving users’ self-efficacy for carrying out a target behavior, it is possible to examine the relationships between having viewed that component, reported changes in self-efficacy, and behavioral outcomes.

The framework for Analyzing and Measuring Usage and Engagement Data (AMUsED) [15] was developed to support systematic usage analyses of digital interventions by guiding researchers through 3 stages of planning and carrying out analyses. Stage 1 focuses on familiarization with the intervention architecture, including content, structure, and data collection. Through a list of generic questions in the first section of stage 2, researchers identify available metrics with which to measure usage, covering both summative measures (eg, number of times the intervention was accessed, completing the intervention, and amount of time spent) and more in-depth measures (eg, type, frequency, and completion of theoretically-based content). Researchers are then encouraged to consider these variables alongside the information from stage 1 and identify usage metrics that are relevant to the intervention structure, theory-based content, and target behavior and that are most likely to provide insight into how the intervention was effective and may be improved and implemented. Sections 2 and 3 of stage 2 then address how these metrics may be used in inferential analyses with self-report measures for user characteristics and target behaviors or outcomes. Stage 3 focuses on planning compatible data collection to ensure that analysis using appropriate analytical software is both possible and less onerous. The framework has previously been used to shape data collection for other digital interventions [15]; however, this process evaluation reports the first application to a usage analysis.

In line with the Medical Research Council guidance, this paper reports the process evaluation of the Internet Dr RCT, including a detailed usage analysis structured by the AMUsED framework [15]. The aims of the evaluation are (1) to identify measures of usage that are meaningful to the intervention and (2) to examine which parts of the intervention worked, for whom, and at what time. These results will generate guidance on how the design, implementation, and dissemination of Internet Dr can be improved but will also have generic implications for guiding the successful design and development of other digital interventions promoting self-care for minor ailments.

An open, pragmatic, parallel-group RCT of the Internet Dr digital intervention had been previously conducted [7]. After completing web-based enrollment in the study and baseline measures, participants were randomized using computer-generated random numbers to either the intervention group who had access to the website or the control group who did not. Having completed outcome measures at 24 weeks, participants in the control group were able to view the intervention. The full details of the Internet Dr RCT and findings from the primary analysis are available in the study by Little et al [7]. The study was registered with a trial registration number of ISRCTN91518452, and ethics approval from the South West Medical Research Ethics Committee, United Kingdom Health Departments’ Research Ethics Service, was obtained.

Adults (aged ≥18 years) registered with GPs within NHS Primary Care were recruited for the RCT by postal invitation. Patients with severe mental health problems or terminal illnesses were excluded. Participants needed to have access to the internet, with only 1 participant per household taking part. The process evaluation only examines participants who were randomly allocated to the intervention group and therefore had access to the intervention during the 24-week trial period.

A plan for conducting a complete process evaluation of data collected during the Internet Dr RCT was designed and conducted in line with the Medical Research Council guidance [12] using the AMUsED framework [15]. On the basis of the AMUsED framework checklists, the intervention’s structure, theoretical underpinning, and data collection points were collated (stage 1; Multimedia Appendix 1 [7,8,22-29]). All available usage metrics were considered in relation to the information in stage 1 to ascertain the types of usage that would be most meaningful to the intervention (stage 2, section 1; Multimedia Appendix 2). A comprehensive list of research questions was then generated to examine associations between the meaningful measures of usage, user characteristics, and outcomes (stage 2, sections 2 and 3; Multimedia Appendix 2). The questions were refined based on the logic model (see the Intervention section). The most appropriate analytical tools for examining the research questions were selected, and the necessary data preparation was considered (stage 3; Multimedia Appendix 3).

The process evaluation team combined expertise in psychology, primary care, statistical analyses, and computing. A total of 4 team members were previously unfamiliar with the intervention. The other 4 researchers had been involved in various stages of the Internet Dr development and primary outcome evaluation of the trial [7] and advised on the intervention content, logic model, and data capture processes and analyses. The first author (SM) had previously developed the AMUsED framework for application in process evaluations but was not familiar with the Internet Dr intervention.

Participants were requested to complete web-based baseline measures at the start of the trial, interim questionnaires every 4 weeks on RTI occurrence, and outcome measures at 24 weeks. Actual GP visits before and during the trial were collected after 1 year from participants’ GP records. Log data for individual users and sessions were collected during the trial (eg, pages accessed, time spent, and order of pages viewed).

Modifiable psychological characteristics thought to underlie decisions to self-care were measured at baseline and follow-up to capture any changes over the trial period that may help to explain outcomes (theory of planned behavior [TPB]) [22]: attitudes and norms; perceived behavioral control (PBC); and beliefs regarding the necessity of HCPs: health locus of control [23] and Krantz Health Opinion Survey [24]). Trait anxiety (Health Anxiety Inventory [25]) and intentions to use and follow intervention advice (TPB) [22] were measured at baseline. Experiences of accessing and using the intervention were collected at follow-up (Problematic Experiences of Therapy Scale [PETS]; [26]), along with the psychological outcome measure of how much users felt their ability to cope with an RTI had changed over the course of the trial (PEI; [8]). Full details of the psychological measures and response items are available in Multimedia Appendix 4 [8,16,19, 22-24].

All analyses examined data from only the intervention group. The LifeGuide Visualization Tool (University of Southampton) [32] was used to examine which pages had been accessed, at what point in the trial, and the number of users who had viewed them. Statistical analyses were conducted using SPSS for Windows (version 24; IBM Corp). All validated scales were used and scored according to the published guidance. Higher scores are indicative of positive change in all measures. Changes in scores for modifiable characteristics were calculated by subtracting individuals’ scores at baseline from their scores at follow-up. The frequency distribution of scores for constructs was visually assessed for normality; where these were inconclusive, Kolmogorov-Smirnov tests were performed. All scales were nonnormally distributed except for change scores. As each advice page was designed to be effective in isolation, as opposed to having an additive effect, a dichotomous categorical variable for users who viewed or did not view advice pages was calculated. All analyses were 2-tailed. Owing to a data collection error, it was not possible to analyze responses for the TPB construct of attitudes to using and following the advice at baseline and follow-up.

The distribution of scores and residuals for the PEI was positively skewed. Therefore, regression analyses were not possible for the PEI. Subsequently, 95% CIs were conducted to examine mean differences (MDs) in enablement based on usage and point biserial Spearman ρ correlation coefficient to examine the relationships between changes in user characteristics and enablement. Differences in scores for PETS based on usage were also examined using 95% CIs. Logistic regression was used to examine whether user characteristics at baseline predicted usage. Simple linear regression analyses were performed to examine whether usage predicted changes in modifiable characteristics.

A total of 31 general practices invited 43,769 patients to take part in the RCT. Of these 43,769 patients, 3044 (6.95%) consented to take part. Of the 3044 patients, 121 (3.98%) participants left their practice over the course of the study, leaving a total of 2923 (96.02%) users; of the 2923 participants, 1491 (51.01%) were in the intervention group, and 1432 (48.99%) were in the control group (Table 1). The patients’ GP notes showed that 18.04% (269/1491) of people in the intervention group had visited their GP for an RTI in the year before the study. Over the course of the trial, 57.14% (852/1491) of participants in the intervention group reported having an RTI.

This paper presents a process evaluation of data previously collected in an RCT of the Internet Dr intervention designed to enable users to appropriately self-care for RTIs [7]. The aims of the evaluation were to identify meaningful measures of usage (ie, types or patterns of interaction that are relevant to the structure and theory-based content of the intervention) with which to undertake a systematic process analysis and to examine the parts of the intervention that worked, for whom, and in what context.

This evaluation provides a clear example of when summative measures of usage (eg, number of log-ins and time spent on the intervention) would not provide the fine-grained details necessary to understand how the intervention worked; instead, it identifies usage metrics that are relevant to the structure and theory-based content of Internet Dr. With regard to the number of log-ins, these include users who logged in to complete interim study questionnaires as well as users who were accessing the intervention. Therefore, inferential analyses using the number of log-ins would capture users’ engagement with the trial as well as with the intervention. Alternatively, if the number of pages viewed had been analyzed and found to be associated with increased enablement, it is unclear what this would have meant or how it may be applied to improve the intervention. Each of the components varied in size, so that users who chose the Doctor’s Questions component may have viewed 5 pages and then left the component without having reached any advice on self-caring (Figure 1). In comparison, if users who visited the Common Questions component also viewed 5 pages, they would have been able to access 4 pages of advice. Without knowing what the content of the pages is, the number of pages viewed provides little insight into how users experienced increases in enablement or for improvements to the website. Instead, as the advice pages are the only content aimed at supporting users to self-care for their illness, viewing advice pages from the different components was identified as the most meaningful metric with which to analyze the use of the intervention. Users who viewed any advice page were more likely to report higher levels of enablement at 24 weeks compared with users who did not. Although this effect was fairly small, when analyzed by individual components, it was apparent that viewing an advice page from Treatment Options led to a practically significant increase in enablement. Therefore, viewing an advice page represents the minimal type and amount of usage required to improve outcomes and may be described as effective engagement with the intervention [18].

The 3 components were initially accessed by similar volumes of users, with almost half accessing >1 component. This suggests that offering a variety of content may be useful to maximize the number of users who reach the key features of an intervention. Most users (478/616, 77.6%) who viewed advice pages did so during their first log-in, having just completed the baseline questionnaire. Given that this figure represents more than half of all users in the intervention group who reported having an RTI during the 24-week trial (852/1491, 57.14%), it is unlikely that everyone accessing advice was experiencing an RTI at that precise time. Most if not all users will have experienced RTIs in the past and become familiar with any symptoms they found challenging. This prior experience could have supported well users to engage fully with the intervention and access pertinent advice without having to experience the symptoms at that point. This would also be the case for many other common ailments, which suggests that promoting intervention usage when users are well would be effective for self-caring for future minor ailments, as seen with RTIs.

As past behavior is typically a strong predictor of future behavior [33], and previous GP visits reinforce a patient’s decision to return to the GP in future [1], it was anticipated that users who had consulted their GP for an RTI in the year before the study would be less likely to use the intervention and view advice pages. However, the process analysis shows the reverse to be true, with these target users being more likely to view advice pages. As using Internet Dr has already been shown to lower the number of GP visits [7], it is probable that reaching these users was key to achieving this.

Viewing advice pages is important for increasing users’ enablement to self-care, with Treatment Options advice showing the greatest impact. However, users’ characteristics measured across the study provide only a limited explanation of the psychological changes that led to better enablement. Internet Dr content is underpinned by social cognitive theory (Treatment Options) and Leventhal’s common sense model [27] (Doctor’s Questions and Common Questions). The TPB was selected, along with measures of beliefs regarding the necessity of HCPs to manage illness to measure psychological changes across the study. Azjen cites Bandura’s definition of self-efficacy as the basis for the construct of PBC within the TPB [34]. Therefore, users who accessed Treatment Options content based on social cognitive theory were expected to report increased PBC; however, this effect was not found. Bandura [35] stipulates the use of measures of self-efficacy with social cognitive theory, and considering that PBC encapsulates several constructs [36], a general self-efficacy measure may be preferable for future studies of this nature [37]. Since the development of Internet Dr in 2008, further research has identified additional barriers to self-care (eg, perceptions of illness severity, not considering alternative options, and cost implications of paying for unprescribed medication [4]). Although these were not intentionally targeted within Internet Dr, the increases in enablement and the lower GP visits suggest that the theoretically underpinned content may have been effective in addressing some of these additional barriers. Future research, including measures of user characteristics that better reflect these barriers, may provide a more in-depth explanation of the association between usage and enablement.

In addition to examining users who viewed advice pages, it is also important to consider that approximately two-thirds of users in the intervention group did not access any advice pages. The first point of attrition for these users occurred at the Welcome Page, with just under one-third of users not progressing any further (Figure 3). An explanation for this might be that many of these users were not ill at the time and intended to return to the intervention if they experienced an RTI. However, these findings suggest that using advice pages when users are well can still be of benefit. Users who did not view advice pages were also more likely to report practical barriers to usage (eg, too tired or busy and forgetting). Applying these findings to future dissemination means that patients can be encouraged to access the intervention at a time that is convenient to them and not have to wait until they are experiencing symptoms. This could be reiterated to users by adding a message to the Welcome Page.

The second incidence of attrition occurred within the components, with almost another one-third of users starting a component but not reaching an advice page. The highest proportion of this attrition occurred in the Doctor’s Questions component, which is the largest component. This section includes a compulsory symptom checker with up to 10 pages of questions necessary in any intervention for minor ailments to ensure that serious infections in need of urgent medical attention are identified. Although these questions could not be omitted, the format and layout could be amended to minimize the burden on the user and subsequent attrition. For example, streamlining content by combining pages or motivating users to continue by including page numbers or breadcrumbs to show progress and location may have lowered attrition. Interestingly, these compulsory pages were not raised as a concern in qualitative evaluations conducted during the development phase [6]. This highlights the ability of process evaluations to establish design precedents from post hoc data analysis, such as ensuring that users are motivated and able to access the active ingredients of interventions with minimal effort.

Scores for increased enablement were low, with most users selecting same or less (0), which probably reflects that most users have experienced and successfully self-cared for RTIs previously. This is supported by the finding that users who had failed to self-care before the study were more likely to use advice pages. The PEI scoring was problematic as there were only 3 response options available, which did not allow users to distinguish between no change and deterioration in enablement. The resultant skew in scores meant that regression analyses were inappropriate for examining variables predicting enablement. Recent studies have measured PEI using Likert scales of ≥5, allowing for multiple, finer-graded levels of response [38,39].

The purpose of this study was to examine the psychological outcomes of intervention users and to explain the effectiveness of the intervention. As a result of the problematic PEI scoring, we considered examining the relationship between the usage of advice pages and the behavioral outcome of GP visits. However, this was not possible for several reasons. Almost half of the users who had viewed an advice page accessed advice from >1 component, and some users viewed several advice pages within a component, meaning that it was not possible to match the advice received to the action taken. In addition, most users had accessed advice before becoming ill. Finally, users who were advised to contact NHS Direct may have been recommended by NHS Direct to contact their GP; however, this was not captured in either self-report data or patients’ GP notes.

Initial RCT uptake by patients was 6.95% (3044/43,769), suggesting that participants were more willing to engage in this type of research and may not be representative of the wider population. Therefore, participant and nonparticipant characteristics were compared for the RCT analysis, and as the index for multiple deprivation showed that participants were less deprived than the wider population, the RCT results were controlled for this variable. However, as the process evaluation was a secondary analysis of the data collected in the RCT, identifiable details (ie, home address) were removed from usage data; as a consequence, it was not possible to control for possible effects that lower levels of deprivation may have had on behavioral engagement.

Internet Dr was developed and trialed >10 years ago, and since that time, digital intervention technology has advanced considerably. In addition to the recommendations from the usage analysis, before further dissemination activities are undertaken, the intervention would need further testing and development to ensure that current accessibility guidelines are met and the content is mobile friendly.

The findings from the process evaluation demonstrate the advantages of using systematic methods for analyzing digital intervention usage. By identifying specific metrics that are meaningful to the intervention structure, theory-based content, and target behavior, it was possible to examine how the intervention was effective, for whom, and in what context, and to provide specific recommendations for improving intervention design and implementation. Inferential analyses of usage identified that viewing advice pages from the Internet Dr intervention is effective at increasing the enablement of self-care for the symptoms of RTIs. Having identified content that is crucial for behavior change, this provides the opportunity to ensure that prior compulsory pages are streamlined to maximize the number of users reaching these active ingredients, thereby minimizing attrition. However, streamlining within components does not necessarily mean reducing the number of components available as users used the choice. These findings suggest that viewing advice pages before having an RTI encourages users to self-care for future symptoms. This means that for Internet Dr’s dissemination, users may be encouraged to access the intervention at their convenience rather than wait for the occurrence of an illness. The intervention was effective at reaching the target population of users who had previously failed to self-care for their symptoms and consulted their GP. Taking these findings into consideration, Internet Dr provides a model for future digital interventions aiming to increase self-care for other minor ailments.