Background: Mental health care counselors have a high intention to use e–mental health (EMH), whereas actual use is limited. Facilitating future use requires insight into underlying factors as well as eligibility criteria that mental health care counselors use in their decision to apply EMH.
Objective: The aim of this study was to unfold the intention and underlying reasons for mental health counselors to use EMH and to unveil the criteria they use to estimate patient eligibility for EMH. The theoretical framework was based on the reasoned action approach model, the Unified Theory of Acceptance and Use of Technology, and the Measurement Instrument for Determinants of Innovation model.
Methods: To empirically validate our theoretical model, a web-based survey was conducted among mental health care counselors (n=132). To unveil the eligibility criteria, participants were asked to rank their reasons for considering EMH suitable or unsuitable for a patient.
Results: The mean intention to use EMH was positive (mean 4.04, SD 0.64). The mean use of EMH before the COVID-19 pandemic was 38% (mean 0.38, SD 0.22), and it was 49% (mean 0.49, SD 0.25) during the pandemic. In total, 57% of the patient population was considered eligible for EMH. Usefulness and benefits (β=.440; P<.001), Task perception (β=.306; P=.001), and Accessibility (β=.140; P=.02) explained the intention to use EMH (F3,131=54.151; P<.001; R2=0.559). In turn, intention explained patient eligibility (F1,130=34.716; P<.001; R2=0.211), whereas intention and patient eligibility explained EMH use (F2,129=41.047; P<.001; R2=0.389). Patient eligibility partially mediated the relationship between intention to use EMH and EMH use, with a larger direct effect (c′=0.116; P<.001) than indirect effect (c=0.065, 95% CI 0.035-0.099; P<.001). Mental health counselors assessed patients’ eligibility for EMH mainly through the availability of computers and the internet and patient motivation.
Conclusions: To stimulate the use of EMH, intention and patient eligibility need to be influenced. Intention, in turn, can be enhanced by addressing the perceived usefulness and benefits of EMH, perceived accessibility, and task perception. Access to a computer and patients’ motivation to use EMH are important in facilitating patient eligibility. To cause an impact with EMH in general practice, mental health counselors need to be convinced of the benefits of EMH and transfer this enthusiasm to the patient. It is recommended to involve mental health counselors in the development of EMH to increase the (perceived) added value and use.
The number of patients who visit their general practitioner (GP) with psychological problems is growing rapidly and, consequently, waiting lists for treatment are increasing . For this reason, GPs and mental health care professionals search for ways to organize this care more efficiently. e–Mental health (EMH) care, often a combined approach with face-to-face care (ie, blended care), could be a solution [ ]. This study applied the following definition for EMH: “The use of information and communication technologies for patients with mental health complaints or disorders to inform and/or support them in recovery from their mental health to ultimately improve quality of life” [ ]. Interventions involve information and communication technologies, including treatment.
For the treatment of mental problems such as depression, there is convincing evidence of the effectiveness of EMH [- ]. Furthermore, a growing number of patients are positive about the incorporation of new remote technologies in health care for their convenience and flexibility and the possibility of following treatment at their own pace [ ]. In addition, the COVID-19 pandemic required reorganization of care as face-to-face contact was problematic and sometimes even impossible. To illustrate, 64% of Dutch GP practices started with videoconferences with patients during the COVID-19 pandemic [ ]. However, the structural implementation of EMH is still limited and faces many difficulties [ - ].
Mental Health Counselors and Adoption of EMH
In the Netherlands, mental health counselors (MHCs) working in general practices operate as gatekeepers in primary care concerning mental health problems [, ]. MHCs have different educational backgrounds—approximately 50% are sociopsychiatric nurses, 20% are psychologists, and 15% are social workers [ ]. These professionals treat patients with mild mental health problems and refer them to specialized care by licensed health care psychologists or psychotherapists in case of severe problems. They use EMH interventions as part of their tools to treat and coach patients. Although technical infrastructures and effective interventions are available, as well as sufficient reimbursement [ , ], actual use is low [ , ]. Facilitating future adoption and use requires insight into the most important underlying factors as well as the eligibility criteria MHCs use in their decision to apply EMH for their (vulnerable) patients. Hence, this study examined the use and nonuse of EMH by MHCs and aimed to unfold the underlying reasons and readiness to adopt EMH.
A study by Lokman et al  showed that 80% of GPs used EMH. Half of the GPs used EMH that was available via subscribed commercial eHealth platforms. The other half only referred patients to freely available self-help and psychoeducation websites. However, EMH was applied in <15% of the patients. According to the MHCs in that study, the currently available EMH was only suitable for one-third of their patients. Van der Vaart et al [ ] revealed that MHCs applied EMH more often than psychologists in basic mental health care (49% vs 21%).
According to MHCs, important facilitators of EMH were the following: the perceived benefits, the perceived enhancement of tools it provides to coach and treat patients, the related enrichment for their own work, and its potential to improve the quality of care. In the long term, it can also save time as patients can proactively work through certain assignments and read or reread information at home . Furthermore, MHCs considered themselves sufficiently digitally skilled and capable of providing EMH [ ]. However, almost half of MHCs expressed the need for a decision aid and information on the effectiveness of EMH applications [ ]. Impediments perceived by MHCs were as follows: the nonadherence of patients, the preference of patients for face-to-face contact, the insufficient possibilities MHCs perceive to be properly equipped to work with the eHealth platforms and the specific EMH applications, the mismatch between the supplied EMH materials and the patients’ needs, and the inflexibility of the EMH platform to attune the EMH content to the patients’ specific mental health problems and needs [ , ]. The most important reasons for the perceived mismatch were insufficient command of the Dutch language, low health literacy, and lack of a computer or low digital skills [ , , ]. MHCs often related these reasons to a low level of education. Furthermore, experienced ambiguity in regulations for the reimbursement of EMH also negatively affected the behavior of MHCs, which was still the case in 2019 [ ]. More studies have been conducted on facilitators of and barriers to EMH, although they were conducted among licensed psychologists [ ] and psychotherapists and often concerned specialized long-lasting psychotherapy [ ].
Vulnerable Patients Visiting the MHC and Their Eligibility for EMH
Healthy life expectancy and the prevalence of chronic diseases and mental health problems are strongly socially patterned, disproportionately affecting individuals with a lower socioeconomic position . These underprivileged individuals use EMH less frequently [ , ]. This is also the case for older adults [ - ] and individuals with severe mental health problems [ , ]. This is very undesirable and regrettable as EMH offers great opportunities given that interventions can be fluidly attuned to the needs of these specific groups through the presentation of bite-sized information in plain language accompanied by reading functions, appealing visuals and animations, and speech recognition [ ]. As previously mentioned, and of utmost importance, the growing use of EMH may provoke further socioeconomic health inequality [ , - ].
In the literature, this low use of EMH by patients in lower socioeconomic positions and senior citizens is often associated with insufficient digital, health-related, and reading skills [, , - ]. It is difficult for patients to find and use information via digital channels to adequately interpret and connect them with behavioral actions. As it is often an individual consideration and decision of the MHC whether EMH elements can be offered to a specific patient, it is important to gain insight into the criteria MHCs use in their consideration of patients’ eligibility for EMH. As there was no theoretical framework available at the time of the study, we developed one, as shown in . We combined the aforementioned aspects of vulnerability from the studies by Krijgsman et al [ ] and Wouters et al [ ], items from the fit-for-blended care checklist [ ], and relevant aspects derived from the studies by Titzler et al [ ] and Osma et al [ ] and from 3 orientational interviews with MHCs before our survey (unpublished).
Theoretical Model of Factors Associated With Behavioral Intention and Use of EMH by MHCs
To explain the behavioral intention and use of the MHC, a theoretical model was designed (). The model was composed of elements from the reasoned action approach (RAA) model [ ], the Unified Theory of Acceptance and Use of Technology (UTAUT) [ , ], the Measurement Instrument for Determinants of Innovation (MIDI) model [ ], and the Diffusion of Innovation Theory [ ]. In line with the RAA and UTAUT models, use of EMH is explained by behavioral intention to use EMH, whereas behavioral intention, in turn, is explained by the constructs Attitude (RAA), Social Influence (RAA and UTAUT) and Self-efficacy (RAA), Effort Expectancy (UTAUT), and Perceived Usefulness (UTAUT). With the aim of formulating a universal model, Venkatesh et al [ ] integrated in the UTAUT elements from 8 models, including the RAA model, to explain the acceptance and use of IT in organizations. Although the UTAUT has been used extensively also to explain the introduction of eHealth [ , , , ], it has been criticized for being too restricted to describe the technology acceptance of individuals [ , ]. As we agreed with this criticism and wanted to develop a model that would show all the key factors influencing the considerations and decisions of MHCs, insights from diffusion and implementation theories were considered to be crucial additions. Hence, elements from the MIDI (Characteristics of innovations) [ ] and Diffusion of Innovation Theory (Compatibility with current practice, Relative advantage, and Complexity) [ , ] were added, which led to further detailing of the rather general determinant Attitude and the selection of the factors of the construct Perceived properties of innovations. shows that our theoretical model, in line with the RAA and UTAUT, consists of 2 parts that will be empirically validated in this study. The first part explains the behavioral intention to use EMH (conceptual model A), and the second part explains the actual use of EMH by the MHC (conceptual model B).
Conceptual model B reflects that it will depend on the MHC’s assessment of the eligibility of the patient whether the behavioral intention is transformed into actual use of EMH. This could lead to a situation in which, although an MHC might be willing to use EMH, one could decide not to use it because of certain patient characteristics. This proposition is in line with the observation by Wouters et al  that MHCs are generally positive about EMH, although they apply it to a minority of their patients. On the basis of the UTAUT model [ , ] and previous studies [ , , , ], facilitating and impeding factors related to the organization of care in the GP practice were also considered as factors that directly influence the actual use of EMH. In line with Venkatesh et al [ ], the construct Facilitating conditions was defined as the degree to which an individual believes that an organizational and technical infrastructure exists to support the use of the system. Examples of facilitating or impeding conditions were ambiguity in reimbursement and the availability of time, management support, and information on the innovation. Although our study used the same starting definition for Facilitating conditions as Venkatesh et al [ ], the scale in the UTAUT questionnaire was operationalized in a different way. It measured not only the availability of resources and technical support (using the definition by Thompson et al [ ]), as in the Facilitating conditions scale in our theoretical model (model B), but also aspects of the self-efficacy of the user [ ].
Focus of the Study
This study aimed to explain the behavioral intention and actual use of EMH by MHCs working in general practices. Most previous studies [, , , , ] described the use of EMH and the reasons for its use by MHCs, but they have not analyzed the correlations between barriers and facilitating factors on the one hand and behavioral intention and use on the other. Others did analyze the predictors of adoption readiness or behavioral intention [ , , ] but did not assess EMH use itself. Our study addressed both the use of EMH and the factors that are associated with behavioral intention. Moreover, our study focused on MHCs and not (licensed) psychologists. To the best of the authors’ knowledge, no such studies have been conducted so far. An additional important aspect of our study is the specific interest in the application of EMH in vulnerable groups as the growing use of EMH might impede their access to health care to a greater extent compared with other groups [ , - ].
This study tested 5 hypotheses.
The first hypothesis stated that MHCs with a high behavioral intention to use EMH score significantly higher compared with those with a low behavioral intention on the following 10 factors: (1) perceived usefulness and benefits of EMH, (2) task perception of the MHC, (3) innovativeness of the MHC, (4) social influence experienced by the MHC, (5) self-efficacy of the MHC toward the use of EMH, (6) digital skills of the MHC, (7) evidence-based effectiveness of EMH, (8) compatibility with current practice, (9) perceived ease of use of EMH for the MHC, and (10) perceived ease of use of EMH for the patient [, , , ].
The second hypothesis proposed that perceived usefulness and benefits have the strongest association with intention to use EMH. The construct Usefulness and benefits is highly comparable with Performance expectancy of the UTAUT model. According to Venkatesh et al , this factor is proposed as the strongest predictor of behavioral intention to use technology in all technology acceptance models. Chismar and Wiley-Patton [ ] confirmed the highest importance of perceived usefulness compared with social influence and ease of use in their empirical study.
The third hypothesis was that there is a significant correlation between the behavioral intention to use EMH and actual EMH use and that the strength of this relationship is largely determined by the assessment made by the MHC of the patients’ eligibility. Building on the findings that the intention to use EMH is much higher than actual EMH use and that patient characteristics might relate to this disparity, the third hypothesis asserts that the association between intention and use is moderated by the estimated patient eligibility for EMH.
Hypothesis 4 stated that facilitating and inhibiting organizational circumstances in the general practice also have a significant relationship with the use of EMH by the MHC [, , , , , ].
According to hypothesis 5, the EMH eligibility assessment that MHCs conduct with regard to their patients includes primarily patient motivation to use EMH and the level of mental health problems. The hypothesized importance of patient motivation and the absence of disease-related contraindications (level of severity, lack of energy, lability, and suicidality) was based on the results of the qualitative study by Titzler et al . Concerning the importance of severity, Osma et al [ ] found therapists to have a positive intention to use EMH except in severe cases, such as psychosis, or if basic preconditions for the use of EMH are not met (eg, no internet access or insufficient literacy). Orientational interviews preliminary to our survey (unpublished) confirmed the importance of these factors.
As the validation of the questionnaire used to test the hypotheses was part of this study, the formulated hypotheses and developed model could undergo slight changes before the start of the analysis. The impact is discussed in the Methods section.
Research Design and Study Population
This was a cross-sectional study among MHCs using a web-based questionnaire (in LimeSurvey; LimeSurvey GmbH). Participants in the study had to be practicing MHCs working for at least 8 hours a week in a general practice.
Convenience sampling was used. The Dutch eHealth platforms Ksyos, Therapieland, and Minddistrict sent a newsletter and email message to their customer field (total >1000) to invite potential participants. However, as this did not lead to a sufficient number of respondents even after sending a reminder, an advertisement on the web page of the National MHC Association and a LinkedIn message were added. To encourage participation, 5 vouchers worth €10 (US $10.32) were raffled.
The web-based survey was reviewed and approved by the Ethical Review Committee of the Open University before the start (U/2020/01469/MQF). All participants gave their informed consent before taking part.
Questionnaire and Validation of the Questionnaire
The web-based questionnaire registered age, gender, type of general practice (with one or more MHCs), and educational background of the MHC (sociopsychiatric nurse, psychologist, social worker, or other) as well as the number of hours that the MHCs worked per week and their years of experience.
Factors Associated With Behavioral Intention and Use of EMH
In our model (, model A), the behavioral intention to use EMH is the dependent variable that is explained by 10 independent variables. In this model ( , model B), behavioral intention to use EMH, in turn, is one of the 2 variables (together with the factor Facilitating conditions) explaining EMH use. To measure the behavioral intention to use EMH, we formulated 4 items in a similar way as for Behavioral intention to use the system in the UTAUT questionnaire [ ]. The 4 items on this scale were scored by the MHCs on a 5-point Likert scale ranging from strongly disagree (1) to strongly agree (5). After item analysis, 3 of the 4 items remained, resulting in a reliable scale (3 items; Cronbach α=.85; ).
In accordance with the theoretical model (conceptual model A;), the 10 scales of the factors that are proposed to relate to behavioral intention to use EMH were measured as independent variables (Usefulness and benefits, Task perception of the MHC, Innovativeness of the MHC, Social influence, Self-efficacy of MHC, Digital skills of the MHC, Evidence-based effectiveness of EMH, Compatibility with current practice, and Ease of use for the MHC and for the patient). The factor Facilitating conditions was also measured as an independent variable, although it was directly related to EMH use (and part of conceptual model B; ).
To measure these 11 scales (ie, Facilitating conditions and the 10 scales of the factors that relate to behavioral intention), the items of the validated eMental Health Adoption Readiness scale by Feijt et al  were used as a starting point and complemented with items based on the MIDI questionnaire [ ], the UTAUT questionnaire [ ], a digital skill measurement tool, and own insights and literature (eg, related to the COVID-19 pandemic). To refine the items, our own scientific expertise [ , , ] and input from practice were important. The latter input was collected from one of the authors working at Therapieland and by conducting 3 interviews with MHCs to collect information on important barriers, facilitators, and other considerations for the implementation of EMH and to check the content and completeness of our questionnaire. Before completion, the questionnaire was pretested among 8 MHCs. All items included in the scales were scored on a Likert scale. To validate the subscales of the questionnaire, these 11 scales were subjected to a principal factor analysis [ ]. As the factor analysis led to some adjustments of the subscales (item composition, meaning, and sometimes factor name), the theoretical model and hypotheses were slightly modified. The changes in the theoretical model are shown in . The factor analysis revealed that items of the scales Usefulness and benefits of EMH and Compatibility with current practice had to be recategorized in the revised scale of Usefulness and benefits of EMH and the new scale New possibilities through EMH. In addition, the items of the scales Ease of use for the MHC and Ease of use for the patient were redistributed, and the scales were renamed (User-friendliness of EMH for both the MHC and the patient and Complexity for the patient). After these adjustments, the measurement scales of the 10 factors that were assumed to influence behavioral intention showed a reasonable to good internal consistency (Cronbach α≥.68 and ≤.89), as shown in . After validation, the Facilitating conditions measurement scale was also adjusted. Its reliability was sufficient (5 items; Cronbach α=.78; ). The content referred to organizational infrastructure with items on available time, finances, and administrative workload. In the factor analysis, External obligation and Accessibility emerged as new scales. The External obligation scale comprised 2 items that originally belonged to the Facilitating conditions scale. As this scale was not reliable (2 items; Cronbach α=.36; ), it was not included in the revised hypothesized model ( ). The Accessibility scale had good reliability (2 items; Cronbach α=.85; ). Accessibility comprised 2 items of the original Ease of use scales and was interpreted as complementary to Facilitating conditions as it measured whether EMH was easy to obtain and use (because of technical infrastructure). It was presumed to have a direct relation to EMH use as its content came close to the definition of Facilitating conditions by Thompson et al [ ]: “objective factors, ‘out there’ in the environment, that several judges or observers can agree make an act easy to do.”
For the factors from, mean scores were used in the regression model. The items that gave a negative evaluation of EMH were recoded. A high score (>3) indicates a positive evaluation by the MHC of that specific factor (eg, the MHC perceives EMH as useful and beneficial, and the MHC does not find that EMH uses complex language).
|Items, N||Cronbach α||Items|
|Factors explaining intention to use EMH|
|Usefulness and benefits of EMH||9||.89|
|New possibilities of EMH||5||.75|
|MHC task perception||3||.78|
|Innovativeness of the MHC||5||.80|
|Self-efficacy of the MHC||5||.81|
|Digital skills of the MHC||4||.89|
|Evidence-based effectiveness of EMH||2||.77|
|User-friendliness of EMH||5||.69|
|Complexity for patients||2||.68|
|Factors explaining EMH use|
|Intention to use EMH||3||.85|
aMHC: mental health counselor.
cGP: general practitioner.
dNot added to the theoretical model because of low reliability.
EMH Use and Patient Eligibility
EMH use and MHCs’ perception of patient eligibility were both operationalized as estimates by the MHC of the proportion of their patient population (1) who used EMH before the COVID-19 pandemic (ie, EMH use) and (2) whom they considered eligible for EMH (ie, patient eligibility). EMH use was measured using the question “Can you give an estimate of the proportion of patients for whom you have used eMH before the Covid pandemic?” MHCs’ perception of patient eligibility was measured using the question “Can you give an estimate of the proportion of your patients that are eligible for EMH?” Answering categories were 90% to 100%, 80% to 90%, 60% to 80%, 40% to 60%, 20% to 40%, 10% to 20%, and <10%. These population estimates were, in fact, alternative assessments as it was impossible in a regression model to relate the eligibility of an individual patient to the decision of the MHC to use EMH, as intended in the draft version of the conceptual model where EMH use was a dichotomous variable. To transform these categorical variables into ratio variables, the response categories were recoded as follows: <10% became 0.05, 10% to 20% became 0.15, 20% to 40% became 0.30, 40% to 60% became 0.50, 60% to 80% became 0.70, 80% to 90% became 0.85, and 90% to 100% became 0.95. The ratio variables were used in the regression model. Subsequently, the main reasons for (not) using EMH were asked using open questions. Respondents with increased use of EMH since the start of the COVID-19 pandemic were asked whether they expected that this increase would become permanent.
This study primarily aimed to explain EMH use before the COVID-19 pandemic. As the study started shortly after the beginning of the COVID-19 pandemic, some questions on the use of EMH during that period were also included. Social distancing in health care, as enforced by the Dutch government in March 2020, required GPs and MHCs to switch to remote patient contact as much as possible. This significantly increased the use of EMH in general practices .
All analyses were performed using SPSS software (version 26; IBM Corp). A 95% significance level was used for all tests (Cronbach α=.05) except for the 2-tailed t tests for hypothesis 1, where a significance criterion of P<.005 was used to correct for multiple testing (based on the Bonferroni correction).
To guarantee sufficient power for the regression analysis with 10 independent variables (m), the number of respondents (n), estimated by the rule of thumb of Tabachnick and Fidell , must exceed 130 (ie, 50+8 m).
The characteristics of the participants were analyzed using descriptive statistics. Correlation analyses were performed using all variables of the research model. To test hypothesis 1, a dummy variable was first calculated using a median split of EMH adoption readiness. Second, t tests were performed to reveal whether there were significant differences in the mean values of the 10 factors between respondents with high and low adoption readiness. To verify the second hypothesis, a multiple backward linear regression analysis was performed with behavioral intention as the dependent variable and the 10 factors from the research model (, model A) as predictor variables.
The third hypothesis consisted of 2 parts. The analysis started with the second part: patient eligibility is a moderator variable in the association between behavioral intention and EMH use. This was investigated with a moderator analysis using the Hayes PROCESS module —EMH use was the dependent variable (Y), behavioral intention was the independent variable (X), patient eligibility was the moderator variable (W), and Accessibility and Facilitating conditions were the covariates. Although this analysis also tested the first part of hypothesis 3 as well as hypothesis 4, a multiple linear regression analysis was also performed (backward) with EMH use as the dependent variable (Y) and behavioral intention, patient eligibility, and Accessibility and Facilitating conditions as independent variables as it facilitated the interpretation of the regression coefficients.
For the fifth hypothesis, descriptive statistics were used.
Description of Participants
The web-based questionnaire was filled out 146 times between April 21, 2020, and August 8, 2020. A total of 4.8% (7/146) of the respondents did not give informed consent and, of the 139 respondents who did give informed consent, 7 (5%) were not active as MHCs. Hence, 132 questionnaires were available for the analysis.
Of the 132 respondents, 111 (84.1%) were female, and the mean age was 47.4 (SD 10.7) years. They worked between 8 and 40 hours per week as MHCs in one or more general practices ().
|Age (years), mean (SD; range)||47.4 (10.7; 27-65)|
|Age (years; median split), median||49.0|
|Younger group (<50), n (%)||68 (51.5)|
|Older group (≥50), n (%)||64 (48.5)|
|Sex, n (%)|
|Professional background, n (%)|
|Sociopsychiatric nurse||26 (19.7)|
|Social worker||25 (18.9)|
|Work hours per week, mean (SD; range)||23.0 (7.6; 8-40)|
|Number of patients per week, mean (SD; range)||25.6 (10.5; 4-65)|
|Work experience (years), mean (SD; range)||4.8 (3.5; 0-20)|
|≤10, n (%)||128 (97)|
|≤2, n (%)||45 (34.1)|
|Number of MHCb colleagues, n (%)|
|Platform used, n (%)|
|Only Therapieland||41 (31.1)|
|Only Minddistrict||80 (60.6)|
|Therapieland and Minddistrict||9 (6.8)|
|Neither Therapieland nor Minddistrict||2 (1.5)|
aThis group was very diverse and comprised, among others, psychiatric nurses, applied psychologists, orthopedagogists, and ergotherapeutists.
bMHC: mental health counselor.
Variables and Correlation Analysis
In, the variables of the research model—mean values and SDs—are listed as well as the Pearson correlation coefficients between them. All Pearson correlation coefficients were below the critical value of 0.80, confirming the independence of the constructs.
The correlation analysis showed that behavioral intention correlated significantly with 8 of the 10 scales. Behavioral intention had a strong positive correlation with Usefulness and benefits of EMH and Task perception and a medium correlation with New possibilities, Innovativeness of the MHC, Social influence, and Evidence-based effectiveness. The correlations between User-friendliness and Complexity were low.
Hypothesis 1: Differences Between Groups With Low and High Behavioral Intention
Via a median split (median value of 4.0), the population was divided into a large group (98/132, 74.2%) with high behavioral intention (≥4; mean 4.32, SD 0.41) and a smaller group (34/132, 25.8%) with lower behavioral intention (<4; mean 3.25, SD 0.48). Using univariate t tests, the scores on the 10 factors were compared for the high– and low–behavioral-intention groups (). This partly confirmed the first hypothesis: MHCs with high behavioral intention scored significantly higher than MHCs with low behavioral intention on 6 of the 10 factors—Usefulness and benefits of EMH, New possibilities through EMH, Task perception of the MHC, Innovativeness of the MHC, Social influence, and Evidence-based effectiveness of the EMH—but not on perceived Self-efficacy, Digital skills, User-friendliness, or Complexityof EMH. These 6 factors were also the factors with a moderate to high correlation with EMH adoption readiness. The mean behavioral intention to use EMH in the total group was positive (mean 4.04, SD 0.64).
|All MHCs, mean (SD)||MHCs with low BI, mean (SD)||MHCs with high BI, mean (SD)||t test (df)a||P valueb|
|Usefulness and benefits of EMHc||3.80 (0.60)||3.19 (0.69)||4.01 (0.38)||−6.56 (40.2)a||<.001a|
|New possibilities through EMH||3.69 (0.60)||3.27 (0.53)||3.83 (0.56)||−5.12 (130)||<.001|
|Task perception of the MHC||4.02 (0.79)||3.29 (0.78)||4.27 (0.62)||−7.40 (130)||<.001|
|Innovativeness of the MHC||2.77 (0.76)||2.38 (0.82)||2.91 (0.69)||−3.67 (130)||<.001|
|Social influence||3.30 (0.66)||2.97 (0.63)||3.42 (0.63)||−3.55 (130)||.001|
|Evidence-based effectiveness of EMH||3.73 (0.58)||3.41 (0.72)||3.84 (0.49)||−3.19 (43.8)||.003a|
|Self-efficacy of the MHC||3.60 (0.73)||3.61 (0.77)||3.60 (0.72)||0.10 (130)||.93|
|Digital skills of the MHC||4.44 (0.59)||4.57 (0.51)||4.39 (0.61)||1.55 (130)||.12|
|User-friendliness for the MHC and the patient||3.58 (0.56)||3.48 (0.48)||3.61 (0.59)||−1.23 (130)||.22|
|Complexity for the patient||3.79 (0.65)||3.69 (0.69)||3.83 (0.64)||−1.04 (130)||.30|
aLevene test for unequal variances; in all other cases, equal variances.
bP<.005 to control for multiple testing.
cEMH: e–mental health.
Hypothesis 2: Factors Associated With Behavioral Intention to Use EMH
Factors associated with behavioral intention were examined using a multiple regression analysis (backward;). The first regression model (model 1; ) had 10 independent variables, and these factors explained 56% of the behavioral intention (F10,131=15.113; P<.001; R2=0.555). However, only the scales on Usefulness and benefits (b=.401; P<.001) and Task perception (b=.339; P=.001) were significant. The last model (model 9; ) explained 54% of the variance in behavioral intention (F2,131=76.102; P<.001; R2=0.541) with Usefulness and benefits (b=.471; P<.001) and Task perception (b=.316; P=.001). Hypothesis 2, which stated that the expected usefulness and benefits of EMH are the most important predictors of behavioral intention, was confirmed.
|R2||B (SE)||β||P value|
|Usefulness and benefits of EMH||0.425 (0.113)||.401||<.001|
|New possibilities through EMH||0.012 (0.080)||.011||.88|
|Task perception of the MHCc||0.274 (0.077)||.339||.001|
|Innovativeness of the MHC||−0.024 (0.065)||−0.028||.72|
|Social influence||0.034 (0.066)||.035||.61|
|Self-efficacy of the MHC||−0.057 (0.057)||−0.065||.32|
|Digital skills of the MHC||−0.012 (0.077)||−0.011||.88|
|Evidence-based effectiveness of EMH||0.054 (0.076)||.049||.48|
|User-friendliness for the MHC and the patient||0.089 (0.074)||.079||.23|
|Complexity for the patient||0.032 (0.065)||.033||.62|
|Usefulness and benefits of EMH||0.499 (0.094)||.471||<.001|
|Task perception of the MHC||0.255||.316||.001|
aThere was no multicollinearity (all variance inflation factor values were <4 and tolerance values were >0.20).
bN/A: not applicable.
cMHC: mental health counselor.
Hypotheses 3 and 4: Factors Explaining EMH Use
EMH use before the COVID-19 pandemic (mean 0.38, SD 0.22) and patient eligibility (mean 0.57, SD 0.23) were determined as percentages of the patient population from the MHCs’ estimation.
Using the Hayes PROCESS module, a moderator analysis was performed with EMH use as the dependent variable (Y), behavioral intention as the independent variable (X), patient eligibility as the moderator variable (W), and Accessibility and Facilitating conditions as covariates. The model was significant (F5, 126=17.246; P<.001; R2=0.406), but the product term X×W was not (F1, 126=3.121; P=.08; DR2=0.015). Therefore, patient eligibility was not a moderator and, thus, the second part of the third hypothesis was rejected. However, the first part of the third hypothesis was confirmed—both r=0.55 () and the PROCESS regression analysis (b=.140; P<.001) showed a significant positive association between behavioral intention and EMH use. Facilitating conditions (mean 3.38, SD 0.77) and Accessibility (mean 3.77, SD 0.68) were not significantly related to EMH use, nor was the correlation significant ( ). Hence, hypothesis 4 (Facilitating conditions and Accessibility have a significant relationship with EMH use) was rejected.
The backward multiple linear regression analysis that was also performed to facilitate the interpretation of the regression coefficients confirmed that only 2 variables were significant factors explaining EMH use: patient eligibility (b=.399; P<.001) and behavioral intention (b=.330; P<.001). Together, they explained 39% of the variance in EMH use (F2,131=41.047; P<.001; R2=0.389).
As patient eligibility was not a moderator variable but a direct scale of EMH use, the Hayes PROCESS module was used to examine whether it could be a mediator variable instead. The results are shown in(model B). Behavioral intention explained patient eligibility (F1,130=34.716; P<.001; R2=0.211), whereas behavioral intention and patient eligibility explained EMH use (F2,129=41.047; P<.001; R2=0.389). Hence, it was concluded that there was a partial mediation between behavioral intention and EMH use through patient eligibility. The direct effect (c’=0.116) of behavioral intention was larger than the indirect effect (c=0.065, 95% CI 0.035-0.099). All coefficients were significant, with P<.001.
Verification of the Complete Model
To verify the complete theoretical model (ie, models A and B incombined), an additional multiple regression analysis (backward) was performed for behavioral intention. In addition to the remaining factors Usefulness and benefits of EMH and Task perception of the MHC (model 9; ), the 2 factors for which a direct relationship with EMH use was assumed (Facilitating conditions and Accessibility) were included in the regression analysis. It turned out that the scales Usefulness and benefits (b=.440; P<.001), Task perception (b=.306; P=.001), and Accessibility (b=.140; P=.02) were significant explanatory factors of behavioral intention (F3,131=54.151; P<.001; R2=0.559). The empirically validated theoretical model in which only the significant factors related to EMH adoption readiness and use were considered is shown in .
Increased Use of EMH Since the Start of the COVID-19 Pandemic
The use of EMH before and since the start of the COVID-19 pandemic was compared. The mean use before the COVID-19 pandemic was 38.1% (mean 0.381, SD 0.219), and it increased to 49.4% after the start of the COVID-19 pandemic (mean 0.494, SD 0.247). There was a significant and strong correlation (r=0.794; P<.001) between use before and use since the start of the COVID-19 pandemic. More than half (70/132, 53%) of the MHCs had increased the use of EMH since the start of the COVID-19 pandemic, and this increase was also expected to be permanent for slightly more than half (36/70, 51%) of this group.
Hypothesis 5: Factors Taken Into Account by MHCs in Assessing Patient Eligibility for EMH
The answers to the question asking to make a top 5 of the most important characteristics to be perceived as not eligible for EMH are summarized in. In this graph, it is shown how often the items were chosen by respondents as top 1, top 2, top 3, top 4, and top 5. The order in which the characteristics are presented is considered the order of importance and is determined by the calculated weighted average score (weight factors: top 1×5, top 2×4, top 3×3, top 4×2, and top 5×1). A patient having no access to a computer or the internet was considered to be the most important reason to deem them not eligible for EMH. The total score for this characteristic was 389 (ie, 5×50+4×16+3×13+2×7+1×22, with 50 being the number of times it was chosen as top 1 by the respondents, 16 being the number of times it was chosen as top 2, and so on). No motivation came in the second position with a total score of 325, followed by not speaking Dutch (whereas EMH programs are offered in Dutch) with a total score of 315.
Regarding the question about the minimum properties or characteristics that a patient needs to have to be eligible for EMH, access to a computer and the internet was chosen 114 times, motivation to work with EMH was chosen 108 times, openness to new experiences was chosen 61 times, reading and writing skills (in Dutch) was chosen 57 times, sufficient cognitive skills was chosen 38 times, discipline was chosen 36 times, demonstrable computer skills was chosen 30 times, other was chosen 6 times, and demonstrable health literacy was chosen once.
The availability of a computer and the internet showed the highest scores in both assessments. When considering the patients’ attributes or skills, patient motivation for EMH use emerged as the most important factor. The severity of the mental health problems was given a low score. Therefore, the fifth hypothesis was partly rejected.
Many patient-related answers were given to the open questions about the main reason for (not) using EMH. This not only showed that the patient plays an important role in MHCs’ decision to use EMH, as was confirmed by the regression analysis, but also provided information about the factors that determine the assessment of the patients’ eligibility. Again, a lack of digital skills, limited language skills, and lack of motivation on the part of the patient were the predominant reasons. Interestingly, 10.6% (14/132) of MHCs mentioned the lack of a suitable EMH module for the patient in question.
This study aimed to unfold the underlying reasons of MHCs to adopt and use EMH as well as unveil the criteria MHCs use to estimate patient eligibility for EMH. The factor Perceived usefulness and benefits was the strongest predictor of behavioral intention to use EMH. In turn, behavioral intention had a direct and indirect effect (via estimated patient eligibility for EMH) on the use of EMH. To estimate patient eligibility for EMH, patients’ access to a computer and sufficient digital and language skills and motivation to use EMH were important.
In this study, the intention to use EMH was high, which is important to further disseminate and embed EMH use in practice as intention positively related to use in our study. The fact that intention to use EMH is high among MHCs is in line with previous findings in the Netherlands [, ] but could be due to the broad definition of EMH and the fact that especially MHCs who are positive toward EMH might have been more likely to participate in this study as other studies have revealed that intentions varied between EMH applications [ ] or were higher for the treatment of mild mental problems [ , ].
During the COVID-19 pandemic, EMH use was seen to increase from 38% on average to 49%; both percentages were already considerably higher than in 2016, when it was <15% . More than half (70/132, 53%) of the MHCs reported increased use of EMH since the start of the COVID-19 pandemic, which was often expected to be permanent. It would be interesting to investigate over time whether this expectation also came true.
MHCs with a high behavioral intention scored significantly higher for 6 of the 10 factors (Usefulness and benefits of EMH, New possibilities through EMH, Task perception of the MHC, Innovativeness of the MHC, Social influence, and Evidence-based effectiveness of EMH) compared with those with a low EMH use intention. The results of the regression analysis stress the importance of Usefulness and benefits of EMH, Accessibility, and Task perception to increase MHCs’ intention to use EMH. Usefulness and benefits of EMH appeared to be the most important predictor of behavioral intention. Accessibility and Task perception were also predictive, although less strong. The primary importance of Usefulness and benefits is in line with relevant literature on the introduction of IT as well as in general  as applied to health care environments [ , ]. As a construct, Usefulness and benefits of EMH is closely related to Performance expectancy from the UTAUT model. Performance expectancy was defined as the extent to which a person believes that using the IT system will improve their work performance and is advantageous for patients, and it was also found to be the determinant variable of intention to use [ ]. In applying the technology acceptance model, Chismar and Wiley-Patton [ ] found that Perceived usefulness had a significantly strong effect on the intention of pediatricians to use internet-based applications, but Subjective norm and Perceived ease of use did not. Van der Vaart et al [ ], who used the UTAUT model in their study on the use of web-based self-management interventions, also found Performance expectancy to be the most significant predictor of intention to use for both MHCs and primary care psychologists.
The second significant factor of behavioral intention in our study (ie, Accessibility) is not 100% equal to any of the scales of the UTAUT model, although it is related to Effort expectancy. Effort expectancy was defined by Venkatesh et al  as the degree of ease of use and is related to Accessibility, User-friendliness, and Perceived self-efficacy in our models A and B ( ). Apparently, we need 3 factors in our model ( ) to do justice to the different aspects of Effort expectancy by Venkatesh et al [ ]. In the study by van der Vaart et al [ ], Effort expectancy was a significant predictor of intention to use for MHCs but not for primary care psychologists. Perceived ease of use was not significant in the study by Chismar and Wiley-Patton [ ] among pediatricians. These inconsistent results may be indicative of a construct that is formulated rather broadly. Still, there is clear consensus among the studies regarding the dominant role of perceived usability over ease of use. Only if users see added value in the use of EMH will they consider using it, and only then will ease of use become important [ ].
The third significant factor of behavioral intention in our study (ie, Task perception) does not exist in the UTAUT model and, therefore, is not in the study by van der Vaart et al  either. However, Task perception relates very well to Job relevance, which was the factor determining perceived usefulness in the study by Chismar and Wiley-Patton [ ]. In this study, the correlation between Task perception and Usefulness and benefits emerged as the strongest, confirming a close relationship.
It is interesting to note that Social influence was not a significant variable of behavioral intention, which is in line with the studies by van der Vaart et al  and Chismar and Wiley-Patton [ ]. Seemingly, MHCs, primary care psychologists, and pediatricians make up their minds independent of others’ opinions and, therefore, do not need emphasis on influencing intentions to use EMH.
Correlation and regression analyses showed that MHCs with a higher behavioral intention were more likely to use EMH more frequently than their colleagues with a lower behavioral intention. The estimated eligibility of the patient population did not appear to be a moderator variable in the relationship between behavioral intention and EMH use. However, patient eligibility was found to be both a direct factor of EMH use and a mediator variable between behavioral intention and EMH use. The direct effect of patient eligibility on the use of EMH is larger than the direct effect of behavioral intention, which makes it the most determining factor in the decision of the MHC whether to use EMH. It is interesting, though unexpected, that the behavioral intention of MHCs influences their estimation of the eligibility of the patient population. This issue will be discussed later in this section. The question about the inverse relationship (ie, whether the patients’ eligibility influences behavioral intention) was not investigated as it is incompatible with the conceptual model. In this model, behavioral intention is a perception of the MHC that already exists before the MHC estimates the eligibility of the patient population involved.
The motivation of the patient emerged as the most important condition to consider a patient eligible for EMH in addition to access to a computer with internet and digital skills. Knowledge of the Dutch language and good reading skills were also high in ranking. The severity of the mental health problems was given a much lower ranking, which was unexpected considering the literature [, ]. The fact that MHCs in our study considered the severity of the patients’ problems less of a barrier to using EMH may be explained by the fact that their average patient has mild to moderate mental problems compared with the patients with depression in the studies by Titzler et al [ ] and Osma et al [ ]. In the Netherlands, only mild to moderate mental problems are treated in general practice. Severe cases are treated in a specialized practice. The importance of motivation of the patient was confirmed in the answers to the open questions about the most important reasons for (not) using EMH in our survey and by the study by Wouters et al [ ].
In this study, the MHCs estimated 57% of their patient population to be eligible for EMH. This is quite a high estimate compared with the 33% in the study by Lokman et al . It is interesting that the intention of the MHCs to use EMH influences this estimate, as the mediation analysis revealed. This would mean that an MHC with a low intention to use EMH will be inclined to disqualify his patients for EMH. In this respect, the MHC can be considered an intermediary who plays a key role in the success of the implementation of EMH. From theory and experiments, it is known that success largely depends on the engagement of intermediaries or champions in the early stages of implementation [ , , ]. Only an enthusiastic MHC will offer the intervention (EMH) to the patient and will be able to motivate them [ ]. Owing to the emphasis that MHCs place on the motivation of the patient, the characteristics of vulnerable patients presumably received a lower ranking. Nevertheless, it became clear that MHCs will not yet use EMH for patients who lack computer and language skills.
Causal relationships were assumed in this study. However, a causal relationship cannot be established in this type of study. Nevertheless, as the research model is based on theoretical models that have already been empirically verified, this study does provide indications for the assumed causal relationships among EMH use, intention to use EMH, and their determinants.
A second limitation concerns the questionnaire, which, although based on validated questionnaires, was developed specifically for this study, which makes it difficult to compare results with the literature. On the positive side, the reliability of most of the scales proved to be good, and only small adjustments were needed to the predefined scales.
Third, the results of this study might be positively biased as convenience sampling was used and participants were mostly clients of Therapieland or Ksyos and Minddistrict. Obviously, MHCs who were positive about EMH and were already working on the web were more likely to complete the questionnaire than those who were less enthusiastic about web-based activities (self-selection bias). In our study, >98% (130/132, 98.5%) of the MHCs had access to a paid EMH platform, whereas, in the study by Lokman et al , approximately half of the GPs did not use purchased EMH. Unfortunately, the lack of insight into the response rate impeded a good estimate of the selection bias that might have occurred. A related issue is that the percentage of women in our sample was high (111/132, 84.1%). However, the occupation of MHC is female-dominated in the Netherlands. A recent (2021) Dutch publication revealed that 71% of MHCs are female [ ].
Fourth, there may also have been response bias as self-report questionnaires with Likert scales were used that respondents filled in according to their own interpretation. A statement that is formulated in a fairly general way is perhaps all too easily answered with agree. This applies, for example, to the 2 items in the Accessibility scale. Feijt et al  came to the same conclusion for these items and removed them from the eMental Health Adoption Readiness scale for this reason. Although Accessibility is substantively related to User-friendliness, it emerged as a separate factor in the factor analysis. In spite of the fact that the reliability was good, there is some doubt regarding its content validity.
The final limitation concerns the difficult interpretation of the intention to use EMH, actual EMH use, and patient eligibility for EMH because of the broad definition of EMH we used (ie, ranging from means of communication to web-based treatment modules). Although the questions were formulated as specifically as possible by adding an example or by speaking of EMH program for therapy using web-based programs, the broad concept was used in the measurements of behavioral intention, EMH use, and patient eligibility. The research by van der Vaart et al  explicitly focuses on the use of a specific part of EMH (ie, the guided web-based self-management interventions in primary care). Therefore, the conclusions of that study are not entirely comparable with those described in this study.
Recommendations for EMH Practice and Researchers
This study revealed that usefulness and benefits are by far the most important factors influencing the intention to use EMH. Only if potential users see added value in the use of an IT system will they consider using it, and only then will user-friendliness also become important . Therefore, the EMH developer must ensure that EMH has added value for MHCs and patients. This is possible, for example, by developing EMH applications in cocreation with MHCs. Additional qualitative research would be ideally suited to obtain more specific input from MHCs (eg, to understand for which patients MHCs did not find a suitable EMH module, what modules are needed, and what characteristics they need to have). The research by Titzler et al [ ] among psychotherapists is a good example of structured interviews that provide useful insights for EMH developers. It would also be good to actively involve the patients as that would help increase patient motivation while at the same time allowing for the determination of whether patient motivation is a real problem and not something that is wrongly estimated or negatively influenced by the MHC or whether the modules themselves are the problem. As the MHCs’ personal motivation to support patients with EMH influences their estimation of patients’ eligibility, it is important to make MHCs aware of this process and provide skill training on motivating patients regarding disease self-management and to use EMH. Motivational interviewing skills are posed as a helpful strategy.
This study also showed the importance of task perception of the MHC with regard to EMH. Government policy is necessary to structurally embed EMH in the task perception of the MHC (eg, by promoting digitalization in health care, providing the right reimbursement for EMH activities in general practice, and making EMH part of higher education curricula ).
The study results are also useful for the broader population of mental health care professionals, especially the findings with regard to eligible patients and the way in which the intention of health care professionals to use EMH affects their estimation of patient eligibility.
For future use of the questionnaire, specifically in questions on EMH use and patient eligibility, we recommend differentiating between the use of web-based communication and the use of web-based treatment modules. Better ways to determine the factors related to patient eligibility would also be worth investigating.
Recommendations for future research include a longitudinal study to verify the relationships we found in this study and investigate whether EMH use has maintained its upward trend after the COVID-19 pandemic. It would be especially interesting to further explore the influence of behavioral intention on perceived eligibility also among other professionals working in mental health care as it is important for practice to make professionals aware of the fact that personal motivation to support patients with EMH also influences the way in which patients’ motivation and skills to use EMH are estimated.
It can be concluded that the intention to use EMH among MHCs was very positive. The most important factors explaining the intention to use were the perceived usefulness and benefits of the use of EMH followed by task perception and accessibility (ie, ease of getting started with it).
The relationship between behavioral intention to use EMH and actual EMH use was partially mediated by the perceived eligibility of the patient population. However, both the behavioral intention to use EMH and the estimated eligibility of the patient population had a significant and direct association with EMH use. The patients’ eligibility was most important, which means that an MHC will use little EMH if they consider the patient unsuitable for EMH even if the MHC is positive about the use of EMH.
To determine whether a patient is eligible for EMH, the patients’ access to a computer and the internet, digital skills, and Dutch language skills were primarily considered. In addition, patient motivation was found to be of utmost importance.
The study revealed that there will only be a future for blended care if the MHC is convinced of the added value of EMH and can transfer their enthusiasm to the patient.
Therapieland, Minddistrict, and the respondents of the questionnaire are acknowledged for their participation in this study. AEMDV thanks Milou Feijt for interesting discussions and the use of the eMental Health Adoption Readiness questionnaire.
The research was initiated by CAWB, VT, and MGS and supervised by CAWB and VT. AEMDV created the web-based survey, gathered the data (with the help of Therapieland and Minddistrict), and performed the statistical analyses. The first draft of the manuscript was written by CAWB (Introduction) and AEMDV (all the other sections). Revisions were mainly made by CAWB and AEMDV. All authors read, modified, and approved the final manuscript.
Conflicts of Interest
MGS is working at Therapieland, in a company that develops and sells e–mental health applications, but he has no personal financial interest related to the subject of this study. The authors have no further conflicts of interest to declare.
Variables of the research model: mean, SD, and Pearson correlation coefficient.PDF File (Adobe PDF File), 171 KB
- Dutch Healthcare Authority. Factsheet waitinglist mental health care 2021 [wachttijden GGZ 2021]. Government of the Netherlands. 2021 Jul 01. URL: https://www.rijksoverheid.nl/documenten/publicaties/2021/07/01/factsheet-wachttijden-ggz-2021 [accessed 2021-09-13]
- van Straten A. E-mental health to bridge the waiting time in secondary mental health care. Directieve Therapie 2013;33(3):251-252. [CrossRef]
- Akwa GGZ (Akwa Mental Health Care). eHealth. Mental Health Care standards [GGZ Standaarden]. 2017. URL: https://www.ggzstandaarden.nl/generieke-modules/ehealth/introductie [accessed 2020-03-29]
- Ebert DD, Berking M, Cuijpers P, Lehr D, Pörtner M, Baumeister H. Increasing the acceptance of internet-based mental health interventions in primary care patients with depressive symptoms. A randomized controlled trial. J Affect Disord 2015 May 01;176:9-17. [CrossRef] [Medline]
- Karyotaki E, Riper H, Twisk J, Hoogendoorn A, Kleiboer A, Mira A, et al. Efficacy of self-guided internet-based cognitive behavioral therapy in the treatment of depressive symptoms: a meta-analysis of individual participant data. JAMA Psychiatry 2017 Apr 01;74(4):351-359 [FREE Full text] [CrossRef] [Medline]
- Königbauer J, Letsch J, Doebler P, Ebert D, Baumeister H. Internet- and mobile-based depression interventions for people with diagnosed depression: a systematic review and meta-analysis. J Affect Disord 2017 Dec 01;223:28-40. [CrossRef] [Medline]
- Andersson G, Cuijpers P, Carlbring P, Riper H, Hedman E. Guided internet-based vs. face-to-face cognitive behavior therapy for psychiatric and somatic disorders: a systematic review and meta-analysis. World Psychiatry 2014 Oct;13(3):288-295 [FREE Full text] [CrossRef] [Medline]
- Ebert DD, Buntrock C, Lehr D, Smit F, Riper H, Baumeister H, et al. Effectiveness of web- and mobile-based treatment of subthreshold depression with adherence-focused guidance: a single-blind randomized controlled trial. Behav Ther 2018 Jan;49(1):71-83. [CrossRef] [Medline]
- van Tuyl LH, Batenburg R, Keuper JJ, Meurs M, Friele R. Increased use of e-health in general practice during the corona pandemic. Organization of remote care in corona time [Toename gebruik e-health in de huisartsenpraktijk tijdens de coronapandemie. Organisatie van zorg op afstand in coronatijd ]. Netherlands Institute for Health Services Research. 2020. URL: https://www.nivel.nl/nl/publicatie/toename-gebruik-e-health-de-huisartsenpraktijk-tijdens-de-coronapandemie-organisatie-van [accessed 2020-08-26]
- Krijgsman J, Swinkels I, van Jong LB, de Out J, Out K, Friele R, et al. More than technology: eHealth monitor 2016. NIVEL. 2016 Oct 6. URL: https://nivel.nl/nl/publicatie/meer-dan-techniek-ehealth-monitor-2016 [accessed 2019-08-15]
- Wouters M, Swinkels I, Sinnige J, de Jong JJ, Brabers A, van Lettow L, et al. Kies bewust voor eHealth. NIVEL. 2017 Nov. URL: https://www.nivel.nl/sites/default/files/bestanden/Nictiz_eHealth_monitor_2017.pdf [accessed 2019-08-15]
- Titzler I, Saruhanjan K, Berking M, Riper H, Ebert DD. Barriers and facilitators for the implementation of blended psychotherapy for depression: a qualitative pilot study of therapists' perspective. Internet Interv 2018 Jun;12:150-164 [FREE Full text] [CrossRef] [Medline]
- Jacob C, Sanchez-Vazquez A, Ivory C. Social, organizational, and technological factors impacting clinicians’ adoption of mobile health tools: systematic literature review. JMIR Mhealth Uhealth 2020 Feb 20;8(2):e15935. [CrossRef]
- Remote care closer? Digital care after the corona crisis. Raad Volksgezondheid & Samenleving. 2020 Aug 27. URL: https://www.raadrvs.nl/documenten/publicaties/2020/08/27/zorg-op-afstand-dichterbij [accessed 2020-09-05]
- van der Vaart R, Atema V, Evers AW. Guided online self-management interventions in primary care: a survey on use, facilitators, and barriers. BMC Fam Pract 2016 Mar 09;17:27 [FREE Full text] [CrossRef] [Medline]
- Verhaak P, van Beljouw I, Have JT. De opmars van de POH-GGZ. Tijdschrift voor Praktijkondersteuning 2010 Oct;5(5):130-134. [CrossRef]
- Topooco N, Riper H, Araya R, Berking M, Brunn M, Chevreul K, E-COMPARED consortium. Attitudes towards digital treatment for depression: a European stakeholder survey. Internet Interv 2017 Jun;8:1-9 [FREE Full text] [CrossRef] [Medline]
- Conijn B, van der Poel K, Abello K, Boon B. Infographic e-mental health by GP and POH-GGZ. Trimbos. 2015. URL: https://www.trimbos.nl/aanbod/webwinkel/product/af1364-infographic-e-mental-health-door-huisarts-en-poh-ggz [accessed 2019-12-01]
- Lokman S, van der Poel A, Smeets O, Martin K. E-mental health in de huisartsenpraktijk. Trimbos. 2016. URL: https://www.trimbos.nl/aanbod/webwinkel/product/inf010-e-mental-health-in-de-huisartsenpraktijk [accessed 2019-08-15]
- Kip H, Sieverink F, van Gemert-Pijnen LJ, Bouman YH, Kelders SM. Integrating people, context, and technology in the implementation of a web-based intervention in forensic mental health care: mixed-methods study. J Med Internet Res 2020 May 26;22(5):e16906 [FREE Full text] [CrossRef] [Medline]
- Wind L, Milo M. E-mental-health bij de huisarts. In: De dokter en de patiënt met psychische problemen. Houten: Bohn Stafleu van Loghum; Oct 2018.
- Feijt MA, de Kort YA, Bongers IM, IJsselsteijn WA. Perceived drivers and barriers to the adoption of eMental health by psychologists: the construction of the levels of adoption of eMental health model. J Med Internet Res 2018 Apr 24;20(4):e153 [FREE Full text] [CrossRef] [Medline]
- Aref-Adib G, McCloud T, Ross J, O'Hanlon P, Appleton V, Rowe S, et al. Factors affecting implementation of digital health interventions for people with psychosis or bipolar disorder, and their family and friends: a systematic review. Lancet Psychiatry 2019 Mar;6(3):257-266. [CrossRef]
- Broeders, DW, Das HD, Jennissen RP, Tiemeijer WL, De Visser M. Van verschil naar potentieel. Een realistisch perspectief op de sociaaleconomische gezondheidsverschillen. The Netherlands Scientific Council for Government Policy (WRR). 2018. URL: https://www.wrr.nl/publicaties/policy-briefs/2018/08/27/van-verschil-naar-potentieel.-een-realistisch-perspectief-op-de-sociaaleconomische-gezondheidsverschillen [accessed 2019-07-21]
- Latulippe K, Hamel C, Giroux D. Social health inequalities and eHealth: a literature review with qualitative synthesis of theoretical and empirical studies. J Med Internet Res 2017 Apr 27;19(4):e136 [FREE Full text] [CrossRef] [Medline]
- Reiners F, Sturm J, Bouw LJ, Wouters EJ. Sociodemographic factors influencing the use of eHealth in people with chronic diseases. Int J Environ Res Public Health 2019 Feb 21;16(4):645 [FREE Full text] [CrossRef] [Medline]
- van Houwelingen CT, Ettema RG, Antonietti MG, Kort HS. Understanding older people's readiness for receiving telehealth: mixed-method study. J Med Internet Res 2018 Apr 06;20(4):e123 [FREE Full text] [CrossRef] [Medline]
- Peek ST, Wouters EJ, van Hoof J, Luijkx KG, Boeije HR, Vrijhoef HJ. Factors influencing acceptance of technology for aging in place: a systematic review. Int J Med Inform 2014 Apr;83(4):235-248 [FREE Full text] [CrossRef] [Medline]
- van Deursen A. Digitale ongelijkheid in Nederland anno 2018. 2018. URL: https://research.utwente.nl/en/publications/digitale-ongelijkheid-in-nederland-anno-2018 [accessed 2019-07-21]
- de Veer AJ, Peeters JM, Brabers AE, Schellevis FG, Rademakers JJ, Francke AL. Determinants of the intention to use e-Health by community dwelling older people. BMC Health Serv Res 2015 Mar 15;15:103 [FREE Full text] [CrossRef] [Medline]
- Bernard R, Sabariego C, Cieza A. Barriers and facilitation measures related to people with mental disorders when using the web: a systematic review. J Med Internet Res 2016 Jun 09;18(6):e157 [FREE Full text] [CrossRef] [Medline]
- Robotham D, Satkunanathan S, Doughty L, Wykes T. Do we still have a digital divide in mental health? A five-year survey follow-up. J Med Internet Res 2016 Nov 22;18(11):e309 [FREE Full text] [CrossRef] [Medline]
- Kim H, Xie B. Health literacy in the eHealth era: a systematic review of the literature. Patient Educ Couns 2017 Jun;100(6):1073-1082. [CrossRef] [Medline]
- Bolman C. eHealth bij kwetsbare groepen een uitdaging. Open Universiteit. 2019 Jan 25. URL: https://www.ou.nl/documents/40554/724769/Oratieboekje_Catherine_Bolman_DEF_15012019.pdf/48046c78-622a-400f-213d-46995a221b46 [accessed 2019-02-07]
- Swinkels I. E-health verhoogt risico op ongelijkheid. Huisarts en Wetenschap 2017 Sep 12;60(9):477. [CrossRef]
- van der Els-Schoenmaker D, Hopmans E, Jansen S, Korte A. E-health kan toegankelijkheid zorg belemmeren. Medisch Contact. 2020 Jan 15. URL: https://tinyurl.com/yvnnma84 [accessed 2020-10-07]
- Plantinga S, Kaal M. Hoe mediawijs is Nederland? Mediawijzer. Kantar Public. 2018 Sep 17. URL: https://www.mediawijzer.net/wp-content/uploads/sites/6/2018/09/Rapport-Mediawijsheid-volwassenen-2018.pdf [accessed 2019-07-21]
- Vrielink J. Pro Deo: Ongevraagd juridisch advies voor vrijheidsstrijders en andere fanatici. Netherlands: Universitaire Pers Leuven; 2018.
- Beuningen J. EHealth: possibilities, use and views [mogelijkheden, gebruik en opvattingen]. CBS. 2019 Apr 19. URL: https://www.cbs.nl/nl-nl/achtergrond/2019/16/ehealth-mogelijkheden-gebruik-en-opvattingen [accessed 2019-07-16]
- Wentzel J, van der Vaart R, Bohlmeijer ET, van Gemert-Pijnen JE. Mixing online and face-to-face therapy: how to benefit from blended care in mental health care. JMIR Ment Health 2016 Feb 09;3(1):e9 [FREE Full text] [CrossRef] [Medline]
- Osma J, Sprenger M, Mettler T. Introduction of e-mental health in national health systems – A health professionals' perspective. Health Policy Technol 2017 Dec;6(4):436-445. [CrossRef]
- Fishbein M, Ajzen I. Predicting and Changing Behavior: The Reasoned Action Approach. New York: Psychology Press; 2010.
- Venkatesh V, Morris MG, Davis GB, Davis FD. User acceptance of information technology: toward a unified view. MIS Q 2003;27(3):425-478. [CrossRef]
- Venkatesh V, Thong JY, Xu X. Consumer acceptance and use of information technology: extending the unified theory of acceptance and use of technology. MIS Q 2012;36(1):157-178. [CrossRef]
- Fleuren MA, Paulussen TG, Van Dommelen P, Van Buuren S. Towards a measurement instrument for determinants of innovations. Int J Qual Health Care 2014 Oct 20;26(5):501-510 [FREE Full text] [CrossRef] [Medline]
- Rogers E. Diffusion of Innovations, 5th Edition. New York, NY: The Free Press; 2003.
- Hoque R, Sorwar G. Understanding factors influencing the adoption of mHealth by the elderly: an extension of the UTAUT model. Int J Med Inform 2017 May;101:75-84. [CrossRef] [Medline]
- IJzerman RV, van der Vaart R, Evers AW. Internet-based cognitive behavioral therapy among psychologists in a medical setting: a survey on implementation. J Med Internet Res 2019 Aug 09;21(8):e13432 [FREE Full text] [CrossRef] [Medline]
- Chao C. Factors determining the behavioral intention to use mobile learning: an application and extension of the UTAUT model. Front Psychol 2019 Jul 16;10:1652 [FREE Full text] [CrossRef] [Medline]
- Peels D, Mudde A, Bolman C, Golsteijn R, de Vries H, Lechner L. Correlates of the intention to implement a tailored physical activity intervention: perceptions of intermediaries. Int J Environ Res Public Health 2014 Feb 10;11(2):1885-1903 [FREE Full text] [CrossRef] [Medline]
- Vis C, Mol M, Kleiboer A, Bührmann L, Finch T, Smit J, et al. Improving implementation of eMental health for mood disorders in routine practice: systematic review of barriers and facilitating factors. JMIR Ment Health 2018 Mar 16;5(1):e20 [FREE Full text] [CrossRef] [Medline]
- Thompson RL, Higgins CA, Howell JM. Personal computing: toward a conceptual model of utilization. MIS Q 1991 Mar;15(1):125. [CrossRef]
- Kivi M, Eriksson MC, Hange D, Petersson E, Björkelund C, Johansson B. Experiences and attitudes of primary care therapists in the implementation and use of internet-based treatment in swedish primary care settings. Internet Interv 2015 Sep;2(3):248-256. [CrossRef]
- Becker EM, Jensen-Doss A. Computer-assisted therapies: examination of therapist-level barriers to their use. Behav Ther 2013 Dec;44(4):614-624. [CrossRef] [Medline]
- Chismar W, Wiley-Patton S. Predicting internet use: applying the extended technology acceptance model to the healthcare environment. In: E-Health Systems Diffusion and Use: The Innovation, the User and the Use IT Model. Pennsylvania: Idea Group Publishing; 2006.
- Feijt MA, de Kort YA, Westerink JH, Bierbooms JJ, Bongers IM, IJsselsteijn WA. Assessing professionals' adoption readiness for eMental health: development and validation of the eMental health adoption readiness scale. J Med Internet Res 2021 Sep 17;23(9):e28518 [FREE Full text] [CrossRef] [Medline]
- Berndt N, Bolman C, de Vries H, Segaar D, van Boven I, Lechner L. Smoking cessation treatment practices: recommendations for improved adoption on cardiology wards. J Cardiovasc Nurs 2013;28(1):35-47. [CrossRef] [Medline]
- Zijlstra DN, Bolman CA, Muris JW, de Vries H. How to persuade more primary care professionals to adopt a valued smoking cessation referral aid: a cross-sectional study of facilitators and barriers. BMC Prim Care 2022 Sep 07;23(1):227 [FREE Full text] [CrossRef] [Medline]
- De Veirman A. Determinants of the use of eMenthal Health by the mental health counsellors working in general practitioner practices. Open Universiteit. 2020 Dec 06. URL: https://research.ou.nl/ [accessed 2022-11-30]
- Tabachnick B, Fidell L. Using Multivariate Statistics. Boston, United States: Allyn and Bacon; 2001.
- Field A. Discovering Statistics Using SPSS 4th edition. Thousand Oaks, California: Sage Publications Limited; 2020.
- Braun P, Drüge M, Hennemann S, Nitsch FJ, Staeck R, Apolinário-Hagen J. Acceptance of e-mental health services for different application purposes among psychotherapists in clinical training in Germany and Switzerland: secondary analysis of a cross-sectional survey. Front Digit Health 2022 Feb 28;4:840869 [FREE Full text] [CrossRef] [Medline]
- Sander J, Bolinski F, Diekmann S, Gaebel W, Günther K, Hauth I, et al. Online therapy: an added value for inpatient routine care? Perspectives from mental health care professionals. Eur Arch Psychiatry Clin Neurosci 2022 Feb 16;272(1):107-118 [FREE Full text] [CrossRef] [Medline]
- Keil M, Beranek PM, Konsynski BR. Usefulness and ease of use: field study evidence regarding task considerations. Decision Support Syst 1995 Jan;13(1):75-91. [CrossRef]
- Ross J, Stevenson F, Lau R, Murray E. Factors that influence the implementation of e-health: a systematic review of systematic reviews (an update). Implement Sci 2016 Oct 26;11(1):146 [FREE Full text] [CrossRef] [Medline]
- Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, A'Court C, et al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res 2017 Nov 01;19(11):e367 [FREE Full text] [CrossRef] [Medline]
- Nuijen J, Kenter A, Ringoir L. Organisatie en uitvoering van de functie POH-GGZ in de praktijk. Trimbos Instituut. 2021. URL: https://www.trimbos.nl/wp-content/uploads/2021/12/AF1936-Organisatie-en-uitvoering-van-de-functie-POH-GGZ-in-de-praktijk.pdf [accessed 2022-11-16]
- Gaebel W, Lukies R, Kerst A, Stricker J, Zielasek J, Diekmann S, et al. Upscaling e-mental health in Europe: a six-country qualitative analysis and policy recommendations from the eMEN project. Eur Arch Psychiatry Clin Neurosci 2021 Sep 11;271(6):1005-1016. [CrossRef] [Medline]
|EMH: e–mental health|
|GP: general practitioner|
|MHC: mental health counselor|
|MIDI: Measurement Instrument for Determinants of Innovation|
|RAA: reasoned action approach|
|UTAUT: Unified Theory of Acceptance and Use of Technology|
Edited by A Mavragani; submitted 06.11.21; peer-reviewed by M Feijt, W Zhang, C El Morr; comments to author 16.02.22; revised version received 16.06.22; accepted 05.10.22; published 20.12.22Copyright
©Ann E M De Veirman, Viviane Thewissen, Matthijs G Spruijt, Catherine A W Bolman. Originally published in JMIR Formative Research (https://formative.jmir.org), 20.12.2022.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.