Background: Home care service in Norway is struggling to meet the increasing demand for health care under restricted budget constraints, although one-fourth of municipal budgets are dedicated to health services. The integration of Web-based technology in at-home care is expected to enhance communication and patient involvement, increase efficiency and reduce cost. DigiHelse is a Web-based platform designed to reinforce home care service in Norway and is currently undergoing a development process to meet the predefined needs of the country’s municipalities. Some of the main features of the platform are digital messages between residents and the home care service, highlighting information on planned and completed visits, the opportunity to cancel visits, and notifications for completed visits.
Objective: This study aimed to test the usability and economic feasibility of adopting DigiHelse in four districts in Oslo by applying registry and behavioral data collected throughout a one-year pilot study. Early health technology assessment was used to estimate the potential future value of DigiHelse, including the predictive value of behavior data.
Methods: Outcome measures identified by stakeholder insights and scenario drafting in the project’s concept phase were used to assess potential socioeconomic benefits. Aggregated data were collected to assess changes in health consumption at baseline, and then 15 and 52 weeks after DigiHelse was implemented. The present value calculation was updated with data from four intervention groups and one control group. A quasi-experimental difference-in-difference design was applied to estimate the causal effect. Descriptive behavioral data from the digital platform was applied to assess the usability of the platform.
Results: Over the total study period (52 weeks), rates increased for all outcome estimates: the number of visits (rate ratio=1.04; P=.10), unnecessary trips (rate ratio=1.37; P=.26), and phone calls (rate ratio=1.24; P=.08). A significant gap was found between the estimated value of DigiHelse in the concept phase and after the one-year pilot. In the present pilot assessment, costs are expected to exceed potential savings by €67 million (US $75 million) over ten years, as compared to the corresponding concept estimates of a potential gain of €172.6 million (US $193.6 million). Interestingly, behavioral data from the digital platform revealed that only 3.55% (121/3405) of recipients actively used the platform after one year.
Conclusions: Behavioral data provides a valuable source for assessing usability. In this pilot study, the low adoption rate may, at least in part, explain the inability of DigiHelse to perform as expected. This study points to an early assessment of behavioral data as an opportunity to identify inefficiencies and direct digital development. For DigiHelse, insight into why the recipients in Oslo have not made greater use of the Web-based platform seems to be the next step in ensuring the right improvement measures for the home care service.
The era of digital health and the demand for health information technology (HIT) brings enormous opportunities for both patients and professional users . While HIT is the technology used in electronic health (eHealth) services, eHealth itself is defined as the interaction between medical informatics, public health, and businesses, referring to health services and information delivered or enhanced through the internet and related technologies [ ]. One promise of eHealth solutions is that, through enhanced communication and patient involvement, and increased efficiency, reduced costs for the health care service may be achieved. It is also assumed that eHealth may enhance the quality of care by increasing transparency and availability between different health suppliers. There is, however, a discrepancy between the expected value of such interventions and the empirically demonstrated benefits [ , ]. There is a lack of case studies demonstrating the assumed cost-effectiveness and efficacy of eHealth solutions, and research to promote value-based health care in this field has been requested [ , ].
Web-based communication platforms are intended to enhance health in both somatic and mental health care [- ]. Such platforms have shown success in reaching individuals who are hard to contact, in lifestyle behavior change, and the delivery of individualized online care [ , , ]. For chronic illnesses, enabling people to administer their treatment and care may increase compliance to treatment regimens and improve quality of life. The translation of the Diabetes Prevention Program to online treatment is one such example [ ]. The failure of adoption by end-users, however, is a challenge faced by these Web-based interventions. Accordingly, end-user engagement in the development of these interventions has been recognized as essential to increase adoption rates when they are introduced [ , ].
A health service characterized by efficiency and high quality can only be achieved if patient outcomes and costs of delivery are addressed . When facing the complex health care system, not only do technical and legal issues appear, but so do organizational, economic, and social aspects [ ]. User-centric design can be employed from the earliest exploratory stages to help understand and design for the needs, goals, limitations, capabilities, and preferences of all stakeholders [ ]. Recommendations from an international workshop in the United Kingdom on how to create, evaluate, and implement effective eHealth interventions highlights new evaluative challenges in the field. Due to the swiftly changing technological landscape, these UK authors emphasized challenges such as continuous technological adaption and problems identifying valid outcome measures for assessment of costs and patient benefits [ ]. Thus, to adjust to the rapidly changing context, standard methods for development and assessment will benefit from including the whole development cycle. Access to data and valid information from a conceptual stage of development may, however, be demanding, which could explain the lack of empirical evidence concerning the effect of eHealth interventions [ , ].
Health technology assessment (HTA) is traditionally used to provide decision support in the implementation phase of new or current health technology. HTA is defined as an interdisciplinary process for synthesizing information about medical, social, economic, and ethical issues related to the introduction of a new health technology . To improve the pace and efficiency of the development and assessment of health innovation, new methods for early HTA are emerging in the literature [ ]. Early HTA is a form of HTA that evaluates technologies still in development and can be defined as the initial examination of the medical, economic, social, and ethical implications of a health intervention to determine the potential of its incremental value in health care [ , ]. A standard model for early HTA is yet to be established, so research is needed to validate the proposed approaches to early HTA emerging in the literature [ ].
DigiHelse is an intervention designed to reinforce the home care service in Norway and is currently undergoing a procurement process in the county’s municipalities. This is the second of a series of two studies reporting on the effects of implementing the Web-based communication platform, and the first study reported on the early assessment of potential socioeconomic value in the concept stage of the project. DigiHelse was designed and developed to integrate a national Web-based communication platform for recipients of home care services. The main features of the platform were digital messages between residents and the home care service, visualizing agreed upon and completed visits with their associated information, the option to cancel visits, and final notifications of completed visits. In the concept stage of development, data was collected from stakeholders and experts to build scenarios to show the potential value of the intervention. Based on the findings, the project was granted additional funding and proceeded to its pilot phase in four districts in Oslo. Throughout this pilot study, the project needed to collect evidence on its potential benefits to ease the procurement process in other municipalities in the country.
DigiHelse is an example of an eHealth intervention still in development; thus, there is an opportunity to perform assessments on the different stages of the development cycle. A stepwise decision process with several evaluation points and iterative adoptions of the solution has been incorporated in the implementation plan to ensure that the final solution meets the needs of the end-users. This study aimed to test the usability and economic feasibility of adopting DigiHelse in four districts in Oslo by applying registry and behavioral data collected throughout a one-year pilot. Early HTA was used to estimate the potential future value of DigiHelse, including the predictive value of behavior data.
The target population for the intervention is composed of all the recipients of the home care service in Norway, their next of kin, and the service providers of the home care service. The home care service in Norway is a part of the country’s primary health care service. Norway has 426 municipalities that are responsible for the provision of services in primary care. Operations directed under primary care are typically health services provided outside an institution (with a preferred emphasis on health promotion and preventive work), general medical care (general practitioner ), and nursing services outside the hospital. Nurses and doctors in preventive and long-term care services are usually employed in municipal health care . Although the municipalities in Norway dedicate a significant part of their budgets to health services (about one quarter), the home care service struggles to meet an increasing demand for health care under the constraints of a restricted budget [ ]. During 2016, there were 355,635 unique recipients of nursing and care services nationally, which equates to 6.7% of the Norwegian population. Of the unique recipients of nursing and care services, 85% received home-based services, and about 2.7 million visits ware performed every week [ ].
This study was set in Oslo in 2018. The purpose of DigiHelse was to digitalize the dialogue between recipients and professionals in home care services in Norway through the development and implementation of a national Web-based platform. All recipients of home care services in four districts in Oslo were offered DigiHelse, in addition to regular services, in a one-year pilot project from autumn 2017 to the next year. The utilization of DigiHelse was completely voluntary. The project is based on the existing “Helsenorge.no” platform from the Norwegian Directorate of e-Health, which provides national digital health services. The realization of digital services in this project supports the overall objective of the development of information and communication technology in the health care sector to provide citizens with access to simple and secure digital services .
The intervention intends to cover the following objectives and needs:
- Support relatives who are involved in care tasks and strengthen the interaction between service providers and relatives through the possibility of secure digital dialogue and an overview of visits.
- Support service recipients in enhanced coping, safety, and involvement in their daily lives by providing an overview of visits and facilitating dialogue with the home care service, so that they can express their experiences and needs.
- Ensure that the home service can organize tasks more rationally and cooperate better with service recipients and relatives.
- Ensure that messages from relatives and recipients are captured and followed up with appropriately, such that phone inquiries are reduced, tasks can be registered at a more favorable time, and unnecessary trips to the recipient can be reduced.
Choice of Health Outcomes
In the concept stage of DigiHelse, a multidisciplinary team of stakeholders managed to identify both quantitative and qualitative outcome measures comparing the new solution to the current situation. Through scenario building, a present value calculation on socioeconomic impact was carried out. The outcome measures, based on each scenario elaborated on in the previous study of DigiHelse, are presented below.
Increased Predictability for Recipients
Notifications of appointments and any delays might give recipients a greater sense of predictability and greater confidence in the home care services. Digital services may also provide better information security for recipients than email and texting, thus more thoroughly safeguarding the privacy of the recipients. In the concept assessment, increased predictability gave a predicted annual value of €408.4 million (US $458.3 million) per year. This was based on the assumption that if the recipients knew the exact arrival time of their home service care, an hour waiting time per visit might be saved. This effect was not included in the present value calculation as the value of free time is debatable.
Increased Involvement From Relatives and Volunteers
Improved communication between relatives and the home service was assumed to amount to savings of €13.8 million (US $15.5 million) a year. For relatively self-sufficient recipients, relatives and volunteers may carry out one visit per month on average.
Increased Predictability of the Home Care Service
The assumption in the concept stage was that the staff in the home care service might be able to better manage their workday by using digital channels rather than the telephone. They may experience reduced time consumption for administrative tasks and have more time for preventive work. Increased predictability of the home service may also result in fewer unnecessary trips to the users, as unwanted visits may be easily canceled in the portal. With the ability of the user to digitally cancel and postpone visits, a reduction of 30% of unnecessary trips was estimated, which results in assumed savings of €3.8 million (US $4.3 million) a year.
Greater Dialogue Efficiency and Time Management
Through interviews with professionals from the home services, the stakeholders estimated potential administrative time savings in administrative time of 30 minutes per day, with an hourly rate of €46 (US $51.60) with digital communication, resulting in savings of €7.1 million (US $8 million) a year.
Reduced Phone Inquiries
The estimated impact of reduced phone inquiries may amount to €1 million (US $1.2 million) per year on a national basis. To assess whether the intervention may reduce phone inquiries that otherwise could be solved digitally, the project group conducted a phone survey in Oslo and Bergen. After the survey, a scenario where digital communication could reduce phone inquiries to the home service by 40% was built.
Provide a Technical Basis for Developing Digital Services
Providing a technological basis for developing digital services may result in a one-time saving of €18.25 million (US $20.5 million). If 50% of the municipalities in Norway each procure a platform, they will, on average, consume €100,000 (US $110,000) each, including procurement, infrastructure, licenses/rent, etc. This effect was not included in the present value calculation because the digitalization of home services is still not statutory in the country’s municipalities.
In the present study, three outcomes (increased involvement from relatives and volunteers, increased predictability of the home care service, and reduced phone inquiries) were reassessed using empirical data from the one-year pilot in four districts in Oslo, and a control district. The remaining outcome measures will appear unchanged in the present value calculation, as will the unit costs of investment, training, and maintenance.
In this pilot assessment, descriptive behavioral data from the Web-based platform was collected to study the usability of the platform. Data points, such as the number of digital users, digital inquiries, and active users, were retrieved from the platform’s server. In this study, we used behavioral data on the number of active users to study usability. All recipients in the intervention districts were offered the chance to log into the platform and create a profile. The number of active users is defined as the number of users who not only created a profile but also had interactions with the home care service in the platform. Aggregated data from the electronic patient record (EPR) system Gerica was retrieved to study changes in health consumption in the home care service in the four intervention districts and one control district in Oslo. Data collection was performed through three measurement points in time: at baseline (the week before the intervention), during the short period (15 weeks after the intervention), and over the total study period (52 weeks after the intervention). Data was collected on the number of visits of the home service to the recipient to assess if the intervention may give an incentive to increase involvement from relatives and volunteers in the care of recipients.
Further, to assess if the option to cancel unwanted trips in the portal may result in fewer unnecessary trips and increased predictability of the service, data was also collected on the number of unnecessary trips by the home care service to the recipient. An unnecessary trip is when the home service arrives at a recipient’s home for a planned visit, and the recipient does not answer the door. Finally, to study if digital dialogue may reduce the number of phone calls to the home service, phone calls to the service were registered during the three measurement points. Input variables on the cost of the present value calculation are shown in.
A 10-year present value calculation model with a discount rate of 4% was used to estimate the potential value of the intervention. The potential value was first estimated every year, and by employing the cost of investment, training, and implementation pace, the overall value was calculated over ten years. The assumption of the 10-year life cycle is based on national recommendations from the Directorate for Financial Management . The data from the intervention and control group was analyzed using the quasi-experimental difference-in-difference design to estimate the causal effect and to update the present value calculation. Such a design is typically used to estimate the effect of an intervention by comparing the changes in outcomes over time between a population exposed to the intervention (intervention group) and a population not exposed (control group) [ ]. A Poisson regression analysis was used to fit the model, as the dependent variables are counts of events.
First, to test for an effect of the intervention, interaction models with dummy variables were used for the intervention and the period. To assess both the short-term and long-term effects, analyses were done separately for time points one week before the intervention versus 15 weeks after, and before intervention versus 52 weeks after. The number of those exposed to the intervention in the model corresponded to the number of home care recipients (user base) in each group because all recipients in the intervention group had, in principle, access to DigiHelse, and all analyses are based on aggregate data. The interaction coefficient between the intervention and the time period dummies indicates the effect of the intervention. Second, to assess the effect of the proportion of active users in the intervention districts, an interaction model with continuous-time and continuous rates of digital users was used in each district. Different rates of active digital users were then extrapolated to assess how this would influence the rates for visits, unnecessary trips, and phone calls, and thereby, the costs in the present value analysis. All calculations were done in kr and converted to euros based on the exchange rate from May 2018 (9.54) . All analyses were performed in Stata 15.1 (StataCorp, College Station, Texas, United States) and Excel 2010 (Microsoft, Redmond, Washington, United States).
and show the demographic distribution and aggregate data from the EPR system Gerica for each of the intervention districts and the control district. District 2 has the highest percentage of active digital users. This district has a relatively high percentage of immigrants, but the lowest percentage of people under retirement age. The user base is the number of recipients of home care services in each district, and the digital users are the recipients who have logged in to the digital platform. The active digital users are the recipients who use the portal to actively administer their services and contact with the home care service. Finally, the demographic data shows the composition of people over retirement age and immigrants of the total population in each district.
shows the rates for the number of visits, the number of unnecessary trips, and the number of phone calls extracted from the EPR system for every ten users. The rates in the intervention and control groups at baseline, after 15 weeks (short period), and after 52 weeks (total study period), with their associated percentage changes compared to baseline, are presented. Also presented are P values for whether the difference over time is significantly different between intervention and control, which corresponds to whether the intervention has an effect.
The intervention group had a 12% (8.32/69.33) higher rate for the number of visits at baseline (77.65) compared to the control group (69.33). After 15 weeks (short period), the rate for the number of visits in the control group increased by 7% (4.97/74.30). In the same period, the rate for the number of visits also increased in the intervention group by 6% (4.26/81.91; rate ratio=1.06; P=.59). In the total study period (after 52 weeks), the rate for the number of visits increased in the control group by 7% (5.05/74.38), but by 11% (8.77/86.42) in the intervention group (rate ratio=1.04; P=.10). Both unnecessary trips and phone calls had a lower rate at baseline in the intervention group (19%) compared to the control group (28%) at baseline. However, over time the rates were further reduced in the control group compared to the intervention group for both unnecessary trips and phone calls.
Over the 52 total weeks of the study period, unnecessary trips decreased in the control group by 33% (–0.21/0.42), and the rate for unnecessary trips reduced in the intervention group by 10% (–0.05/0.46). This is still less than in the control group, with a rate ratio of 1.37 (P=.26). Phone calls were reduced in the control group by 2% (–0.05/2.66) and increased in the intervention group by 22% (0.42/2.36), by a rate ratio of 1.24 (P=.08). In conclusion, all point estimates indicate that the intervention increases the rates for all outcomes, although none of the intervention effects were significant.
|Users||District 1||District 2||District 3||District 4||Control|
|User base, n|
|Digital users, n|
|Active digital users, n|
|District||Population, N||Over retirement age, n (%)||Immigrants, n (%)|
|1||57,000||6954 (12.2)||15,960 (28)|
|2||36,000||1836 (5.1)||12,600 (35)|
|3||49,200||5806 (11.8)||17,220 (35)|
|4||49,800||6823 (13.7)||8964 (18)|
|Control||51,400||2878 (5.6)||20,046 (39)|
|Outcome||Baseline (week 0)||Short period after intervention (after 15 weeks)||Short period change, rate (%)||P value||Total study period (after 52 weeks)||Long-period change, rate (%)||P value|
|Rate of visits|
|Intervention||77.65||81.91||4.26 (6)||—||86.42||8.77 (11)||—|
|Control||69.33||74.30||4.97 (7)||.59||74.38||5.05 (7)||.10|
|Rate of unnecessary trips|
|Intervention||0.51||0.41||–0.1 (–20)||—||0.46||–0.05 (–10)||—|
|Control||0.63||0.48||–0.15 (–24)||.83||0.42||–0.21 (–33)||.26|
|Rate of phone calls|
|Intervention||1.94||1.81||–0.13 (–7)||—||2.36||0.42 (22)||—|
|Control||2.71||2.64||–0.07 (–3)||.75||2.66||–0.05 (–2)||.08|
Incremental Costs and Outcomes
In the prior concept stage assessment of the project, a 90% adoption rate of the digital portal DigiHelse was assumed. Applying behavioral data made available from the platform’s server revealed that the adoption rate after the one-year pilot was not as expected. Only 3.55% (121/3405) of active users were registered in the data, which makes it a challenge to both predict whether the precision and the fit of the concept model were good and compare the present value calculation with and without empirical pilot data. As such, the present analysis may only show that the control district improved over time compared to the intervention districts and that the adoption rate of the intervention was considerably lower than expected. From the difference-in-difference analysis, a 37% (0.46/0.34) increase in the rate of unnecessary trips in the intervention group was found, but this was given the observed adoption rates of around 3.55% (121/3405). Using continuous-time and adoption rates in the model and extrapolation to 50% active digital users, the effect of the intervention would have been a 128-fold yearly increased rate of unnecessary trips. The same trend was found for the number of visits. When extrapolating for 50% of active digital users, the effect of the intervention would be a 1.04 times higher increase in the intervention group compared to the control group. Finally, if there were 50% active users, the effect of the intervention would be a 55-fold increase in the phone call rate.
When including the outputs from the difference-in-difference model comparing the intervention and control group into the present value calculation model from the concept stage, the estimated value of the intervention changes radically (see). The net present value of the intervention after adding data form the pilot is reduced by €241.8 million (US $271.3 million) over ten years from the first assessment, resulting in a loss of €62.2 million (US $69.8 million) over ten years. Based on the present pilot assessment, costs are expected to exceed potential savings by €67 million (US $ 75.2 million) over ten years, compared to potential gains of €172.6 million (US $ 193.7 million) from the prior concept assessment.
Through a case of early HTA employing empirical data from a pilot study, the present study updated effect estimates made in the concept stage of the development of DigiHelse. Based on the present pilot assessment, costs are expected to exceed potential savings by €67 million (US $75.2 million) over ten years, compared to potential gains of €172.6 million (US $193.7 million) from the first assessment. After one year, only 3.55% (121/3405) of recipients used the platform actively. The prior socioeconomic analysis, conducted in the concept stage of DigiHelse, was based on stakeholder insight and scenario drafting. Collecting empirical data from the one-year pilot of DigiHelse, the present study evaluated the potential value of the intervention and assessed the precision of early HTA using stakeholder analysis and scenario drafting. Three of the outcome measures identified in the first study constituted the basis for the difference-in-difference analysis, and related costs were analyzed using a 10-year present value calculation with a rate of 4%. We found a significant gap between the estimated value in the concept stage of DigiHelse and the estimated value using empirical data from the one-year pilot.
This may indicate that early assessment using stakeholder insight and scenario drafting applied in the concept stage was less precise than expected. Another explanation may be, at least in part, suboptimal pilot implementation, as it is known that adoption and diffusion of eHealth solutions may be time-consuming and require significant adaptation of work practices . However, by assessing behavioral data on the actual use of the platform, an important issue likely to affect the outcome of the assessment was found: a very low rate of DigiHelse users among recipients of home care services. This may explain why there was no significant change in the outcome measures between the control and the intervention districts after the pilot.
A review study highlighting methodological challenges in early HTA emphasizes both the lack of proof on the efficacy of the methods and the absence of a standardized framework for early assessments . Empirical and theoretical attempts have been made to fill the evidence gaps in early assessment modeling, with theoretical recommendations on the use of sophisticated mathematical techniques such as Bayesian modeling or Markov modeling [ - ]. Empirical models based on scenario drafting and expert elicitation have also been used to compensate for the lack of data and steer the innovation in the right direction [ , - ].
Findings from the review showed how stakeholder insights and scenario drafting might be used in an early phase to collect data on patient outcomes and effects on costs [, , ]. However, there are some presented concerns are, such as the high uncertainty regarding the availability of adequate data sources for modeling outcomes and that the models suffer from the precision required for data input [ , ]. Although a strength of the present study was the availability of concept stage assumptions and assessment based on stakeholder insights and scenario drafting when empirical data from the present pilot were analyzed, lack of precision was found. In line with similar research on the subject, we found that early-stage analyses may suffer from loss of information, as they are unable to reflect all possible outcomes [ , ]. Further, it cannot be excluded that stakeholders may be positively biased towards the value of the technology in which they have a particular interest [ ]. This may explain the identified gap between the estimated socioeconomic value and the value assessment based on empirical pilot data in the present study.
While the low acceptability rate among recipients of the home care service in Oslo was an important concern found in the present study, other studies addressing the acceptance of eHealth solutions tested in clinical settings have indicated high patient acceptability rates [- ]. However, it is unclear how the adoption rate of eHealth solutions may be affected once the technology is moved outside the boundaries of the clinic and is implemented in users’ homes. Discrepancies in access to the internet and technological literacy in different subgroups may influence the adoption rate and, thus, the estimated improvement in efficiency and cost reduction expected from the implementation of eHealth [ - ]. Identified subgroups that are especially challenged by eHealth solutions are the elderly [ , ], minorities [ ], and the socioeconomically disadvantaged [ , ].
Effective adoption among users is a prerequisite for successful implementation, and the effectiveness of eHealth is compromised if the solutions are suboptimally implemented. Discomfort with the new technology and a preference for well-known, earlier provided services are reasons reported to influence the adoption of eHealth technologies [, ]. Qualitative methods are needed to explore the experienced discomfort about or preference for existing analog health services. Such methods are increasingly being explored to accompany quantitative assessments of complex innovations to provide a deeper understanding of the adoption of eHealth [ ]. While quantitative methods explore relationships between digitalization and disease outcomes, qualitative methods may provide a deeper understanding of contextual factors influencing these relationships, such as information on drivers and barriers to technological implementation [ ].
The engagement of end-users in collaboration with product developers may succeed in increasing acceptability in particularly vulnerable groups by incorporating favorable eHealth designs to overcome barriers, although this may not be sufficient . Predictive behavioral data represents another important tool, as it provides valuable information on the usability of a digitalized service and its corresponding population, and thus may determine whether the predefined intent of the new service is met. The digitalization of services in health care provides a new, potentially valuable data source as real-time data can be extracted and analyzed at any time [ ]. According to the Lean Startup framework, behavioral data from initial testing provides essential information on how the market will respond to a service or a product [ ]. Measuring quantifiable behavioral data outcomes provides an opportunity to assess usability [ ]. Qualitative information on the directions of the developmental improvements of services may then be assembled from the same study sample. This allows for iterative modifications and adaptations at the initial project phase to avoid the implementation of ineffective services. Through this process, the likelihood of developing a user-centric service which complies with market expectations may increase as the early assessment of behavioral data provides the ability to test whether the service meets its initial intent and contributes to value-based health care [ ].
There are several limitations of this study. Firstly, health economic analysis commonly presents results as cost per patient. The present analysis applied the net present value of the presented investment, weighting potential benefits against investment costs. In this case, the model was chosen due to the significant heterogeneity among recipients receiving home services and the early nature of the analysis.
Further, a quasi-experimental design was used. This means that many confounders may affect the results, such as changes over time independent of the intervention, aging in the population, and heterogeneity between the intervention and control groups. The homogeneity of the districts in the analysis may also be questioned due to the baseline data. To increase the representativeness of the selected control group, data could preferably have been collected from more than one control district. An increased number of measurement points before the intervention would have provided an opportunity to assess trend assumptions between the control and intervention group, which is crucial for difference-in-difference analyses.
Further, if the behavioral data had shown a higher adoption rate, both these issues would have been resolved before the difference-in-difference analysis. In addition to empirical results, the present value model could have been used to predict the socioeconomic outcomes if the adoption rate was 90%. However, given the unexpectedly low adoption rate, collecting more measurement points, and performing a sensitivity analysis of the findings was deemed futile. It should also be taken into consideration that the value of DigiHelse was calculated on a national basis, although, due to the Norwegian municipal health budget autonomy, it is uncertain whether all municipalities will implement the service. A final limitation of this study is that the analyses are based on aggregated numbers and not individual data. On the positive side, the database is larger than typical pilot studies; however, it comes with an inability to connect data sources to adjust for confounders on the individual level.
Measuring objective behavioral data provides an important source to assess usability. This study reported on the attempt to evaluate methods for early HTA by reassessing DigiHelse by comparing pilot intervention data to a corresponding control group. In this pilot study, the low adoption rate may, at least in part, explain the inability of the DigiHelse pilot to perform as expected. This study points to an early assessment of behavioral data as an opportunity to identify inefficiencies and direct digital development. Implementing eHealth solutions is known to be challenging and time-consuming. To ensure adoption, effective diffusion strategies are needed, including user training programs. For DigiHelse, learning strategies may be targeted to increase adoption in the next phase.
The integration of behavioral data in early planning and assessment provides an opportunity to address implementation challenges and user adherence, where early HTA modeling has a purpose. For DigiHelse, insight into why the recipients in Oslo have not made greater use of the Web-based platform seems to be the next step in ensuring the right improvement measures for the home care service. We encourage more research on early HTA and the use of behavioral data in case studies as tools to empirically demonstrate eHealth intervention benefits.
The authors would like to thank all the staff in the Oslo municipality who made it possible for us to collect data on costs and effects to estimate the potential socioeconomic value of DigiHelse. This submission will be funded through the Norwegian Research Council grant #237766/O30.
LNS and TM conceived, designed, and performed the experiments. LNS and TM analyzed the data. LNS authored the manuscript. KJK and KK critically reviewed the manuscript.
Conflicts of Interest
Summary of cost units.DOCX File , 18 KB
Summary of priced effects in the concept phase and after the pilot phase.DOCX File , 18 KB
CONSORT‐EHEALTH checklist (V 1.6.1).PDF File (Adobe PDF File), 3533 KB
- Fourneyron E, Wittwer J, Rachid Salmi L, Groupe de recherche Eva TSN. [Health information technology: current use and challenges for primary healthcare services]. Med Sci (Paris) 2018;34(6-7):581-586 [FREE Full text] [CrossRef] [Medline]
- Eysenbach G. What is e-health? J Med Internet Res 2001;3(2):E20 [FREE Full text] [CrossRef] [Medline]
- Black AD, Car J, Pagliari C, Anandan C, Cresswell K, Bokun T, et al. The impact of eHealth on the quality and safety of health care: a systematic overview. PLoS Med 2011 Jan 18;8(1):e1000387 [FREE Full text] [CrossRef] [Medline]
- Petersen C, Adams SA, DeMuro PR. mHealth: Don't Forget All the Stakeholders in the Business Case. Med 2 0 2015 Dec 31;4(2):e4 [FREE Full text] [CrossRef] [Medline]
- de Keizer NF, Ammenwerth E. The quality of evidence in health informatics: how did the quality of healthcare IT evaluation publications develop from 1982 to 2005? Int J Med Inform 2008 Jan;77(1):41-49. [CrossRef] [Medline]
- de Beurs D, van Bruinessen I, Noordman J, Friele R, van Dulmen S. Active Involvement of End Users When Developing Web-Based Mental Health Interventions. Front Psychiatry 2017;8:72 [FREE Full text] [CrossRef] [Medline]
- Van Dijk MR, Huijgen NA, Willemsen SP, Laven JS, Steegers EA, Steegers-Theunissen RP. Impact of an mHealth Platform for Pregnancy on Nutrition and Lifestyle of the Reproductive Population: A Survey. JMIR Mhealth Uhealth 2016 May 27;4(2):e53 [FREE Full text] [CrossRef] [Medline]
- Noergaard B, Sandvei M, Rottmann N, Johannessen H, Wiil U, Schmidt T, et al. Development of a Web-Based Health Care Intervention for Patients With Heart Disease: Lessons Learned From a Participatory Design Study. JMIR Res Protoc 2017 May 17;6(5):e75 [FREE Full text] [CrossRef] [Medline]
- Probst T, Pryss RC, Langguth B, Spiliopoulou M, Landgrebe M, Vesala M, et al. Outpatient Tinnitus Clinic, Self-Help Web Platform, or Mobile Application to Recruit Tinnitus Study Samples? Front Aging Neurosci 2017;9:113 [FREE Full text] [CrossRef] [Medline]
- Dahlberg LE, Grahn D, Dahlberg JE, Thorstensson CA. A Web-Based Platform for Patients With Osteoarthritis of the Hip and Knee: A Pilot Study. JMIR Res Protoc 2016 Jun 03;5(2):e115 [FREE Full text] [CrossRef] [Medline]
- Sepah SC, Jiang L, Peters AL. Long-term outcomes of a Web-based diabetes prevention program: 2-year results of a single-arm longitudinal study. J Med Internet Res 2015 Apr 10;17(4):e92 [FREE Full text] [CrossRef] [Medline]
- Torous J, Kiang MV, Lorme J, Onnela J. New Tools for New Research in Psychiatry: A Scalable and Customizable Platform to Empower Data Driven Smartphone Research. JMIR Ment Health 2016 May 05;3(2):e16 [FREE Full text] [CrossRef] [Medline]
- Liu C, Lim RL, McCabe KL, Taylor S, Calvo RA. A Web-Based Telehealth Training Platform Incorporating Automated Nonverbal Behavior Feedback for Teaching Communication Skills to Medical Students: A Randomized Crossover Study. J Med Internet Res 2016 Sep 12;18(9):e246 [FREE Full text] [CrossRef] [Medline]
- Brown MM, Brown GC, Sharma S, Landy J. Health care economic analyses and value-based medicine. Surv Ophthalmol 2003;48(2):204-223. [CrossRef] [Medline]
- Ghazali M, Ariffin N, Omar R. User centered design practices in healthcare: A systematic review. 2014 Presented at: 3rd International Conference on User Science and Engineering (i-USEr); September 2-5; Shah Alam, Malaysia.
- Michie S, Yardley L, West R, Patrick K, Greaves F. Developing and Evaluating Digital Interventions to Promote Behavior Change in Health and Health Care: Recommendations Resulting From an International Workshop. J Med Internet Res 2017 Jun 29;19(6):e232 [FREE Full text] [CrossRef] [Medline]
- Esposito D, Taylor EF, Gold M. Using qualitative and quantitative methods to evaluate small-scale disease management pilot programs. Popul Health Manag 2009 Feb;12(1):3-15. [CrossRef] [Medline]
- Kristensen FB, Lampe K, Chase DL, Lee-Robin SH, Wild C, Moharra M, European network for Health Technology Assessment (EUnetHTA). Practical tools and methods for health technology assessment in Europe: structures, methodologies, and tools developed by the European Network for Health Technology Assessment, EUnetHTA. Int J Technol Assess Health Care 2009 Dec;25 Suppl 2:1-8. [CrossRef] [Medline]
- Poulin P, Austen L, Scott CM, Poulin M, Gall N, Seidel J, et al. Introduction of new technologies and decision making processes: a framework to adapt a Local Health Technology Decision Support Program for other local settings. Med Devices (Auckl) 2013;6:185-193 [FREE Full text] [CrossRef] [Medline]
- Støme LN, Moger T, Kidholm K, Kværner KJ. Early assessment of innovation in a healthcare setting. Int J Technol Assess Health Care 2019 Jan;35(1):17-26. [CrossRef] [Medline]
- Huygens SA, Rutten-van Mölken MPMH, Bekkers JA, Bogers AJJC, Bouten CVC, Chamuleau SAJ, et al. Conceptual model for early health technology assessment of current and novel heart valve interventions. Open Heart 2016;3(2):e000500 [FREE Full text] [CrossRef] [Medline]
- Markiewicz K, van Til JA, IJzerman MJ. Medical devices early assessment methods: systematic literature review. Int J Technol Assess Health Care 2014 Apr;30(2):137-146. [CrossRef] [Medline]
- Ringard �, Sagan A, Sperre Saunes I, Lindahl AK. Norway: health system review. Health Syst Transit 2013;15(8):1-162 [FREE Full text] [Medline]
- Rasmussen B. Between Endless Needs and Limited Resources: The Gendered Construction of a Greedy Organization. Gender Work & Org 2004 Sep;11(5):506-525. [CrossRef]
- Statistics Norway. 2019 Jun 24. Care Services URL: https://www.ssb.no/en/pleie [accessed 2020-01-02]
- Norwegian Ministry of Health and Care Services. Regjeringen. 2016 May 18. (2012-2013) [One resident-One journal, Digital services in the health and care sector] URL: https://www.regjeringen.no/no/tema/helse-og-omsorg/e-helse/innsikt/en-innbygger---en-journal/id2480065/ [accessed 2020-01-02]
- NAV. 2008 Jun 04. Labour Market Information in English URL: https://www.nav.no/en/home/work-and-stay-in-norway/relatert-informasjon/labour-market-information-in-english [accessed 2019-02-15]
- Hyldgård VB, Laursen K, Poulsen J, Søgaard R. Robot-assisted surgery in a broader healthcare perspective: a difference-in-difference-based cost analysis of a national prostatectomy cohort. BMJ Open 2017 Jul 21;7(7):e015580 [FREE Full text] [CrossRef] [Medline]
- Norges Bank. 2019. Exchange rates URL: https://www.norges-bank.no/en/topics/Statistics/exchange_rates/ [accessed 2020-01-02]
- Tsiknakis M, Kouroubali A. Organizational factors affecting successful adoption of innovative eHealth services: a case study employing the FITT framework. Int J Med Inform 2009 Jan;78(1):39-52. [CrossRef] [Medline]
- Craig J, Carr L, Hutton J, Glanville J, Iglesias C, Sims AJ. A review of the economic tools for assessing new medical devices. Appl Health Econ Health Policy 2015 Feb;13(1):15-27. [CrossRef] [Medline]
- Hartz S, John J. Public health policy decisions on medical innovations: what role can early economic evaluation play? Health Policy 2009 Feb;89(2):184-192. [CrossRef] [Medline]
- Retèl VP, Grutters J, van Harten WH, Joore MA. Value of research and value of development in early assessments of new medical technologies. Value Health 2013;16(5):720-728 [FREE Full text] [CrossRef] [Medline]
- Bartelmes M, Neumann U, Lühmann D, Schönermark MP, Hagen A. Methods for assessment of innovative medical technologies during early stages of development. GMS Health Technol Assess 2009 Nov 05;5:Doc15 [FREE Full text] [CrossRef] [Medline]
- Retèl VP, Joosten SE, van Harten WH. Expert Elicitation Used for Early Technology Assessment to Inform on Cost-Effectiveness of Next Generation Sequencing. Value Health 2014 Nov;17(7):A652 [FREE Full text] [CrossRef] [Medline]
- Sayres L, Allyse M, Cho MK. Integrating stakeholder perspectives into the translation of cell-free fetal DNA testing for aneuploidy. Genome Med 2012;4(6):49 [FREE Full text] [CrossRef] [Medline]
- Joosten S, Retèl VP, Coupé VMH, van den Heuvel MM, van Harten WH. Scenario drafting for early technology assessment of next generation sequencing in clinical oncology. BMC Cancer 2016 Feb 06;16:66 [FREE Full text] [CrossRef] [Medline]
- Girling A, Young T, Brown C, Lilford R. Early-stage valuation of medical devices: the role of developmental uncertainty. Value Health 2010 Aug;13(5):585-591 [FREE Full text] [CrossRef] [Medline]
- Retèl VP, Joore M, Linn S, Rutgers E, van Harten WH. Scenario drafting to anticipate future developments in technology assessment. BMC Res Notes 2012 Aug 16;5:442 [FREE Full text] [CrossRef] [Medline]
- Harris-Roxas BF, Harris PJ. Learning by doing: the value of case studies of health impact assessment. N S W Public Health Bull 2007;18(9-10):161-163 [FREE Full text] [CrossRef] [Medline]
- Bartelmes M, Neumann U, Lühmann D, Schönermark MP, Hagen A. Methods for assessment of innovative medical technologies during early stages of development. GMS Health Technol Assess 2009 Nov 05;5:Doc15 [FREE Full text] [CrossRef] [Medline]
- Kummer T, Schäfer K, Todorova N. Acceptance of hospital nurses toward sensor-based medication systems: a questionnaire survey. Int J Nurs Stud 2013 Apr;50(4):508-517. [CrossRef] [Medline]
- Lambooij MS, Hummel MJ. Differentiating innovation priorities among stakeholder in hospital care. BMC Med Inform Decis Mak 2013 Aug 16;13:91 [FREE Full text] [CrossRef] [Medline]
- Bryant J, Bonevski B, Paul C, Lecathelinais C. Assessing smoking status in disadvantaged populations: is computer administered self report an accurate and acceptable measure? BMC Med Res Methodol 2011 Nov 21;11(1):153 [FREE Full text] [CrossRef] [Medline]
- Paul CL, Carey M, Yoong SL, D'Este C, Makeham M, Henskens F. Access to chronic disease care in general practice: the acceptability of implementing systematic waiting-room screening using computer-based patient-reported risk status. Br J Gen Pract 2013 Sep;63(614):e620-e626 [FREE Full text] [CrossRef] [Medline]
- Noble N, Paul C, Carey M, Sanson-Fisher R, Blunden S, Stewart J, et al. A cross-sectional survey assessing the acceptability and feasibility of self-report electronic data collection about health risks from patients attending an Aboriginal Community Controlled Health Service. BMC Med Inform Decis Mak 2014 Apr 16;14:34 [FREE Full text] [CrossRef] [Medline]
- Olphert W, Damodaran L. Older people and digital disengagement: a fourth digital divide? Gerontology 2013;59(6):564-570. [CrossRef] [Medline]
- Kruse RL, Koopman RJ, Wakefield BJ, Wakefield DS, Keplinger LE, Canfield SM, et al. Internet use by primary care patients: where is the digital divide? Fam Med 2012 May;44(5):342-347 [FREE Full text] [Medline]
- Choi NG, Dinitto DM. The digital divide among low-income homebound older adults: Internet use patterns, eHealth literacy, and attitudes toward computer/Internet use. J Med Internet Res 2013 May 02;15(5):e93 [FREE Full text] [CrossRef] [Medline]
- Subramanian U, Hopp F, Lowery J, Woodbridge P, Smith D. Research in home-care telemedicine: challenges in patient recruitment. Telemed J E Health 2004;10(2):155-161. [CrossRef] [Medline]
- Mair F, Goldstein P, Shiels C, Roberts C, Angus R, O'Connor J, et al. Recruitment difficulties in a home telecare trial. J Telemed Telecare 2006;12 Suppl 1:26-28. [CrossRef] [Medline]
- Sanders C, Rogers A, Bowen R, Bower P, Hirani S, Cartwright M, et al. Exploring barriers to participation and adoption of telehealth and telecare within the Whole System Demonstrator trial: a qualitative study. BMC Health Serv Res 2012 Jul 26;12:220 [FREE Full text] [CrossRef] [Medline]
- Fernandez M, Ten Hoor GA, van Lieshout S, Rodriguez S, Beidas R, Parcel G, et al. Implementation Mapping: Using Intervention Mapping to Develop Implementation Strategies. Front Public Health 2019;7:158 [FREE Full text] [CrossRef] [Medline]
- Carey M, Noble N, Mansfield E, Waller A, Henskens F, Sanson-Fisher R. The Role of eHealth in Optimizing Preventive Care in the Primary Care Setting. J Med Internet Res 2015 May 22;17(5):e126 [FREE Full text] [CrossRef] [Medline]
- Montini L, Prost S, Schrammel J, Rieser-Schüssler N, Axhausen KW. Comparison of Travel Diaries Generated from Smartphone Data and Dedicated GPS Devices. Transportation Research Procedia 2015;11:227-241. [CrossRef]
- Ries E. The Lean Startup: How Today's Entrepreneurs Use Continuous Innovation To Create Radically Successful Businesses. New York City, New York: Crown Publishing Group; 2011.
- Kauer SD, Reid SC, Sanci L, Patton GC. Investigating the utility of mobile phones for collecting data about adolescent alcohol use and related mood, stress and coping behaviours: lessons and recommendations. Drug Alcohol Rev 2009 Jan;28(1):25-30. [CrossRef] [Medline]
|eHealth: electronic health|
|EPR: electronic patient record|
|HIT: health information technology|
|HTA: health technology assessment|
Edited by G Eysenbach; submitted 22.05.19; peer-reviewed by A Triantafyllidis, H Van Lente; comments to author 07.11.19; revised version received 19.11.19; accepted 16.12.19; published 20.01.20Copyright
©Linn Nathalie Støme, Tron Moger, Kristian Kidholm, Kari J Kværner. Originally published in JMIR Formative Research (http://formative.jmir.org), 20.01.2020.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on http://formative.jmir.org, as well as this copyright and license information must be included.