Original Paper
Abstract
Background: Disease-related malnutrition is a challenge among hospitalized patients. Despite guidelines and recommendations for prevention and treatment, the condition continues to be prevalent. The MyFood system is a recently developed decision support system to prevent and treat disease-related malnutrition.
Objective: To investigate the possible implementation of the MyFood system in clinical practice, the aims of the study were (1) to identify current practice, routines, barriers, and facilitators of nutritional care; (2) to identify potential barriers and facilitators for the use of MyFood; and (3) to identify the key aspects of an implementation plan.
Methods: A qualitative study was performed among nurses, physicians, registered dietitians, and middle managers in 2 departments in a university hospital in Norway. Focus group discussions and semistructured interviews were used to collect data. The Consolidated Framework for Implementation Research (CFIR) was used to create the interview guide and analyze the results. The transcripts were analyzed using a thematic analysis.
Results: A total of 27 health care professionals participated in the interviews and focus groups, including nurses (n=20), physicians (n=2), registered dietitians (n=2), and middle managers (n=3). The data were analyzed within 22 of the 39 CFIR constructs. Using the 5 CFIR domains as themes, we obtained the following results: (1) Intervention characteristics: MyFood was perceived to have a relative advantage of being more trustworthy, systematic, and motivational and providing increased awareness of nutritional treatment compared with the current practice. Its lack of communication with the existing digital systems was perceived as a potential barrier; (2) Outer settings: patients from different cultural backgrounds with language barriers and of older age were potential barriers for the use of the MyFood system; (3) Inner settings: no culture for specific routines or systems related to nutritional care existed in the departments. However, tension for change regarding screening for malnutrition risk, monitoring and nutritional treatment was highlighted in all categories of interviewees; (4) Characteristics of the individuals: positive attitudes toward MyFood were present among the majority of the interviewees, and they expressed self-efficacy toward the perceived use of MyFood; (5) Process: providing sufficient information to everyone in the department was highlighted as key to the success of the implementation. The involvement of opinion leaders, implementation leaders, and champions was also suggested for the implementation plan.
Conclusions: This study identified several challenges in the nutritional care of hospitalized patients at risk of malnutrition and deviations from recommendations and guidelines. The MyFood system was perceived as being more precise, trustworthy, and motivational than the current practice. However, several potential barriers were identified. The assessment of the current situation and the identification of perceived barriers and facilitators will be used in planning an implementation and effect study, including the creation of an implementation plan.
doi:10.2196/11890
Keywords
Introduction
Disease-related malnutrition is a challenge in hospitals, with 30% to 50% of patients being malnourished or at risk for malnutrition [
- ]. The condition leads to higher morbidity and mortality rates among patients [ - ] and longer length of stay [ , , ]. This generates increased economic costs for the health care sector [ , , ]. According to Norwegian [ ] and European [ ] guidelines, all patients at malnutrition risk should have an individualized nutrition care plan, including documentation of nutritional status, needs, dietary intake, and recommended treatment. The reported barriers to adequate nutritional care for hospitalized malnourished patients include the absence of routines [ , ], lack of knowledge, assignment of responsibility [ ], and lack of skills and tools to estimate individual dietary needs and the energy and protein content in hospital food [ , ].Studies have shown that hospitals can benefit from implementing technology to identify, handle, and follow up with patients at risk of malnutrition. Digital tools and apps may reduce the workload of health care professionals and the time spent for nutritional assessment [
].We developed the MyFood tool, a decision support system for use among hospitalized patients at risk of malnutrition. The MyFood system includes an app for tablets and a website.
shows the intended use of the MyFood system.A consistent finding in clinical and health services research is the failure to translate evidence into practice [
]. The implementation of electronic health (eHealth) interventions is often challenging, with many failing to demonstrate predicted benefits [ ]. For implementation to succeed, it is recommended that the readiness for implementation be assessed and the barriers and facilitators be identified in advance [ ]. Theoretical frameworks may guide this assessment. The Consolidated Framework for Implementation Research (CFIR) [ ] is widely used to identify barriers and facilitators [ - ].To obtain a better understanding of how to implement the MyFood system in clinical hospital practice and to be able to create an implementation plan, we performed a qualitative study among health care professionals. The specific aims were (1) to identify current practice, routines, barriers, and facilitators for nutritional care; (2) to identify potential barriers and facilitators for the use of a decision support system (MyFood); and (3) to identify the key factors for an implementation plan.
Methods
This study is part of a research project involving the development and evaluation of a decision support system to prevent and treat disease-related malnutrition as a proof of concept. The MyFood intervention will be implemented in the hospital departments in a randomized controlled trial after this study is completed.
The MyFood System
The MyFood system is developed in response to an identified need for better tools to follow up with patients who suffer from disease-related malnutrition. The functions and content of the tool are based on the Norwegian guidelines for prevention and treatment of disease-related malnutrition [
], the Norwegian Directorate of Health recommendations on nutrition in health and care services [ ], and the recommended tasks included in the focus area of disease-related malnutrition in the Norwegian Patient Safety Program [ ]. According to the patient safety program, 4 tasks are necessary to prevent and treat disease-related malnutrition in hospitals: (1) screening for risk of malnutrition; (2) dietary assessment; (3) nutritional treatment; and (4) documentation [ ]. Hence, MyFood does not provide new tasks for health care professionals but intends to provide a system to perform and follow the guidelines and recommendations available.The MyFood system consists of 4 modules: module 1, collection of information about the patient (body weight, height, nutrition-related symptoms, nutritional situation, and allergies); module 2, dietary assessment function; module 3, evaluation of recorded dietary intake compared with individual needs for energy, protein, and liquids; and module 4, report function, including recommendations for nutrition-related actions tailored to the individual patient and a template for a nutrition care plan.
illustrates the dietary assessment and evaluation functions (modules 2 and 3) of the app. The patients record their daily dietary intake in the app. If the patient is unable to record, the nurses perform the recording on behalf of the patient. Both patients and health care professionals may keep track of the evaluation in module 3. The development of the MyFood app (modules 1 to 3) and evaluation of the dietary assessment function are described in a previous study [ ]. The report function (module 4) is intended for use by nurses or other health care professionals to monitor and follow up on a patient’s nutritional status and treatment. Module 4 is a website where the nurses gain access and retrieve information about patients by logging into the system.The MyFood system was externally developed in cooperation with selected hospital departments. The managers of the hospital departments were involved in the structural issues and facilitated the research project. Nurses, registered dietitians, and patients participated in the development of the design, content, usability, and functionality [
].Study Design and Participants
We conducted a qualitative study among health care professionals from 2 departments at a university hospital in Norway. The data collection period was January to February 2018. The study was based on 4 focus group discussions and 7 individual interviews. The health care professionals were purposively selected for the focus group discussions and interviews.
The study was performed in accordance with the Helsinki declaration and was acknowledged by the Norwegian Regional Ethical Committee (2016/1464). Written informed consent was obtained from all participants.
The Consolidated Framework for Implementation Research Framework as a Basis for the Interview Guide
The CFIR is a compilation of 39 constructs related to implementation and divided into 5 domains: characteristics of the intervention, outer setting, inner setting, characteristics of the individuals involved, and the process of implementation. These constructs can be considered when identifying local barriers to implementation [
]. According to Damschroder et al [ ], researchers may select the constructs from the CFIR that are most relevant for their study setting. In this study, the 39 constructs of the CFIR [ ] were explored and used to develop a semistructured interview guide. A total of 13 constructs were considered relevant for the context, and open-ended questions based on these were included. The interview guide was adapted for different groups of health care professionals to adjust for relevant differences in roles or tasks. For example, the construct regarding structural characteristics in the outer setting domain was only addressed to middle managers. Not all the CFIR constructs were considered relevant. For example, several of the constructs related to the process and the outer setting domains were not included. This study was a preimplementation study, and at this stage, we were most interested in the local factors in the 2 hospital departments to be able to set the performance goals of an implementation and effect study and to develop an implementation plan. The interview guide included questions about the organization and the routines related to the food and nutritional care of the patients, including responsibility, management commitment, and challenges. Perceived barriers and facilitators for the use of the MyFood tool and for performing an intervention study in the departments were also included in the guide. During the focus group discussions and interviews, the MyFood app was demonstrated for the health care professionals.Focus Group and Interview Procedure
The focus group discussions and individual interviews were conducted by the first author in a meeting room in the hospital department or at the interviewee’s office. A secretary assisted the first author during the focus groups. The focus group discussions were 45 to 55 min long, and the individual interviews were 30 to 50 min long. Focus groups facilitate communication between participants [
] and were chosen as the method for nurses because they are engaged in the daily care of patients. Each focus group included 4 to 7 nurses. The first focus group discussion served as a pilot to test the interview guide. After the focus group discussion, the interviewees were asked for feedback on the structure and phrasing of questions as well as the focus group situation. The pilot focus group did not result in any fundamental changes to the interview guide and was therefore included in the main analysis. Individual interviews were performed among the middle managers, physicians, and registered dietitians for feasibility reasons.The focus group discussions and the individual interviews were recorded with a digital voice recorder (Olympus WS-853). A dictaphone app developed by the University Center for Information Technology at the University of Oslo (UiO) [
] was used as a backup. In addition, notes were taken immediately after each focus group and interview. The audio recordings were transcribed verbatim using the software f4transkript (Marburg).Analysis
The transcripts and notes were analyzed using a thematic analysis in a stepwise manner as described by Braun and Clarke [
], using a deductive approach. The transcripts were analyzed using NVivo version 11 (QSR International). The first step in the analysis was to read through all the transcripts and take notes to obtain an overall understanding of the material. Second, initial codes were created as nodes based on the 5 domains in the CFIR framework and subnodes for the 39 CFIR constructs [ ]. Some parts of the transcripts did not directly fit into any of the CFIR constructs, and in these cases, new codes were created. Phase 3 involved searching for themes. As we followed a deductive approach, based on the domains and constructs in the CFIR framework, the primary task here involved resorting and reevaluating the codes. The final step was the review process. The codes that did not fit into the CFIR framework were particularly evaluated and reconsidered. If they were found relevant, they were included in the current constructs. A total of 22 CFIR constructs were included in the analysis ( ). The results described for the 22 CFIR constructs were reviewed and have been elaborated with regard to the specific study aims in the Discussion section.To enhance trustworthiness [
], including credibility, confirmability, dependability, and transferability [ ], the results were analyzed systematically in a stepwise manner. This included the following: involving all authors in the development of the interview guide and involving the first (MMP) and second (CV) authors in the development of the coding categories and the interpretation of the results; including different health care professionals in the interviews; and audio taping and transcribing the material verbatim.Results
Demographics
The focus group discussions included 20 nurses, with a mean age of 30 years and a range of 24 to 39 years.
shows the characteristics of the nurses in the 4 focus group discussions.The individual interviews included 2 physicians, 2 registered dietitians, and 3 middle managers. They were all female with a mean age of 39 years, ranging from 27 to 45 years.
Characteristic | FGDa 1 (n=4) | FGD 2 (n=4) | FGD 3 (n=7) | FGD 4 (n=5) | |
Gender (n) | |||||
Male | 1 | 1 | 0 | 0 | |
Female | 3 | 3 | 7 | 5 | |
Age (years) | |||||
Mean | 32 | 27 | 29 | 31 | |
Range | 26-39 | 26-30 | 25-38 | 25-36 | |
Experience (n) | |||||
<4 years | 0 | 3 | 2 | 1 | |
≥4 years | 4 | 1 | 5 | 4 |
aFGD: focus group discussion.
CFIRa domain | Barriers | Facilitators |
Intervention characteristics | Lack of automatic transfer to the electronic patient record; Hygienic aspects of using tablet computers among the patients; Potentially demotivational for patients who strive to meet their dietary needs | More trustworthy, systematic, fun, and easy to use than the current practice; May increase awareness on nutritional care and treatment; Positive attitudes among health care providers to test the MyFood tool in an intervention study; Intuitive, neat, and user-friendly design |
Outer setting | Lack of current routines for screening for malnutrition risk; Nurses’ perceptions of nagging patients regarding food intake; Different cultural backgrounds among patients; Language barriers among non-native patients; Patients fasting before surgery or medical examinations; Elderly patients not familiar with tablet computers | Potentially earlier implementation of nutritional treatment among the patients; Empowerment of patients in the recording of dietary intake |
Inner setting | Ambiguity among health care providers who have the primary responsibility for nutritional care and treatment; Prejudices among some physicians regarding the role of nutrition in the treatment process; Diverging focus between different health care providers, which may confuse the patients; Lack of culture and specific routines for nutritional care; Weak foundation on nutritional care among management; Limited availability of computers to use the MyFood report function; Limited available time | High stability in the departments’ staff of health care professionals; Good cooperation between health care professionals; Assumptions among nurses regarding the importance of nutrition; Desire among nurses for better tools for dietary assessment and follow-up; Potentially time saving if nurses do not have to do manual calculations of dietary intake themselves |
Individual characteristics | —b | Perceived self-efficacy among nurses in the ability to use the MyFood tool. |
aCFIR: Consolidated Framework for Implementation Research.
bNot applicable.
Identification of Barriers and Facilitators Using the Consolidated Framework for Implementation Research
The current practice with nutritional care, perceived barriers and facilitators for the use of the MyFood system, and the identified key aspects to include in an implementation plan are presented according to the 5 domains of the CFIR framework and subdivided into the relevant constructs (
).The perceived barriers and facilitators for use of the MyFood system are summarized in
.Intervention Characteristics
Evidence strength and quality relates to stakeholders’ perceptions of the quality and validity of evidence supporting the belief that the intervention will have desired outcomes [
]. The interviewees acknowledged that the evidence-based recommendations forming the basis of the MyFood system were known and accepted. They claimed that several of the functions in the MyFood tool were already performed at the hospital departments, although in a more unstructured manner:I think this is kind of the same, but gathered more in one place. And this [MyFood] provides a better overview.
[Registered dietitian]
Relative advantage is the stakeholders’ perceptions of the advantage of implementing the intervention versus an alternative solution [
]. Most of the interviewees perceived the dietary assessment function in MyFood as easier, more trustworthy, systematic, and precise compared with the paper-based dietary assessment forms currently in use. They also reported that MyFood could increase awareness of nutritional deficiencies and lead to the implementation of nutritional treatment at an earlier stage:I think it’s easier when you can trust it. [...] Then the physician will trust it more I think, that this is actually correct recorded, this is exactly what was eaten [...] Compared to using a form that you don’t know is complete. Then it’s easier to take action if you trust the recording. I think.
[Nurse]
The health care professionals’ perceptions of the dietary recording in the app were that it would be more fun and motivational than traditional paper recordings and that the tool was better suited for the future:
You know, we are not spoiled with new, fun technical solutions in the healthcare system. So most of us think it’s fun when something new arrives. Because it’s fun to have a gadget, you know. I think people would suddenly regard it as fun to record food, compared to that form [the paper-based dietary recording form] for which you need to scratch your head to guess the calorie intake.
[Nurse]
Adaptability relates to the degree to which an intervention can be adapted, tailored, refined, or reinvented to meet local needs [
]. The respondents gave feedback on how they perceived MyFood could fit into their existing work practice. A potential barrier was that MyFood does not communicate with the electronic patient record (EPR), which means that the health care professionals need to copy the information from the MyFood website and paste it into the EPR. However, suggestions for how to overcome this issue were proposed:It’s quite okay because we try to become paperless. And if we can just copy from that [MyFood] to the electronic journal. The dietary paper forms [paper-based dietary assessment forms used today] easily gets lost. This is like... It seems more secure.
[Nurse]
The hygienic aspects of using tablet computers among the patients were discussed, including patients with special considerations regarding infections. Several solutions for getting around this issue were suggested, for example, using a cover or plastic bag around the tablet computer.
Trialability is defined as the ability to test the intervention on a small scale in the organization and to be able to reverse course if warranted [
]. Attitudes to being part of an intervention study to test the MyFood system were positive. All groups of respondents reported being used to participate in clinical trials owing to having an ongoing study in the department at almost any time.The complexity construct describes the perceived difficulty of implementation, reflected by duration, scope, radicalness, disruptiveness, centrality, intricacy, and the number of steps required for implementation [
]. MyFood was perceived as easy to use and navigate. None of the interviewees reported that the tool seemed complex:I’m technically retarded and even I think this seems okay.
[Nurse]
Design quality and packaging are defined as the perceived excellence in how the intervention is bundled, presented, and assembled [
]. The layout of the MyFood system was described by health care professionals as having a user-friendly, intuitive, and neat design. The possibility to record only the components of a dish, in addition to the proportion of portion size ( ), was highlighted as an advantage. However, a few nurses mentioned that the illustration of the percent of achievement of energy, protein, and liquid intake compared with individual needs ( ) could potentially be demotivational for some patients:When you have only eaten 10% of your need, and feel that you have eaten a lot and that you’ll never be able to achieve your goal.
[Nurse]
Outer Setting
The patient needs and resources construct concerns the extent to which patient needs, as well as barriers and facilitators to meet those needs, are accurately known and prioritized by the organization [
]. The health care professionals elaborated on current practices and whether screening for malnutrition risk was performed. Some routines with screening existed in one of the departments. In the other department, there were preconceptions that few of the patients eat and, therefore, they did not conduct routine malnutrition risk screening:We haven’t done that [screening for malnutrition] until now. We really expect that no one eats. We expect that they either become nauseous or the food tastes strange, after quite a short time. And we expect that everyone at some time point will start with TPN [total parenteral nutrition]. So I think we aren’t good enough to..., you know, we know that we’ll get there [TPN] in 4 days anyway, so there’s no point to keep it going with ordinary food.
[Nurse]
The experience of nagging patients about food intake was highlighted by several nurses, middle managers, and registered dietitians:
[...] They [the patients] think we are whining too much about the food because they don’t regard it as very important. You know, sick, reduced appetite, and all that...
[Nurse]
However, some patients were described as being very motivated and perceived the achievement of eating enough and being independent of total parenteral nutrition (TPN) or tube feeding as their ticket home from the hospital. Those patients were often classified as being the most resourceful and understanding of the importance of nutrition for their overall health status.
Barriers to good nutritional care include fasting before surgery and being transferred to other hospitals with loss of the opportunity to follow-up. Different cultural backgrounds or language barriers of patients were mentioned as other potential barriers. Older patients were identified as a group that could potentially have some challenges with dietary recording in the app, especially the elderly who are not used to smartphones or tablets. However, most of the interviewees thought the elderly would be able to use the app after a short introduction:
I even think that elderly persons who are not so fond of technical gadgets would have understood this, you know.
[Nurse]
Inner Setting
The social architecture, age, maturity, and size of an organization constitute the structural characteristics construct [
]. Stability in the staff was described as being high. The 2 departments included in the study were organized differently. One was subdivided into groups, where each nurse belonged to one group taking care of patients in that specific group, whereas the physicians were rotating between the groups. This department also had group leaders organizing each of the groups, which was highlighted as being successful for the organization. The other department did not have any subdivision, and all nurses were potentially involved with all patients. The registered dietitians served the whole hospital, except for the children’s department.The networks and communication construct involves the nature and quality of social networks and the formal and informal communication within an organization [
]. The middle managers of the department that was subdivided into groups reported that the communication and social networks were stronger within specific groups; however, they all recognized each other as colleagues.There was uncertainty among the interviewees about the primary responsibility for nutritional care of patients. The majority described nurses as having the primary responsibility:
It’s mostly something the nurses try to talk about and assess. Ehm... but personally, I usually ask how things are going related to nutrition, and the nurses notify us how they [the patients] are doing with regard to food intake and digestion in general. [...] And we may contact the dietitian if we really need help, you know.
[Physician]
However, some respondents claimed that the physicians had formal responsibility for nutritional treatment, whereas the nurses took care of the day-to-day follow-up. Several nurses reported that they had to repeatedly remind the physicians about the prospect of tube feeding if the patient had no intake or very low intake.
The cooperation among nurses, physicians, and registered dietitians on the nutritional care of patients was, however, described as good in most cases. A diverging focus among the different groups of health care professionals was described among some of the nurses. This difference could potentially be confusing for patients:
It’s like, the physiotherapists are concerned about one thing [eg, do not drink juice because of coughing and possibility to get it in the lungs], and the dietitians are concerned about another thing [eg, eating enough protein]. We [the nurses] try to keep the threads together, and then the others are never satisfied.
[Nurse]
Culture includes norms, values, and basic assumptions of a given organization [
]. The majority of the health care professionals expressed that nutrition has an important role in the overall course of the disease for the patient, but the middle managers reported that they had no culture for specific routines related to screening for malnutrition or nutritional care:Maybe it’s kind of based on what you feel, I don’t think it’s like it’s done the same way for all... [...] I guess we aren’t good enough to add something extra to the food or think about whole fat milk instead of fat-reduced milk, butter instead of... I guess it’s like... there’s no system I think. We could get very much better.
[Middle manager]
However, a positive shift with increased focus on nutrition had occurred recently. This included increased monitoring of food intake among malnourished or at-risk patients, use of medical nutrition drinks, and availability of food service hosts in the department’s buffet kitchen. Whereas an increased focus on the importance of nutrition was mentioned by several respondents, others reported on the challenges still present, especially among the physicians:
The physicians, the surgeons, are often pretty far away from recognizing nutrition as part of the whole. So I guess that’s a group who are a little more narrow-minded than the other physicians. But, it’s understandable. When you are so highly specialized you focus on your thing.
[Registered dietitian]
Tension for change is the degree to which stakeholders perceive the current situation as intolerable or needing change [
]. A general tension for change with regard to screening for malnutrition risk, monitoring, and treatment was highlighted by all groups of health care professionals. A perception among nurses was that they should probably have taken action with regard to nutrition earlier:I wonder how many times I have heard like “oh, but I have several kilos to take away, so it’s no problem, it doesn’t matter if I don’t eat.” Because they are used to the disease passing away after a week when they are sick, and then there is no big deal because I eat when I get better. So I think we... Maybe you let it go too far before we start pushing that food, you know.
[Nurse]
The respondents were positive about the MyFood intervention because they wanted better tools for dietary assessment and follow-up. The nurses find the paper-based dietary recording forms used today to be time-consuming and unprecise. Uncertainty regarding the purpose of using the dietary recording forms existed. For some patients, dietary recording forms were used to identify the amount the patients could eat by themselves to supplement the remaining nutritional requirement through TPN or tube feeding. In some cases, the registered dietitians used the form to create a nutrition care plan. Several respondents described that the number of calories calculated from the forms was noted in the patient’s EPR, whereas others reported that this was only done in rare cases. They reported that the nurses working night shifts were supposed to perform the calorie calculations of the forms, but the compliance varied:
[...] I experience many patients having a dietary recording form lying on their nightstand that no one really... that has been lying there for several days, you know. And when the night shift replaces the form it’s like “Oh, this is from last week.”
[Nurse]
The readiness for implementation construct describes tangible and immediate indicators of organizational commitment to its decision to implement an intervention, including leadership engagement and available resources [
]. Leadership engagement is the commitment, involvement, and accountability of leaders and managers regarding the implementation [ ]. Nutrition was described as having low priority in hospital management. None of the nurses or physicians was told by the management that nutrition should be prioritized:Ehm... very seldom [signals from the management of nutrition focus]. This [nutrition] is seldom an issue from the management; at least as I have noticed.
[Physician]
A similar opinion was expressed by the middle managers. Nutrition was not a particular focus of the departments. The middle managers did not experience challenges regarding nutrition in their position between the nurses and top management:
No, I really think it works fine. Nutrition has so far been kind of a thing like anything else. It’s something we’re aware of, but maybe not enough. It constitutes a small part of all the challenges our patients have. But now that more focus has been set on nutrition, I feel that it’s established in all parts. That the nurses are positive about it and also those above me.
[Middle manager]
Available resources are the level of resources dedicated for implementation and ongoing operations, including finance, training, education, physical space, and time [
]. A concern regarding the availability of computers to check the reports in the MyFood system and read through recommended measures was raised. Some nurses mentioned a lack of time as a potential barrier:The only thing I can think of is of course time, you know. Because that’s often a challenge in everything we do and all focus areas we’re supposed to have.
[Nurse]
Others expressed that the MyFood system would be a time saver:
This would have saved us a lot of time–not having to do that calculation [manual calculation of nutritional content] yourself.
[Nurse]
Characteristics of Individuals
Knowledge and beliefs about the intervention involve individuals’ attitudes and the value placed on the intervention as well as familiarity with facts, truths, and principles related to the intervention [
]. The health care professionals expressed, in general, a positive attitude toward the MyFood intervention and saw several potential advantages related to the system compared with the current practice. Self-efficacy is the individuals’ belief in their own capabilities to execute courses of action to achieve implementation goals. The health care professionals expressed that they believed they would be able to use and follow up with MyFood.Process
Planning is defined as the degree to which a scheme or method of behavior and tasks for implementing an intervention are developed in advance, with the consideration of the quality of those schemes or methods [
]. The interviewees were asked to elaborate on their thoughts on how to perform an intervention study in the departments, including how to engage the nurses to follow up on the intervention within their busy schedules. The importance of providing everyone with information and assigning responsibility was highlighted among the nurses. The lack of information and assignment of responsibility will potentially decrease motivation. As the nurses are shift workers, it might be challenging to reach all nurses:I think it’s important to inform absolutely everyone. Because we work in triple turrets many don’t get information, especially those working night shifts, and then they don’t see the importance of it maybe, because they haven’t received the information we have gotten now. And that has a lot to do with motivation, because if you haven’t received information and don’t know why we are doing it, then no one cares. So I think it’s very important to inform absolutely everyone who is going to take part, you know.
[Nurse]
Concrete examples of suggestions received were communicating information during morning meetings, increasing the night shift by an extra 30 min at the end of the shift, or requiring the nurses to arrive half an hour before the shift to reach all nurses working on all 3 shifts. Email communication was not recommended, as many nurses do not read their emails daily. Availability and daily visits to the department, including assistance and follow-up, were suggested. The possibility for nurses to call if they have questions was also recommended.
Engaging involves attracting and involving appropriate individuals in the implementation and use of the intervention through a combined strategy of social marketing, education, role modelling, training, and other similar activities [
]. Opinion leaders are individuals in an organization who have a formal or informal influence on the attitudes and beliefs of their colleagues with regard to implementing the intervention [ ]. The physicians were described by some of the nurses as filling such an opinion leader role. However, not all nurses had this impression. Some claimed that some authorities among the nurses were more important as opinion leaders than the physicians. Group leaders and nurses with developmental responsibility were suggested as important to fill the position of implementation leaders. Nurses on the night shift were also suggested as key personnel, as they have the task of summarizing daily nutritional intake. Others expressed that a criterion for success was to assign the same responsibility to all nurses in the department. Group leaders and nurses with developmental responsibility were seen as potential champions to support and drive the implementation. Creating superusers with more experience who may inspire and help other nurses was also suggested.Discussion
Principal Findings
This study used the CFIR framework [
] to identify current practices related to nutritional care in 2 departments in a university hospital in Norway. Perceived barriers and facilitators for the use of the MyFood system were assessed, and key aspects for an implementation plan were discussed. Screening for malnutrition risk was not prevalent or established as a routine in these departments. Dietary assessment and monitoring varied, as the nurses considered current procedures as being time- and resource-demanding. The use of the MyFood system was perceived as easier, more trustworthy, precise, fun, timesaving, and potentially facilitating increased awareness and implementation of nutritional treatment compared with the current practice. Cultural and language barriers, age of the patient, hygiene, availability of computers, time, and lack of interaction with EPRs were identified as potential barriers for use.Current Nutritional Care Practices
To explore how the MyFood system may be utilized in a hospital setting, it was important to obtain information on the current practices related to nutritional care. Despite national and European guidelines for screening for malnutrition risk [
, ], this was not routinely performed in the 2 departments (CFIR: Culture). This corresponds with results from Eide et al [ ]. A recent scoping review on the use of technology to identify hospital malnutrition revealed malnutrition in the acute hospital setting to largely be an unrecognized problem, owing to insufficient monitoring, identification, and assessment of malnourished patients [ ]. We emphasized a general tension for changing nutritional care practice among all the health care professional groups investigated. For the purpose of planning the upcoming implementation and effect study in the MyFood project, it is important to be aware that screening for malnutrition risk is not routinely performed. In this study, some of the interviewees mentioned that a screening procedure was now being implemented as part of the Norwegian Patient Safety Program [ ].Dietary recording among patients at risk of malnutrition was performed to some extent, and all interviewees seemed to be aware of this practice. However, dietary recordings were seldom followed up and the forms were frequently forgotten on the patient’s nightstand. A recent study at Oslo University Hospital based on the nutritionDay survey identified that only 41% of patients at malnutrition risk received nutritional treatment [
]. Several challenges with the current practice of using paper-based forms were described. The nurses in this study found it difficult to calculate the patients’ intake of energy, and they described the hospital food lists as containing too few details. This is in line with previous findings in which Eide et al [ ] identified nurses to be uncertain about how to evaluate nutritional status, estimate nutritional needs, and measure energy and nutrient intake among hospitalized patients. An Australian study showed that poor knowledge of the nutrition care processes among nondietetic staff was a barrier to nutritional care of elderly hospitalized patients [ ]. A lack of knowledge on nutritional treatment and follow-up has been reported as an important barrier to nutritional care among physicians and nurses in Scandinavian hospitals [ , ].We did not reveal significant differences in the responses between the different groups of health care professionals. As described in the results for the networks and communication construct, the physicians stated that the nutritional care of patients was the nurses’ responsibility, whereas several nurses described the physicians as having the primary, formal responsibility. Eide et al [
] found that nurses were frustrated about the physicians’ low involvement and engagement in nutritional care of the patients. They also identified that the support from physicians in nutritional care made it easier to prioritize nutrition. This corresponds to our finding of the physicians’ important role in implementing new tools. Some nurses described communication between disciplines as challenging when different types of health care professionals have conflicting views. A literature review on communication between physicians and nurses revealed that communication tends to be unclear and unprecise, delaying patient care and increasing medical errors [ ].Facilitators for the Use of the MyFood System
The health care professionals were generally positive about the MyFood system and acknowledged that the evidence-based recommendations forming the basis of the tool [
, ] were acceptable, as described for the CFIR construct of evidence strength and quality. They perceived the tool as easy to use (CFIR: Complexity), having a user-friendly and intuitive design (CFIR: Design, quality, and packaging), and believed they would be able to use the tool (CFIR: Self-efficacy). They saw the tool as potentially time saving, more precise, and trustworthy compared with current practices, as it related to the CFIR constructs of relative advantage and knowledge and beliefs about the intervention. The perceptions of the preciseness and trustworthiness of the dietary recording function in the MyFood system seemed to be based on assumptions that everything recorded in the app would be correct. Self-reported dietary assessment methods are, however, often associated with errors. The memory of intake, lack of motivation to record over several days, ability to estimate portion sizes, and perceptions of socially desirable responses are well-known challenges in self-reported intake [ ]. An evaluation of the dietary recording function in the MyFood app found that MyFood was relatively accurate in estimating the patients’ intake of energy, protein, liquids, food, and beverages [ ].The MyFood system was perceived as potentially more fun and motivational to use compared with the current practice. Studies among adolescents have shown that dietary assessment using technology is preferred over paper-based food recording because electronic methods are perceived as more fun and motivational [
, ]. A systematic review of electronic methods to record food intake described that seeing progress toward fulfilment of goals can be highly motivational [ ].Barriers for the Use of the MyFood System
Although the health care professionals were positive about MyFood, several potential barriers were identified. MyFood was recognized as potentially time saving, but some also described time used to follow up as a barrier for use. A lack of automatic transfer to the EPR was described as another potential barrier, related to the adaptability construct. Lack of time and integration with the EPR were also found to be barriers in the implementation of the eHealth intervention Choice for symptom reporting into clinical practice in a Norwegian university hospital [
]. Another potential barrier linked to adaptability was hygiene aspects related to the use of tablet computers among patients.Perceived barriers associated with the CFIR construct patient needs and resources concerned different languages and cultural backgrounds among the patients. The MyFood tool includes both icons and pictures that may overcome some language challenges. If the use of the MyFood system turns out to be effective, the inclusion of several languages may be considered in the future. With regard to cultural barriers and patients eating foods not included in the hospital’s assortment, the MyFood app includes the possibility to record intake manually using a description of food or beverages consumed. In the long run, a wider range of food items may be included in the system. Challenges related to hygiene aspects may be solved by using plastic covers around the tablets. Older age was reported as a potential barrier to the use of MyFood owing to an increased risk of cognitive deficits or low self-efficacy among the elderly. However, qualitative studies among older persons have demonstrated that elderly people are often positive about using tablets and eager to learn [
, ]. A recent study describing an app to inspire home-dwelling elderly at nutritional risk to eat healthy foods showed that the elderly found the app easy to use [ ].Key Aspects for an Implementation Plan
The health care professionals were positive about performing an intervention study to test the MyFood system in their department (CFIR: Trialability). The results related to the CFIR construct planning, where the interviewees elaborated on their thoughts on how to perform the intervention study, follow up, and engage the nurses, will be particularly relevant for the creation of an implementation plan. Of interest for the implementation plan are also results from the CFIR constructs: structural characteristics, networks and communication, and available resources. Important elements may include ongoing training, local technical assistance, clinical supervision, educational materials, support, availability, establishing an implementation team, and organizing clinician implementation meetings. To engage potential users of MyFood and identify opinion leaders, the involvement of potential champions and early adopters may also be of importance. These findings are previously described as relevant implementation strategies in the Expert Recommendations for Implementing Change project [
, ]. An important finding in this study is that nutritional care has low priority in management (CFIR: Leadership engagement). Leadership support and engagement are crucial [ ] for successful implementation [ ] and strategies toward the leaders should be included in the implementation plan.Strengths and Limitations
The strengths of the study are the inclusion of different health care professions and middle managers to reveal several views. The majority of the respondents were nurses, as they were considered to be the most important group with regard to the day-to-day nutritional care of the patients. Saturation is often described as the basis for sample size in qualitative studies [
]. However, this might not be the most appropriate [ ]. Malterud et al [ ] describe information power as related to the specificity of experiences, knowledge, or properties among the participants included in the sample. In this study, the health care professionals were holding characteristics specific to the study aim, in light of their professions. The aims of this study were relatively narrow and precise. Information power indicates that the more information the sample holds, relevant to the actual study, the lower the number of participants needed [ ]. On the basis of these criteria, we considered our sample size to be sufficient.The focus groups were originally composed of nurses with the same level of work experience to ensure that everyone’s voice was heard and to enable the more inexperienced nurses to talk freely in a separate group that was not dominated by more experienced nurses. Owing to illness among some nurses on the days of the interviews, some adjustments had to be made to be able to perform the focus group discussion with a sufficient number of nurses. Interviewing patients may have strengthened this study. Therefore, the patients will be included in the planned study of the implementation and effect of using the MyFood system.
Using an existing framework within the field of implementation science is considered an important strength to better understand, describe, and identify factors that predict the likelihood of implementation success. The CFIR framework identified the importance of knowledge for designing an implementation plan [
].This study was performed in the same departments where the planned implementation and effect study will take place, thereby providing a local identification of potential barriers and facilitators that may be crucial. However, conducting the study at only 2 departments in 1 hospital may imply that our results are not necessarily representative of other departments or hospitals. A limitation is that this study was performed before the implementation of the MyFood system. The impression of MyFood was therefore based on a demonstration and perceived barriers and facilitators for use. This perception does not necessarily correspond to the real barriers and facilitators experienced in practice.
Whether or not the MyFood or a similar system, at some point in time, will be included and implemented in the Norwegian health care system is yet to be determined. There is a shift in the health care system toward increased use of digital systems. A need for the development of information and communication technology tools to screen, assess needs, monitor food intake, create a nutrition care plan, and follow up on disease-related malnutrition is described in a report from the Norwegian National Council for Nutrition [
].Conclusions
This study identified several challenges in the nutritional care of hospitalized patients at malnutrition risk in 2 departments in a university hospital in Norway. The use of the decision support system MyFood was anticipated to have several advantages compared with the current practice with nutritional follow-up. The MyFood system was perceived as more precise, trustworthy, fun, and motivational than the current practice. However, cultural, language, age, and hygiene aspects were perceived as potential barriers. The identification of perceived barriers and facilitators will be used in the creation of a plan to implement the MyFood system in clinical practice in an implementation and effect study.
Acknowledgments
The development of the MyFood app was performed on the services for sensitive data (tjenester for sensitive data [TSD]) facilities, owned by the UiO, operated and developed by the TSD service group at the UiO IT Department.
Conflicts of Interest
None declared.
References
- Henriksen C, Gjelstad IM, Nilssen H, Blomhoff R. A low proportion of malnourished patients receive nutrition treatment-results from nutritionDay. Food Nutr Res 2017;61(1):1391667 [FREE Full text] [CrossRef] [Medline]
- Poulia K, Klek S, Doundoulakis I, Bouras E, Karayiannis D, Baschali A, et al. The two most popular malnutrition screening tools in the light of the new ESPEN consensus definition of the diagnostic criteria for malnutrition. Clin Nutr 2017 Dec;36(4):1130-1135. [CrossRef] [Medline]
- Rondel AL, Langius JAE, de van der Schueren MA, Kruizenga HM. The new ESPEN diagnostic criteria for malnutrition predict overall survival in hospitalised patients. Clin Nutr 2018 Dec;37(1):163-168. [CrossRef] [Medline]
- Schindler K, Pernicka E, Laviano A, Howard P, Schütz T, Bauer P, et al. How nutritional risk is assessed and managed in European hospitals: a survey of 21,007 patients findings from the 2007-2008 cross-sectional nutritionDay survey. Clin Nutr 2010 Oct;29(5):552-559. [CrossRef] [Medline]
- Tangvik RJ, Tell GS, Eisman JA, Guttormsen AB, Henriksen A, Nilsen RM, et al. The nutritional strategy: four questions predict morbidity, mortality and health care costs. Clin Nutr 2014 Aug;33(4):634-641 [FREE Full text] [CrossRef] [Medline]
- Agarwal E, Ferguson M, Banks M, Batterham M, Bauer J, Capra S, et al. Malnutrition and poor food intake are associated with prolonged hospital stay, frequent readmissions, and greater in-hospital mortality: results from the Nutrition Care Day Survey 2010. Clin Nutr 2013 Oct;32(5):737-745. [CrossRef] [Medline]
- Correia MI, Waitzberg DL. The impact of malnutrition on morbidity, mortality, length of hospital stay and costs evaluated through a multivariate model analysis. Clin Nutr 2003 Jun;22(3):235-239. [Medline]
- Norman K, Pichard C, Lochs H, Pirlich M. Prognostic impact of disease-related malnutrition. Clin Nutr 2008 Feb;27(1):5-15. [CrossRef] [Medline]
- Kondrup J, Johansen N, Plum LM, Bak L, Larsen IH, Martinsen A, et al. Incidence of nutritional risk and causes of inadequate nutritional care in hospitals. Clin Nutr 2002 Dec;21(6):461-468. [Medline]
- Kyle UG, Genton L, Pichard C. Hospital length of stay and nutritional status. Curr Opin Clin Nutr Metab Care 2005 Jul;8(4):397-402. [Medline]
- Freijer K, Tan SS, Koopmanschap MA, Meijers JM, Halfens RJ, Nuijten MJ. The economic costs of disease related malnutrition. Clin Nutr 2013 Feb;32(1):136-141. [CrossRef] [Medline]
- The Norwegian Directorate of Health. National guidelines for preventing and treatment of malnutrition (Nasjonale faglige retningslinjer for forebygging og behandling av underernæring). Oslo: The Norwegian Directorate of Health; 2009.
- Kondrup J, Allison SP, Elia M, Vellas B, Plauth M, EducationalClinical PCESOPN. ESPEN guidelines for nutrition screening 2002. Clin Nutr 2003 Aug;22(4):415-421. [Medline]
- Eide HD, Halvorsen K, Almendingen K. Barriers to nutritional care for the undernourished hospitalised elderly: perspectives of nurses. J Clin Nurs 2015 Mar;24(5-6):696-706 [FREE Full text] [CrossRef] [Medline]
- Khalaf A, Berggren V, Westergren A. Caring for undernourished patients in an orthopaedic setting. Nurs Ethics 2009 Jan;16(1):5-18. [CrossRef] [Medline]
- Fjeldstad SH, Thoresen L, Mowé M, Irtun Ø. Changes in nutritional care after implementing national guidelines-a 10-year follow-up study. Eur J Clin Nutr 2018 Jul;72(7):1000-1006. [CrossRef] [Medline]
- Ross LJ, Mudge AM, Young AM, Banks M. Everyone's problem but nobody's job: staff perceptions and explanations for poor nutritional intake in older medical patients. Nutr Diet 2011;68(1):41-46. [CrossRef]
- Trtovac D, Lee J. The use of technology in identifying hospital malnutrition: scoping review. JMIR Med Inform 2018 Jan 19;6(1):e4 [FREE Full text] [CrossRef] [Medline]
- Grimshaw JM, Eccles MP, Lavis JN, Hill SJ, Squires JE. Knowledge translation of research findings. Implement Sci 2012 May 31;7:50 [FREE Full text] [CrossRef] [Medline]
- Murray E, Burns J, May C, Finch T, O'Donnell C, Wallace P, et al. Why is it difficult to implement e-health initiatives? A qualitative study. Implement Sci 2011;6:6 [FREE Full text] [CrossRef] [Medline]
- Powell BJ, Waltz TJ, Chinman MJ, Damschroder LJ, Smith JL, Matthieu MM, et al. A refined compilation of implementation strategies: results from the Expert Recommendations for Implementing Change (ERIC) project. Implement Sci 2015 Feb 12;10:21 [FREE Full text] [CrossRef] [Medline]
- Damschroder LJ, Aron DC, Keith RE, Kirsh SR, Alexander JA, Lowery JC. Fostering implementation of health services research findings into practice: a consolidated framework for advancing implementation science. Implement Sci 2009;4:50 [FREE Full text] [CrossRef] [Medline]
- Garbutt JM, Dodd S, Walling E, Lee AA, Kulka K, Lobb R. Barriers and facilitators to HPV vaccination in primary care practices: a mixed methods study using the Consolidated Framework for Implementation Research. BMC Fam Pract 2018 May 07;19(1):53 [FREE Full text] [CrossRef] [Medline]
- Hadjistavropoulos HD, Nugent MM, Dirkse D, Pugh N. Implementation of internet-delivered cognitive behavior therapy within community mental health clinics: a process evaluation using the consolidated framework for implementation research. BMC Psychiatry 2017 Sep 12;17(1):331 [FREE Full text] [CrossRef] [Medline]
- Sopcak N, Aguilar C, O'Brien MA, Nykiforuk C, Aubrey-Bassler K, Cullen R, et al. Implementation of the BETTER 2 program: a qualitative study exploring barriers and facilitators of a novel way to improve chronic disease prevention and screening in primary care. Implement Sci 2016 Dec 01;11(1):158 [FREE Full text] [CrossRef] [Medline]
- Varsi C, Ekstedt M, Gammon D, Ruland CM. Using the consolidated framework for implementation research to identify barriers and facilitators for the implementation of an internet-based patient-provider communication service in five settings: a qualitative study. J Med Internet Res 2015 Nov 18;17(11):e262 [FREE Full text] [CrossRef] [Medline]
- The Norwegian Directorate of Health. [The Dietary Planner: Supervisor in Nutrition in Health and Care Services]. Oslo: Helsedirektoratet; 2012.
- The Norwegian Directorate of health. [In Safe Hands]. The Norwegian Patient Safety Programme URL: https://www.pasientsikkerhetsprogrammet.no/om-oss/innsatsomr%C3%A5der/ern%C3%A6ring [accessed 2018-05-10] [WebCite Cache]
- Paulsen MM, Hagen MLL, Frøyen MH, Foss-Pedersen RJ, Bergsager D, Tangvik RJ, et al. A dietary assessment app for hospitalized patients at nutritional risk: development and evaluation of the MyFood app. JMIR Mhealth Uhealth 2018 Sep 07;6(9):e175 [FREE Full text] [CrossRef] [Medline]
- Pope C, Mays N. Qualitative Research in Health Care. USA: BMJ Books, Blackwell Publishing LTD; 2006:9781405135122.
- University of Oslo. 2018. [Webform-dictaphone] URL: https://www.uio.no/tjenester/it/applikasjoner/nettskjema/hjelp/tips-triks/diktafon.html [accessed 2018-04-05] [WebCite Cache]
- Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol 2006 Jan;3(2):77-101 [FREE Full text] [CrossRef]
- Nowell LS, Norris JM, White DE, Moules NJ. Thematic analysis: striving to meet the trustworthiness criteria. Int J Qual Methods 2017 Oct 02;16(1):160940691773384. [CrossRef]
- Lincoln Y, Guba E. Naturalistic Inquiry. Newbury Park: Sage; 1985.
- Mowe M, Bosaeus I, Rasmussen HH, Kondrup J, Unosson M, Irtun Ø. Nutritional routines and attitudes among doctors and nurses in Scandinavia: a questionnaire based survey. Clin Nutr 2006 Jun;25(3):524-532. [CrossRef] [Medline]
- Tang CJ, Chan SW, Zhou WT, Liaw SY. Collaboration between hospital physicians and nurses: an integrated literature review. Int Nurs Rev 2013 Sep;60(3):291-302. [CrossRef] [Medline]
- Willett W. Nutritional Epidemiology. Oxford: Oxford University Press; 2012:0190240849.
- Aizawa K, Maeda K, Ogawa M, Sato Y, Kasamatsu M, Waki K, et al. Comparative study of the routine daily usability of FoodLog: a smartphone-based food recording tool assisted by image retrieval. J Diabetes Sci Technol 2014 Mar;8(2):203-208 [FREE Full text] [CrossRef] [Medline]
- Boushey CJ, Kerr DA, Wright J, Lutes KD, Ebert DS, Delp EJ. Use of technology in children's dietary assessment. Eur J Clin Nutr 2009 Feb;63(Suppl 1):S50-S57 [FREE Full text] [CrossRef] [Medline]
- Rusin M, Arsand E, Hartvigsen G. Functionalities and input methods for recording food intake: a systematic review. Int J Med Inform 2013 Aug;82(8):653-664. [CrossRef] [Medline]
- Varsi C, Ekstedt M, Gammon D, Børøsund E, Ruland CM. Middle managers' experiences and role in implementing an interactive tailored patient assessment eHealth intervention in clinical practice. Comput Inform Nurs 2015 Jun;33(6):249-257. [CrossRef] [Medline]
- Alvseike H, Brønnick K. Feasibility of the iPad as a hub for smart house technology in the elderly; effects of cognition, self-efficacy, and technology experience. J Multidiscip Healthc 2012;5:299-306 [FREE Full text] [CrossRef] [Medline]
- Vaportzis E, Clausen MG, Gow AJ. Older adults perceptions of technology and barriers to interacting with tablet computers: a focus group study. Front Psychol 2017 Oct 04;8:1687 [FREE Full text] [CrossRef] [Medline]
- Fuglerud KS, Leister W, Bai A, Farsjø C, Moen A. Inspiring Older People to Eat Healthily. Stud Health Technol Inform 2018;249:194-198. [Medline]
- Waltz TJ, Powell BJ, Matthieu MM, Damschroder LJ, Chinman MJ, Smith JL, et al. Use of concept mapping to characterize relationships among implementation strategies and assess their feasibility and importance: results from the Expert Recommendations for Implementing Change (ERIC) study. Implement Sci 2015 Aug 07;10:109 [FREE Full text] [CrossRef] [Medline]
- Kirchner JE, Parker LE, Bonner LM, Fickel JJ, Yano EM, Ritchie MJ. Roles of managers, frontline staff and local champions, in implementing quality improvement: stakeholders' perspectives. J Eval Clin Pract 2012 Feb;18(1):63-69. [CrossRef] [Medline]
- Sandström B, Borglin G, Nilsson R, Willman A. Promoting the implementation of evidence-based practice: a literature review focusing on the role of nursing leadership. Worldviews Evid Based Nurs 2011 Dec;8(4):212-223. [CrossRef] [Medline]
- Morse JM. The significance of saturation. Qual Health Res 2016 Jul;5(2):147-149 [FREE Full text] [CrossRef]
- O'Reilly M, Parker N. 'Unsatisfactory Saturation': a critical exploration of the notion of saturated sample sizes in qualitative research. Qual Res 2012 May 17;13(2):190-197. [CrossRef]
- Malterud K, Siersma VD, Guassora AD. Sample size in qualitative interview studies: guided by information power. Qual Health Res 2016 Nov;26(13):1753-1760. [CrossRef] [Medline]
- Nilsen P. Making sense of implementation theories, models and frameworks. Implement Sci 2015;10:53 [FREE Full text] [CrossRef] [Medline]
- The Norwegian National Council for Nutrition. 2017. [Disease-related malnutrition. Challenges, opportunitites, and recommendations] URL: https://helsedirektoratet.no/Lists/Publikasjoner/Attachments/1287/Sykdomsrelatert%20underernæring%20-%20Utfordringer%20muligheter%20og%20anbefalinger%20IS-0611.pdf [accessed 2018-05-04] [WebCite Cache]
Abbreviations
CFIR: Consolidated Framework for Implementation Research |
eHealth: electronic health |
EPR: electronic patient record |
TPN: total parenteral nutrition |
TSD: tjenester for sensitive data (services for sensitive data) |
UiO: University of Oslo |
Edited by G Eysenbach; submitted 09.08.18; peer-reviewed by J Constantin, AK Lindroos, M Mileski, G Fico; comments to author 07.01.19; revised version received 18.02.19; accepted 29.03.19; published 09.05.19
Copyright©Mari Mohn Paulsen, Cecilie Varsi, Ingvild Paur, Randi Julie Tangvik, Lene Frost Andersen. Originally published in JMIR Formative Research (http://formative.jmir.org), 09.05.2019.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on http://formative.jmir.org, as well as this copyright and license information must be included.