Background: The design and conduct of research to develop, test, and evaluate complex health care interventions is challenging. Although the existing literature describes key challenges associated with the design and conduct of definitive (evaluation) trials, there is a lack of information concerning specific challenges associated with the intervention development phase and setup of feasibility studies. In particular, the literature is scarce concerning the challenges associated with conducting digital health care research, such as research on internet-administered interventions and research using digital features to support the execution of study procedures (eg, recruitment, consent, retention, and data collection and management). This study is conducted in the context of the intervention development and feasibility study setup phases of an internet-administered, guided, low-intensity cognitive behavioral therapy–based intervention for parents of children previously treated for cancer.
Objective: The aim of this study is to explore the challenges experienced during the development phase of the internet-administered intervention and digital features to support the execution of the study procedures and a feasibility study setup.
Methods: To explore the key challenges experienced, we conducted a document analysis of written records from all study meetings held by the research team (meeting minutes) between June 7, 2018, and January 10, 2020, guided by a thematic analysis approach. Furthermore, discussion groups with members of the research team were held to develop a more detailed understanding of the key challenges experienced. Methods and results are reported in accordance with the relevant items from the Standards for Reporting Qualitative Research checklist.
Results: Six main themes were identified: decision-making and communication, expertise, external constraints, flexibility, planning and scheduling, and technical constraints.
Conclusions: Significant challenges were experienced during the intervention development and setup phases of the feasibility study. Implications are discussed to inform future design, conduct, and planning of internet-administered intervention development and feasibility studies, especially within the context of digital health care research.
Clinical trials are essential to inform evidence-based health care . However, clinical trials are costly and resource-intensive for both researchers and funders [ ], and approximately 85% of research investment is wasted [ ]. Examples of research waste include asking the wrong research questions [ ], using inappropriate study designs and methods [ , ], poor and biased reporting [ ], and underreporting [ ]. There are a number of inefficiencies related to clinical trial conduct that can lead to research waste, especially given that the design and efficient conduct of clinical trials is challenging [ ] and operationally complex [ , ]. Challenges include the creation and management of trial procedures and materials; communication with multiple stakeholders; ethical and regulatory requirements; the recruitment, training, and turnover of trial personnel; budget management; the recruitment and retention of trial participants; and data monitoring and assurance of data quality [ - ].
While numerous barriers have been identified in the successful conduct of clinical trials , there has been less focus on sharing experiences and lessons learned by trial teams [ ]. Publications tend to focus on trial outcomes, with little reporting on trial conduct [ ]. Consequently, there is scarce evidence to inform decisions concerning the management of clinical trials [ ]. Furthermore, in contrast to the literature focused on the conduct of definitive (evaluation) trials, there is a lack of literature concerning the challenges of conducting intervention development research and pilot and feasibility studies, following the Medical Research Council (MRC) framework for the development and evaluation of complex interventions [ ]. Indeed, there is limited guidance on how to design and conduct feasibility studies, leading to poor study design and reporting [ , ]. Given that many preparatory development [ , ] and clinical, methodological, and procedural uncertainties require testing [ ] before progressing to a definitive trial, development and feasibility phases are substantial works in and of themselves, with findings often underreported [ , ].
In the context of this feasibility study (ENGAGE study: ISRCTN 57233429; ISRCTN 18404129) , the internet-administered low-intensity cognitive behavioral therapy (LICBT) intervention (the EJDeR intervention) is delivered on the U-CARE-portal, hereafter referred to as the Portal [ ]. The Portal is designed to deliver internet-administered cognitive behavioral therapy interventions and support the execution of study procedures, for example, randomization, web-based informed consent, and data collection [ ]. Digital technologies (eg, technologies using the internet) are facilitating the delivery of health care worldwide [ , ]. Indeed, internet-administered interventions have been posited as a solution to the global mental health crisis and to help overcome significant barriers to treatment access (eg, geographical and resource barriers) [ ]. The promise of digital technologies to deliver mental health care interventions has been further amplified by the current COVID-19 pandemic [ ], given the negative psychosocial consequences of the pandemic itself [ , ] and the ability to facilitate access to mental health care [ ]. However, while the evidence base for internet-administered psychological interventions is well-established [ ], many publicly available interventions delivered via digital technologies are not evidence-based [ ].
Furthermore, digital technologies are being increasingly incorporated into the design and execution of health care research, for example, to facilitate recruitment, enhance retention, and collect data [, , , ]. The use of digital technologies in health care research has intensified during the COVID-19 pandemic in an attempt to continue health care research in the absence of in-person contact [ ]. Digital health care research is associated with reduced trial costs, improved trial efficiency [ , ], and recruitment of more diverse populations [ , ]. As such, the use of digital features to execute health care research is likely to continue to grow beyond the pandemic [ ]. However, at present, challenges related to the conduct of digital health care research are less well documented [ , ] and focus on topics such as patient privacy and confidentiality, adequate infrastructure, data accuracy and integrity, and user acceptability [ , , ]. Some challenges are beginning to be addressed by the development of new ethical and regulatory standards and increased provision of guidance for investigators in the use of digital technologies [ , , ]. However, the literature remains in its infancy, and there is a need for researchers to publish their experiences in conducting digital health care research [ ].
Context: The ENGAGE Feasibility Study
Globally, approximately 300,000 children are diagnosed with cancer each year , and cancer remains a leading cause of death in children worldwide [ ]. Typically, parents are the primary source of support for children with cancer and report significant negative psychological [ - ] and socioeconomic impacts [ - ]. Mental health difficulties are reported after cancer treatment [ , , ] and years after the end of treatment [ , ]. However, parents of children treated for cancer report an unmet need for psychological support [ - ]. To improve access to evidence-based psychological support, innovative solutions are being developed worldwide [ ]. One such solution is the provision of guided internet-administered LICBT, which may help improve access to psychological support for parents of children treated for cancer.
Given the promise of internet-administered LICBT, we have adopted the MRC complex interventions framework  to develop an internet-administered LICBT intervention (the EJDeR intervention) tailored to the specific needs of parents of children previously treated for cancer [ ]. Significant previous research [ , , , - ] has informed the development of the EJDeR intervention alongside multiple stakeholders, including parent research partners (PRPs), clinical psychologists, software developers, and pediatric oncologists. The EJDeR intervention is delivered on the Portal and includes written, film, audio content, videoconferencing, and in-portal email guidance from an e-therapist [ ].
The objectives of this study are to explore the challenges experienced during (1) the development phase of the internet-administered intervention and digital features to support the execution of the study procedures and (2) a feasibility study setup [, ]. To explore the key challenges experienced, we conducted a document analysis of written records from all study meetings held by the research team (meeting minutes) between June 7, 2018, and January 10, 2020, guided by a thematic analysis approach. Furthermore, discussion groups with members of the research team were held to develop a more detailed understanding of the key challenges experienced.
The methods and findings are reported in accordance with relevant items from the Standards for Reporting Qualitative Research checklist .
Qualitative Approach and Research Paradigm
A document analysis  of study meeting minutes was guided by a thematic analysis approach [ ]. Document analysis was considered suitable given that it allows an examination of contextual and background information to provide historical insights into the challenges experienced by the ENGAGE research team and provides a way of tracking challenges over time [ ].
Researcher Characteristics and Reflexivity
Document analysis was primarily conducted by 2 members of the research team (MK and JW). MK is a female research assistant with a Bachelor of Sport Science who joined the research team toward the end of the study setup phase and was therefore able to conduct the analysis from an outsider perspective. MK was trained in using thematic analysis by JW. JW is a female researcher with a PhD in psychology with experience in conducting qualitative research. JW is a coinvestigator, supervisor of the study coordinator, and research assistant in the research team and has been a member of the research team from the beginning of the study setup phase. All manuscript authors are, or have been, members of the research team.
All meeting minutes were taken by the ENGAGE study coordinator, or a substitute, and meetings were held at the Department of Women and Children’s Health, Uppsala University, Uppsala, Sweden. In most cases, all core members of the research team were in attendance, including the principal investigator (LVE), researchers (JW and HG), study coordinator (KN), Portal development team coordinator (YHS), and research assistants (JH and MK). Occasionally, wider research team members, such as software developers, licensed psychologists, and student interns, attended meetings. Meetings were held weekly and scheduled for 2 hours. Meeting minutes were circulated to research team members for approval after the meeting and saved on a shared folder and thus visible to all research team members.
Ethical approval was not deemed necessary as the study analyzed documentary data only.
All meeting minutes (N=78) from ENGAGE study meetings conducted between June 7, 2018, and January 10, 2020, were included to account for meetings concerning the development of the intervention, the development of digital features to support study execution, and the concurrent setup of the feasibility study. The number of meeting minute pages ranged from 1 to 6 and the word count range was 85-2350. Meeting minutes were designed to document (1) progress toward specific study milestones, (2) problems or challenges arising, (3) decisions, and (4) actions moving forward.
Given that documents cannot be considered to be completely accurate recordings of past events , throughout the analysis process, MK and JW actively reflected upon the meaning of the meeting minute content and how this was related to challenges experienced by the research team. Guidelines for conducting a reflexive thematic analysis followed [ ]. An inductive approach was adopted with codes and themes driven by the data [ ]. The analysis of meeting minutes took place between January and May 2020. Meeting minutes were read multiple times by MK to enable familiarization with the data set as a whole. MK identified initial codes across the data set to begin organizing the data. To enhance rigor, initial codes were discussed in weekly meetings with JW. After initial coding across the data set was complete, MK sorted initial codes into initial main themes and subthemes within main themes, with continued weekly discussions with JW to establish initial consensus and refine the main themes and subthemes [ ]. Next, the initial thematic map ( ) was presented to the research team (HG, JH, JW, KN, LVE, MK, and YHS) for feedback in a face-to-face discussion group held on February 14, 2020 (120 minutes). During the discussion group, the initial thematic map was presented, and discussions were held concerning whether the initial main themes and subthemes reflected the main challenges experienced. One major discussion concerned whether the Portal was a main theme or an element of all the main themes.
Subsequently, a refined thematic map () was developed by MK with continued weekly discussions with JW and applied across the data set to ensure a good fit to the data. A second face-to-face discussion group (120 minutes) was held on March 5, 2020, to present the refined thematic map to the research team (HG, JH, JW, KN, LVE, MK, and YHS) for further discussion and feedback.
On the basis of feedback received in the second discussion group, the main themes and subthemes were further revised by MK and JW, and a final thematic map was developed with six main themes and no subthemes (). The thematic map, alongside descriptions of each main theme, was sent to the wider research team via email for final feedback and approval, including the provision of salient examples of challenges missing from descriptors.
To enhance the trustworthiness of the analysis, a number of strategies were adopted as follows: (1) member-checking with research team members to test that the main themes and detailed descriptors were recognized , (2) use of thematic maps to explore the main theme and subtheme connections during the analysis process, (3) repeated returning to the whole data set to check the adequacy of each thematic map, and (4) member-checking and (5) peer debriefing with initial codes and subsequent thematic maps reviewed and discussed by MK and JW throughout the analysis process.
The thematic analysis resulted in a three-step process, an initial thematic map (), a refined thematic map ( ), and a final thematic map ( ). presents the content of the final main themes.
Final themes and descriptors of challenges.
Descriptors of challenges identified in main themes
- Decision-making and communication
- Involving all members of the research team in making certain decisions. Considering the different preferences of research team members
- Communicating between the research team and the portal development team concerning technical requirements, time estimation, and planning
- Identifying and recruiting study personnel with the necessary expertise and experience to design and set up the feasibility study
- External constraints
- Uncertainties regarding time taken for tasks to be completed by contracted personnel outside of the research team and time taken for public authorities to make decisions and provide approvals
- Changing intervention requirements, elements of the study protocol, and associated study documentation
- Planning and scheduling
- Identifying required tasks to complete
- Estimating time to complete tasks
- Deciding the order in which tasks should be completed, especially concerning task interdependency
- Technical constraints
- Managing within the constraints of available technical resources and known limitations of the Portal
- Identifying necessary technical requirements on the Portal
Decision-making and Communication
Decision-making was a significant challenge during the development of the intervention and digital features to support study execution, and the feasibility study setup phase, resulting in time inefficiencies and a delay to study start. A particular challenge was related to occasions in which the entire research team was involved in the decision-making process. Group decision-making resulted in the time taken for the principal investigator or a researcher to make a final decision being longer than was always necessary. A tension was identified concerning the desire to build and maintain a supportive team environment versus the need for research team members with more experience and leadership responsibilities to make quick decisions. This resulted in examples from the meeting minutes of the group decision-making process taking weeks, or sometimes months, before a final decision was made by the principal investigator. Group decision-making was particularly challenging when research team members had different preferences, opinions, and different levels of experience and expertise, resulting in difficulties in reaching a consensus. One example was deciding on a name for the intervention, whereby several opinion polls were made over a number of weeks before a final decision was made.
Communication was also identified as a challenge that contributed to difficulties in the decision-making process. Challenges regarding communication were particularly exacerbated in situations where research team members had different areas of expertise. A prevalent example concerned communication between researchers and the Portal development team. For example, during the development of the intervention and digital features to support study execution, the research team had many technical requirements. However, sometimes, there was a mismatch between researchers and the Portal development team in their understanding of technical requirements (eg, how interactive homework exercises were presented on the Portal). Furthermore, difficulties were identified concerning time estimation and subsequent planning. For example, the Portal team at times underestimated the length of time to deliver a technical requirement, or technical requirements were changed by the research team, subsequently impacting the study time plan. In addition, researchers sometimes held unrealistic expectations concerning delivery time for a new requirement, negatively impacting the study time plan. Difficulties with communication were more prevalent in the study setup phase. During the study setup phase, a software developer from the Portal team began to attend research team meetings, helping to avoid misunderstanding and clarifying requirements.
Identifying personnel with the required research and clinical expertise was challenging. One challenge related to the recruitment of research team personnel with the correct competencies and experience, for example, prior experience of trial management, design and conduct of feasibility studies, and/or experience of internet-administered intervention development. For example, multiple recruitment rounds were held to recruit a postdoctoral researcher to work full-time in the study; however, no candidates with the correct expertise were found. In addition, retention of research team personnel (eg, research assistants) was raised as a difficulty, for example, staff turnover was experienced due to short-term temporary contracts and the uncertainty of future employment being tied to study funding. As such, difficulties were experienced needing to replace temporary personnel and provide training to new research team members, taking significant time and further impacting the study time plan. This was especially problematic as it was difficult to recruit research assistants with prior experience of working in similar research environments, and thus the need for training was understandably greater.
Another challenge related to expertise concerned the further development and refinement of the intervention material. Difficulties were experienced in identifying a licensed psychologist with specific experience in writing LICBT interventions in Swedish to finalize the intervention material. Ultimately, 2 English-speaking experts in LICBT were engaged in later iterations of the intervention material after feedback from PRPs. However, this resulted in the need for translation into Swedish, back-translation into English, and cultural adaptation of the intervention material, which further delayed intervention development.
Challenges were also experienced when expectations for the required expertise changed. One example is related to illustrations for the EJDeR intervention, where illustrations were initially developed by a research team member with some experience in using design software. However, it was later decided to engage an external company with specialist experience of developing illustrations for websites and mobile devices. This decision was made to enhance the quality of the EJDeR intervention alongside considerations concerning resource allocation, as illustration design is time consuming and the skillset of the member of the research team was better used elsewhere.
During the development of the intervention and digital features to support study execution and feasibility study setup phases, the research team liaised with public authorities (eg, the Swedish Ethical Review Authority, the Childhood Cancer Registry, and the Swedish Tax Agency), external companies (eg, professional illustrators), and external contractors (eg, licensed psychologists and e-therapists). One challenge relates to the time taken by public authorities to make decisions and provide approval. A specific example related to the submission of an ethical amendment to the Swedish Ethical Review Authority regarding the implementation of a study within a trial  embedded in the ENGAGE feasibility study. The study within a trial was approved; however, the authority raised concerns regarding previously approved parts of the study design (eg, an opt-out recruitment procedure involving telephone reminders to parents who do not actively decline study participation). Raising concerns regarding a previously approved application was potentially because the Swedish government replaced regional ethical review boards with a single national review authority during the study setup phase, meaning applications and requests for amendment were no longer necessarily reviewed by the same local authority. This resulted in additional delays, with the research team needing to respond to the authority to further justify the use of an opt-out procedure. Additional delays were experienced by external companies and contractors, for example, not working at full capacity during the summer of 2019 (eg, taking a summer vacation). Another delay related to external factors was experienced when Uppsala University upgraded their servers, resulting in members of the Portal team being allocated to mitigate risks to the Portal during the upgrade and thus unable to prioritize technical requirements for the study and intervention.
A further challenge was related to the development of the intervention and digital features to support study execution and the feasibility study setup taking place concurrently, and the subsequent need for flexibility. For example, during the intervention development phase, multiple changes and continued improvements were made to the intervention content, language, and overall design (eg, font, color palette, logo design, professional illustrations, and layout). However, any change to language and design had a subsequent impact on other study components, for example, participant information sheets. It was perceived that there were many interrelated moving parts and any detail changed in the intervention or feasibility study resulted in a snow-ball effect on multiple other intervention and study components. A further example relates to study documentation. While refining the intervention, specific study terminology was still under development, for example, intervention name, e-therapists being referred to in the study as parent guides, the structure of the intervention, module, and chapter titles. During the feasibility study setup phase, study documentation was under development (eg, case report forms, data management, SMS text message and email reminder protocols and reminder content, standard operating procedures, and the study handbook). However, each terminology change resulted in revisions to all study documentation. Another example related to the study information video, which had to be rerecorded after numerous changes were made to the study information sheets based on changes to terminology used in the intervention. Research team members raised that these challenges seemed related to setting up a feasibility study when the intervention had not yet been finalized. The research team needed to be flexible and adaptable during this process and keep track of any detail and subsequent impact on other elements of the intervention and feasibility study.
Planning and Scheduling
Another challenge is related to the overall timeframe and associated planning and scheduling of tasks. For example, given the aforementioned challenges with delays in intervention development, meeting minutes documented difficulties scheduling specific events, such as training e-therapists, or presenting the intervention to PRPs for feedback. On several occasions, e-therapist training was rescheduled due to delays in intervention development; for example, finalizing the written intervention material and some technical features, such as videoconferencing, taking longer than initially anticipated to develop and test. Furthermore, time for task completion was often underestimated, which was discussed at times as being related to expertise. This could be observed by how several research team personnel had no prior experience of working within a similar research environment, resulting in some tasks taking longer than anticipated. Planning and scheduling were also described as generally difficult given the interrelatedness of intervention development and feasibility study setup tasks. For example, any changes made to the intervention affected the entire planning chain.
A further complexity regarding planning and scheduling related to the different project management tools used by the research team versus the Portal team. For example, the research team used a Gantt chart, mainly focused on research-associated tasks, alongside Microsoft To Do, to enable task breakdown and allocation of tasks between research team members. However, the Portal team used Atlassian Jira Software Server Version 8.5.1, a software development tool used by agile teams to plan, track, and release software features. At times, this was experienced as a challenge as while the research team tracked overall technical feature development in the Gantt chart, this was at times out of sync with more detailed software development planning in Jira Software Server. During the study setup phase, the developer team began to report updates to the research assistant responsible for the Gantt chart each week, which improved planning. However, research team members discussed how a more integrated solution may have facilitated scheduling and planning, especially because of the detail and complexity involved in the development of new digital features for both the intervention and support study execution.
A challenge frequently mentioned in meeting minutes pertained to the availability of technical resources, such as the Portal team’s time and the existing functionality of the Portal. The existing Portal did not have all the digital features required to deliver the internet-administered intervention or to support the execution of the feasibility study. Meeting minutes documented numerous new or adjusted intervention features, for example, (1) color palette change, (2) e-therapist notifications via email and SMS text message when participants send internal messages or submit chapters or homework exercises, (3) font selection, (4) intervention display to allow e-therapists to tailor the intervention, (5) carrousel feature to enhance library navigation, (6) printable PDF documents of homework exercise, (7) tab-based intervention view, and (8) videoconferencing. Meeting minutes also document new digital features required to support study execution, for example, (1) character limitation removed for SMS text messages sent via the Portal; (2) customized study registration process, including the use of recruitment ID to identify the source of recruitment; (3) individual preferences for reminders (eg, email, SMS text message, post, or telephone); (4) new home page to facilitate study sign-up and log-in for existing participants; (5) newsletter scheduling based on parents’ progress in the study; (6) opt-out procedure for participants declining study participation; (7) personalization of reminders (eg, use of first name); (8) reporting features on intervention use, newsletters, reminders, and suicide alerts; and (9) study-specific technical help-texts throughout the Portal.
Some new digital feature requirements were known from study conception (eg, included in the grant application and study protocol), for example, the newsletter, individualized reminders, and opt-out procedure. However, other digital requirements, especially in relation to the intervention, were not planned and arose during the intervention development phase. During the discussion groups, it was raised that as it was very difficult to know all Portal requirements in advance, there was a need for good communication between the research team and Portal team. Some research team members discussed how their lack of prior knowledge of the Portal, especially in relation to how interventions were delivered, resulted in underestimations or differing expectations concerning the amount of technical and aesthetic changes required. Furthermore, the research team did not always have a clear understanding of how much work might be involved in developing a new technical feature or changing an existing feature. Meeting minutes also listed several occasions when new digital features, or requested changes to existing digital features, took longer than anticipated to develop and test. One reason for the difficulties in time estimations is related to the existing software architecture of the Portal. As such, the development of new digital features, or making changes to existing digital features, could have unintended or unanticipated consequences on other existing features on the Portal, and subsequently impact other studies running on the Portal.
This paper describes the challenges experienced during the development phase of the internet-administered intervention and digital features to support the execution of the study procedures and setup of the ENGAGE feasibility study. To summarize the main findings, a document analysis of meeting minutes adopting a thematic analysis approach and subsequent research team discussions resulted in six main themes as follows: (1) decision-making and communication, (2) expertise, (3) external constraints, (4) flexibility, (5) planning and scheduling, and (6) technical constraints.
Comparison With Prior Work
To provide an overview of prior work exploring challenges experienced in traditional and digital health care research and software development projects, we present each of the main themes and descriptors of challenges identified alongside similar research findings from others (). We included 10 publications [ - ] in , which have not yet been mentioned.
|Main themes||Descriptors of challenges identified in main themes||Similar research findings from others|
|Decision-making and communication|
|Planning and scheduling|
One interesting challenge was identified in relation to the group decision-making process affecting the study timeframe. Research concerning cultural expectations and decision-making indicates that, within the Swedish culture, there are high expectations for shared authority and decision-making between managers and personnel . Generally, the workplace is less hierarchical with decisions and responsibilities shared and decisions made in larger groups [ ]. However, research suggests that more unilateral or directive decision-making may be required when there are significant time pressures and critical deadlines and with more novice team members [ ]. While adopting group decision-making processes may help to maintain good relationships and is more in line with cultural expectations, this was at tension with the need for more unilateral and directive decision-making, given the time critical nature of the research. Indeed, one reason for failing clinical trials may pertain to a lack of structured business-like trial management [ ].
A related challenge concerned the difficulties in recruiting experienced study personnel. Difficulties with recruitment and inadequate training of study personnel have been identified in the literature as a key inefficiency in successful trial delivery . Despite multiple recruitment attempts, it was not possible to recruit a postdoctoral researcher with the required competencies to work on the study. However, competent trial management by a dedicated trial manager responsible for day-to-day operations is essential for efficient trial conduct. Indeed, once funding is awarded, the trial manager, rather than the principal investigator and coinvestigators, has been suggested to be the most important research team member to successfully deliver a clinical trial [ ]. Furthermore, it is suggested that some trials fail not due to the study design but rather problems with trial management [ , ]. In the context of the ENGAGE feasibility study, no research team member working on the study on a full-time basis had experience of similar research environments and advanced training in research methodology. Those in senior roles with advanced training and expertise in research methodology (eg, the principal investigator and coinvestigator) were key to organizing, planning, and decision-making; however, their time was spread across multiple studies and competing responsibilities. This light project management approach has been demonstrated to lead to team members’ expectations being unfulfilled and commitment of the team may decrease [ ].
In countries such as the United Kingdom, there has been an increased emphasis on efficient clinical trials to reduce research waste, for example, by the establishment of the UK Trial Managers’ Network for trial managers on academic-led noncommercial trials. To the best of our knowledge, there is no specific trial management professional career structure within academic settings in Sweden, which may account for some of the difficulties recruiting a postdoctoral researcher with the required competencies. An added complexity in the Swedish context is the employment projection law (LAS § 5a 1982:80), which stipulates that personnel can be employed on a fixed-term contract for a maximum of 2 years and must be offered a permanent position thereafter to remain in employment. However, given the time-limited nature of health care research and uncertainty concerning continued funding, this often results in research personnel only being employed for a maximum 2-year fixed-term contract. Subsequently, it is difficult to maintain experienced research staff for the duration of the trial funding periods. However, even in countries such as the United Kingdom, where trial management is more established within academic settings, barriers are experienced, such as a lack of clarity around career structures, progression, and professional status, lack of training opportunities, lack of professional accreditation, and staff retention due to lack of funding and instability of short-term contracts [, , ].
A further key challenge related to both the development phase of the internet-administered intervention and digital features to support the execution of study procedures and setup of the feasibility study, occurring concurrently. Complex health care intervention development and refinement involves a number of iterative and interacting stages . The development of interventions using internet-based technologies is complex, and specific challenges have been identified in relation to their development (eg, iterative development life cycles, relationship between academics and developers, and characterization of intervention components and essential features) [ , ]. Furthermore, feasibility studies are complex and can involve a number of iterative phases and the testing of multiple procedural, methodological, and clinical uncertainties [ ]. In the present context, a number of challenges were related to the interrelatedness of the intervention and subsequent feasibility study, for example, further development, refinement, and adaptation of the intervention resulted in multiple changes to associated documentation for the feasibility study. The development of high-quality study documentation is a difficult and lengthy process [ ], and the focus on intervention development meant allocating resources away from the preparation of study documentation. As such, working on intervention development concurrent with the feasibility study setup may have added to the complexity of already complex research processes.
In addition, while a significant phased approach to intervention development had already taken place [, , , ], a structured approach was not adopted regarding the technical development of the intervention. Given the complexity of the development of internet-administered interventions [ , ], it may have been beneficial to adopt a specific development model designed to inform the development of health care interventions using internet-based technologies, such as the Behavioral Intervention Technology model [ ]. Adopting a more structured approach to technical development may have facilitated the communication of technical requirements to the Portal team. Indeed, challenges relating to communication between the research and Portal team may have been experienced, given that requirements were requested from those unfamiliar with software programming, and software developers are unfamiliar with research methodologies and LICBT interventions [ ]. The Portal team adopts agile software development, therefore working with software requirements in short iterations [ ] with a need for frequent, quick, and short-term decisions [ ]. However, this approach can cause challenges when there is a need for collaborative decision-making with multiple stakeholders with varying backgrounds, expertise, and goals [ ]. Barriers to efficient communication and development of shared knowledge may be related to a lack of understanding of each other’s field (eg, different educational backgrounds, technical knowledge, and experience), difficulties with clients describing functional requirements, and difficulties with software developers communicating time expectations with clients in a direct manner [ ]. A lack of shared understanding between software development teams and end-clients can result in unrealistic planning and frequent changes in planning [ ]. In addition, transforming study procedures normally conducted in-person to digital form is a complex procedure requiring significant planning, time, and expertise [ , ]. However, involving a member of the Portal team in the weekly research team meetings significantly improved the development of shared knowledge and understanding and helped to overcome some of the aforementioned challenges. Indeed, more active engagement between software developers and the end client has been posited as a helpful strategy to facilitate communication [ ].
Another noteworthy challenge is related to the constraints of the Portal. Health care interventions delivered by digital technologies have evolved rapidly over the past two decades . However, this also means that technology becomes quickly outdated [ ], given the technological advancements and changing end-user expectations. However, rapid technical evolution is not easily compatible with traditional research approaches such as randomized controlled trials (RCTs) [ ], and translating traditional clinical trial procedures to digital form may require the complete re-engineering of trial design and processes [ ]. The Portal was developed to support traditional health care research, for example, RCTs and observational studies, and multiple RCTs of internet-administered cognitive behavioral therapy interventions have been conducted on the Portal [ - ]. The Portal was originally designed to deliver two internet-administered cognitive behavioral therapy interventions [ , ]; however, over time, the Portal has been developed to support the needs of different studies and has experienced a continuous flow of technical requirements from researchers since its conception in 2010 [ , ]. However, the time taken to design and conduct traditional health care research is not in line with fast-paced technical advances [ ]. Therefore, when using digital technologies for intervention delivery and the execution of study procedures, careful planning is required as technology may change [ , ]. Multiple requests for new digital features and changes to existing features were required for both the intervention and the execution of study procedures; however, challenges were experienced relating to the legacy software architecture of the Portal. For example, new feature requests or changes to existing features can result in unanticipated negative impacts on existing features used by ongoing studies. This highlights a key challenge for research environments using internet-based technologies, especially in a context where new digital features and complex configurations are added over time [ ]. Appropriate technical infrastructure (eg, having the appropriate hardware, software, and technical expertise) has been highlighted as a significant challenge in conducting successful digital health care research [ ].
Meeting minutes were written independently to the objective of the analysis, and therefore may not provide sufficient detail to reveal all the challenges experienced by the research team. As is common in document analysis, documents are often uneven in length, providing more detailed information about some topics than others . As such, there is a potential that some challenges experienced were missed in the analysis as less focus was placed on them when writing meeting minutes; however, we attempted to overcome this limitation by member-checking through two group discussions. An additional limitation relates to selectivity bias [ ], and it is difficult to separate the analysis from the context of the research team, for example, biases and opinions held by research team members. Attempts to limit selectivity bias were made by the analysis being primarily conducted by a new member of the research team who had not been present in any of the meetings wherein minutes were analyzed.
The development and feasibility testing of health care interventions using digital technologies is time- and resource-intensive. Recommendations for improving efficiency include (1) the development of networks to share good practice and training opportunities for trial staff, especially in the area of complex digital health care interventions; (2) the employment of advanced research methodology–educated, senior dedicated trial personnel who can be responsible for the day-to-day operations; (3) the completion of the intervention development phase (including technical requirements) before the feasibility study setup; and (4) the integration of members of the software development team into the research team to improve communication and develop shared knowledge and understanding. We hope that our experiences may be useful for others who are planning to conduct future research within the development and feasibility phases of the MRC complex intervention framework, especially for internet-administered interventions and research using digital features to support the execution of study procedures. Publishing challenges experienced during intervention development  and trial setup and conduct [ ] may help to reduce future research waste, improve the quality of digital health care research, and add to the emerging literature concerning challenges experienced by integrating digital technologies into health care research [ ].
Despite experiencing a number of challenges, the ENGAGE feasibility study commenced recruitment on July 3, 2020, and recruitment targets were successfully met by October 14, 2020, well within the projected 6-month recruitment period . Furthermore, preliminary posttreatment follow-up data indicated that retention targets were successfully met. While delays to study start were experienced, taking a careful and detail-orientated approach to intervention development and feasibility study setup may have helped facilitate meeting recruitment and retention targets and will hopefully enhance efficiencies across subsequent phases of our planned research, for example, a definitive (evaluation) trial.
This work was supported by the Swedish Research Council (K2015-99X-20836-08-4 and 2018-02578), the Swedish Cancer Society (grants 150673 and 180589), and the Swedish Childhood Cancer Foundation (grant PR2017-0005). The funders had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript. The funders do not bear any responsibility for the analyses or interpretations presented here. The authors thank the 4 parent research partners (Helena Börjesson, Martin Hedqvist, Anna Mautner, and Erik Olsson) and the Portal team members outside of the ENGAGE research team (Fabian Holmberg and Ian Horne). The authors are also very grateful to ETC Communications, Stockholm, for providing the graphic design and to Rebecka Porse Schalin for designing all illustrations in the EJDeR intervention.
Author contributions have been written in accordance with the CRediT author statement. JW contributed to the development and design of the methodology, formal analysis, writing the original draft, data visualization, supervision, and project administration. MK contributed to formal analysis, data curation, reviewing and editing the original draft, and data visualization. JH contributed to reviewing and editing the original draft, and data visualization. YHS was responsible for programming and software development and reviewing and editing the original draft. KN and HG reviewed and edited the final draft. LVE was responsible for conceptualization, methodology, provision of resources, supervision, project administration, funding acquisition, and reviewed and edited the final draft. All authors approved the final version of the manuscript before submission.
Conflicts of Interest
- Treweek S, Altman DG, Bower P, Campbell M, Chalmers I, Cotton S, et al. Making randomised trials more efficient: report of the first meeting to discuss the Trial Forge platform. Trials 2015 Jun 05;16:261 [FREE Full text] [CrossRef] [Medline]
- Moher D, Glasziou P, Chalmers I, Nasser M, Bossuyt PM, Korevaar DA, et al. Increasing value and reducing waste in biomedical research: who's listening? Lancet 2016 Apr 09;387(10027):1573-1586. [CrossRef] [Medline]
- Chalmers I, Glasziou P. Avoidable waste in the production and reporting of research evidence. Lancet 2009 Jul 04;374(9683):86-89. [CrossRef] [Medline]
- Chalmers I, Bracken MB, Djulbegovic B, Garattini S, Grant J, Gülmezoglu AM, et al. How to increase value and reduce waste when research priorities are set. Lancet 2014 Jan 11;383(9912):156-165. [CrossRef] [Medline]
- Ioannidis JP, Greenland S, Hlatky MA, Khoury MJ, Macleod MR, Moher D, et al. Increasing value and reducing waste in research design, conduct, and analysis. Lancet 2014 Jan 11;383(9912):166-175 [FREE Full text] [CrossRef] [Medline]
- Yordanov Y, Dechartres A, Atal I, Tran V, Boutron I, Crequit P, et al. Avoidable waste of research related to outcome planning and reporting in clinical trials. BMC Med 2018 Jun 11;16(1):87 [FREE Full text] [CrossRef] [Medline]
- Glasziou P, Altman DG, Bossuyt P, Boutron I, Clarke M, Julious S, et al. Reducing waste from incomplete or unusable reports of biomedical research. Lancet 2014 Jan 18;383(9913):267-276. [CrossRef] [Medline]
- Glasziou P, Chalmers I. Research waste is still a scandal—an essay by Paul Glasziou and Iain Chalmers. BMJ Clin Res 2018 Nov 12;363:k4645. [CrossRef]
- Duley L, Gillman A, Duggan M, Belson S, Knox J, McDonald A, et al. What are the main inefficiencies in trial conduct: a survey of UKCRC registered clinical trials units in the UK. Trials 2018 Jan 08;19(1):15 [FREE Full text] [CrossRef] [Medline]
- Arundel C, Gellatly J. Learning from OCTET - exploring the acceptability of clinical trials management methods. Trials 2018 Jul 13;19(1):378 [FREE Full text] [CrossRef] [Medline]
- Beaumont D, Arribas M, Frimley L, Balogun E, Roberts I, Shakur-Still H. Trial management: we need a cadre of high-class triallists to deliver the answers that patients need. Trials 2019 Jun 13;20(1):354 [FREE Full text] [CrossRef] [Medline]
- Farrell B, Kenyon S, Shakur H. Managing clinical trials. Trials 2010 Jul 13;11(1):78 [FREE Full text] [CrossRef] [Medline]
- Goodarzynejad H, Babamahmoodi A. Project management of randomized clinical trials: a narrative review. Iran Red Crescent Med J 2015 Aug 24;17(8):e11602 [FREE Full text] [CrossRef] [Medline]
- McCaskell DS, Molloy AJ, Childerhose L, Costigan FA, Reid JC, McCaughan M, et al. Project management lessons learned from the multicentre CYCLE pilot randomized controlled trial. Trials 2019 Aug 28;20(1):532 [FREE Full text] [CrossRef] [Medline]
- Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M, Medical Research Council Guidance. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ 2008 Sep 29;337:a1655 [FREE Full text] [CrossRef] [Medline]
- Eldridge SM, Chan CL, Campbell MJ, Bond CM, Hopewell S, Thabane L, PAFS consensus group. CONSORT 2010 statement: extension to randomised pilot and feasibility trials. BMJ 2016 Oct 24;355:i5239 [FREE Full text] [CrossRef] [Medline]
- Lancaster GA, Thabane L. Guidelines for reporting non-randomised pilot and feasibility studies. Pilot Feasibility Stud 2019 Oct 6;5:114 [FREE Full text] [CrossRef] [Medline]
- Bleijenberg N, de Man-van Ginkel JM, Trappenburg JC, Ettema RG, Sino CG, Heim N, et al. Increasing value and reducing waste by optimizing the development of complex interventions: enriching the development phase of the Medical Research Council (MRC) Framework. Int J Nurs Stud 2018 Mar;79:86-93 [FREE Full text] [CrossRef] [Medline]
- O'Cathain A, Croot L, Duncan E, Rousseau N, Sworn K, Turner KM, et al. Guidance on how to develop complex interventions to improve health and healthcare. BMJ Open 2019 Aug 15;9(8):e029954 [FREE Full text] [CrossRef] [Medline]
- Thabane L, Lancaster G. A guide to the reporting of protocols of pilot and feasibility trials. Pilot Feasibility Stud 2019 Feb 28;5:37 [FREE Full text] [CrossRef] [Medline]
- Woodford J, Wikman A, Cernvall M, Ljungman G, Romppala A, Grönqvist H, et al. Study protocol for a feasibility study of an internet-administered, guided, CBT-based, self-help intervention (ENGAGE) for parents of children previously treated for cancer. BMJ Open 2018 Jun 14;8(6):e023708 [FREE Full text] [CrossRef] [Medline]
- Sjöström J, von Essen L, Grönqvist H. The origin and impact of ideals in eHealth research: experiences from the U-CARE research environment. JMIR Res Protoc 2014 May 23;3(2):e28 [FREE Full text] [CrossRef] [Medline]
- Grönqvist H, Olsson EM, Johansson B, Held C, Sjöström J, Lindahl Norberg A, et al. Fifteen challenges in establishing a multidisciplinary research program on eHealth research in a university setting: a case study. J Med Internet Res 2017 May 23;19(5):e173 [FREE Full text] [CrossRef] [Medline]
- Eagleson R, Altamirano-Diaz L, McInnis A, Welisch E, De Jesus S, Prapavessis H, et al. Implementation of clinical research trials using web-based and mobile devices: challenges and solutions. BMC Med Res Methodol 2017 Mar 17;17(1):43 [FREE Full text] [CrossRef] [Medline]
- Rosa C, Marsch LA, Winstanley EL, Brunner M, Campbell AN. Using digital technologies in clinical trials: current and future applications. Contemp Clin Trials 2021 Jan;100:106219. [CrossRef] [Medline]
- Wilhelm S, Weingarden H, Ladis I, Braddick V, Shin J, Jacobson NC. Cognitive-behavioral therapy in the digital age: presidential address. Behav Ther 2020 Jan;51(1):1-14 [FREE Full text] [CrossRef] [Medline]
- Strudwick G, Sockalingam S, Kassam I, Sequeira L, Bonato S, Youssef A, et al. Digital interventions to support population mental health in Canada during the COVID-19 pandemic: rapid review. JMIR Ment Health 2021 Mar 02;8(3):e26550 [FREE Full text] [CrossRef] [Medline]
- Rauschenberg C, Schick A, Hirjak D, Seidler A, Paetzold I, Apfelbacher C, et al. Evidence synthesis of digital interventions to mitigate the negative impact of the COVID-19 pandemic on public mental health: rapid meta-review. J Med Internet Res 2021 Mar 10;23(3):e23365 [FREE Full text] [CrossRef] [Medline]
- Torous J, Jän Myrick K, Rauseo-Ricupero N, Firth J. Digital mental health and COVID-19: using technology today to accelerate the curve on access and quality tomorrow. JMIR Ment Health 2020 Mar 26;7(3):e18848 [FREE Full text] [CrossRef] [Medline]
- Andersson G, Carlbring P, Rozental A. Response and remission rates in internet-based cognitive behavior therapy: an individual patient data meta-analysis. Front Psychiatry 2019 Oct 25;10:749 [FREE Full text] [CrossRef] [Medline]
- Hwang WJ, Ha JS, Kim MJ. Research trends on mobile mental health application for general population: a scoping review. Int J Environ Res Public Health 2021 Mar 02;18(5):2459 [FREE Full text] [CrossRef] [Medline]
- Rosa C, Campbell AN, Miele GM, Brunner M, Winstanley EL. Using e-technologies in clinical trials. Contemp Clin Trials 2015 Nov;45(Pt A):41-54 [FREE Full text] [CrossRef] [Medline]
- Baker TB, Gustafson DH, Shah D. How can research keep up with eHealth? Ten strategies for increasing the timeliness and usefulness of eHealth research. J Med Internet Res 2014 Feb 19;16(2):e36 [FREE Full text] [CrossRef] [Medline]
- Noonan D, Simmons LA. Navigating nonessential research trials during COVID-19: the push we needed for using digital technology to increase access for rural participants? J Rural Health 2021 Jan;37(1):185-187 [FREE Full text] [CrossRef] [Medline]
- National Academies of Sciences, Engineering, and Medicine, Health and Medicine Division, Board on Health Sciences Policy, Forum on Drug Discovery, Development, and Translation. Virtual Clinical Trials: Challenges and Opportunities: Proceedings of a Workshop. Washington, DC: National Academies Press; 2019.
- Steinhubl SR, Wolff-Hughes DL, Nilsen W, Iturriaga E, Califf RM. Digital clinical trials: creating a vision for the future. NPJ Digit Med 2019 Dec 12;2:126 [FREE Full text] [CrossRef] [Medline]
- Clinical Trials Transformation Initiative (CTTI) Program: Digital health trials. Clinical Trials Transformation Initiative. URL: https://www.ctti-clinicaltrials.org/programs/mobile-clinical-trials [accessed 2021-04-13]
- Geoghegan C, Nido V, Bemden AB, Hallinan Z, Jordan L, Kehoe LS, et al. Learning from patient and site perspectives to develop better digital health trials: recommendations from the Clinical Trials Transformation Initiative. Contemp Clin Trials Commun 2020 Aug 13;19:100636 [FREE Full text] [CrossRef] [Medline]
- Steliarova-Foucher E, Colombet M, Ries LA, Moreno F, Dolya A, Bray F, IICC-3 contributors. International incidence of childhood cancer, 2001-10: a population-based registry study. Lancet Oncol 2017 Jun;18(6):719-731 [FREE Full text] [CrossRef] [Medline]
- International incidence of childhood cancer 3. International Agency for Research on Cancer. URL: http://iicc.iarc.fr/results/ [accessed 2020-08-18]
- Carlsson T, Kukkola L, Ljungman L, Hovén E, von Essen L. Psychological distress in parents of children treated for cancer: an explorative study. PLoS One 2019 Jun 21;14(6):e0218860 [FREE Full text] [CrossRef] [Medline]
- Ljungman L, Cernvall M, Grönqvist H, Ljótsson B, Ljungman G, von Essen L. Long-term positive and negative psychological late effects for parents of childhood cancer survivors: a systematic review. PLoS One 2014 Jul 24;9(7):e103340 [FREE Full text] [CrossRef] [Medline]
- Wakefield CE, McLoone JK, Butow P, Lenthen K, Cohn RJ. Parental adjustment to the completion of their child's cancer treatment. Pediatr Blood Cancer 2011 Apr;56(4):524-531. [CrossRef] [Medline]
- Wikman A, Hovén E, Cernvall M, Ljungman G, Ljungman L, von Essen L. Parents of children diagnosed with cancer: work situation and sick leave, a five-year post end-of-treatment or a child's death follow-up study. Acta Oncol 2016;55(9-10):1152-1157. [CrossRef] [Medline]
- Hovén E, von Essen L, Norberg AL. A longitudinal assessment of work situation, sick leave, and household income of mothers and fathers of children with cancer in Sweden. Acta Oncol 2013 Aug;52(6):1076-1085. [CrossRef] [Medline]
- Lindahl Norberg A, Montgomery SM, Bottai M, Heyman M, Hovén EI. Short-term and long-term effects of childhood cancer on income from employment and employment status: a national cohort study in Sweden. Cancer 2017 Apr 01;123(7):1238-1248 [FREE Full text] [CrossRef] [Medline]
- Öhman M, Woodford J, von Essen L. Socioeconomic consequences of parenting a child with cancer for fathers and mothers in Sweden: a population-based difference-in-difference study. Int J Cancer 2020 Dec 15;148(10):2535-2541 [FREE Full text] [CrossRef] [Medline]
- van Warmerdam J, Zabih V, Kurdyak P, Sutradhar R, Nathan PC, Gupta S. Prevalence of anxiety, depression, and posttraumatic stress disorder in parents of children with cancer: a meta-analysis. Pediatr Blood Cancer 2019 Jun;66(6):e27677. [CrossRef] [Medline]
- Ljungman L, Boger M, Ander M, Ljótsson B, Cernvall M, von Essen L, et al. Impressions that last: particularly negative and positive experiences reported by parents five years after the end of a child's successful cancer treatment or death. PLoS One 2016 Jun 7;11(6):e0157076 [FREE Full text] [CrossRef] [Medline]
- Christen S, Mader L, Baenziger J, Roser K, Schindera C, Tinner EM, et al. "I wish someone had once asked me how I'm doing": disadvantages and support needs faced by parents of long-term childhood cancer survivors. Pediatr Blood Cancer 2019 Aug;66(8):e27767. [CrossRef] [Medline]
- Kukkola L, Hovén E, Cernvall M, von Essen L, Grönqvist H. Perceptions of support among Swedish parents of children after end of successful cancer treatment: a prospective, longitudinal study. Acta Oncol 2017 Dec;56(12):1705-1711. [CrossRef] [Medline]
- Wakefield CE, McLoone J, Butow P, Lenthen K, Cohn RJ. Support after the completion of cancer treatment: Perspectives of Australian adolescents and their families. Eur J Cancer Care (Engl) 2013 Jul;22(4):530-539. [CrossRef] [Medline]
- Patel V, Saxena S, Lund C, Thornicroft G, Baingana F, Bolton P, et al. The Lancet Commission on global mental health and sustainable development. Lancet 2018 Oct 27;392(10157):1553-1598. [CrossRef] [Medline]
- Woodford J, Farrand P, Hagström J, Hedenmalm L, von Essen L. Internet-administered cognitive behavioral therapy for common mental health difficulties in parents of children treated for cancer: intervention development and description study. JMIR Form Res 2021 Jul 22;5(7):e22709 [FREE Full text] [CrossRef] [Medline]
- Ljungman L, Cernvall M, Ghaderi A, Ljungman G, von Essen L, Ljótsson B. An open trial of individualized face-to-face cognitive behavior therapy for psychological distress in parents of children after end of treatment for childhood cancer including a cognitive behavioral conceptualization. PeerJ 2018 Apr 11;6:e4570 [FREE Full text] [CrossRef] [Medline]
- Wikman A, Mattsson E, von Essen L, Hovén E. Prevalence and predictors of symptoms of anxiety and depression, and comorbid symptoms of distress in parents of childhood cancer survivors and bereaved parents five years after end of treatment or a child's death. Acta Oncol 2018 Jul;57(7):950-957. [CrossRef] [Medline]
- Woodford J, Wikman A, Einhorn K, Cernvall M, Grönqvist H, Romppala A, et al. Attitudes and preferences toward a hypothetical trial of an internet-administered psychological intervention for parents of children treated for cancer: web-based survey. JMIR Ment Health 2018 Dec 18;5(4):e10085 [FREE Full text] [CrossRef] [Medline]
- Woodford J, Norbäck K, Hagström J, Grönqvist H, Parker A, Arundel C, et al. Study within a trial (SWAT) protocol. Investigating the effect of personalised versus non-personalised study invitations on recruitment: an embedded randomised controlled recruitment trial. Contemp Clin Trials Commun 2020 Apr 24;18:100572 [FREE Full text] [CrossRef] [Medline]
- O'Brien BC, Harris IB, Beckman TJ, Reed DA, Cook DA. Standards for reporting qualitative research: a synthesis of recommendations. Acad Med 2014 Sep;89(9):1245-1251 [FREE Full text] [CrossRef] [Medline]
- Bowen GA. Document analysis as a qualitative research method. Qualitative Research Journal 2009 Aug 03;9(2):27-40. [CrossRef]
- Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol 2006 Jan;3(2):77-101. [CrossRef]
- Boyatzis R. Transforming Qualitative Information: Thematic Analysis and Code Development. Thousand Oaks, CA: Sage Publications; 1998.
- Lietz CA, Langer CL, Furman R. Establishing trustworthiness in qualitative research in social work: implications from a study regarding spirituality. Qual Soc Work 2006 Dec 1;5(4):441-458. [CrossRef]
- Auer-Rizzi W, Berry M. Business vs. Cultural frames of reference in group decision making: interactions among Austrian, Finnish, and Swedish business students. J Bus Commun 2000 Jul 01;37(3):264-288. [CrossRef]
- Lämsä T. Leadership styles and decision-making in Finnish and Swedish organizations. Int J Comp Manag 2010 Jan;11(1):139-149 [FREE Full text]
- Moe NB, Aurum A, Dybå T. Challenges of shared decision-making: a multiple case study of agile software development. Inf Softw Technol 2012 Aug;54(8):853-865. [CrossRef]
- Ellis E, Riegel B, Hamon M, Carlson B, Jimenez S, Parkington S. The challenges of conducting clinical research: one research team's experiences. Clin Nurse Spec 2001 Nov;15(6):286-92; quiz 293. [CrossRef] [Medline]
- Ghobadi S, Mathiassen L. Perceived barriers to effective knowledge sharing in agile software teams. Info Systems J 2014 Dec 18;26(2):95-125. [CrossRef]
- Bonsu JM, Frasso R, Curry AE. Lessons from the field: the conduct of randomized controlled trials in Botswana. Trials 2017 Oct 27;18(1):503 [FREE Full text] [CrossRef] [Medline]
- Petrova M, Barclay S. Response to Correspondence from Kolstoe and colleagues concerning our paper entitled, Research approvals iceberg: how a 'low-key' study in England needed 89 professionals to approve it and how we can do better. BMC Med Ethics 2019 Dec 24;20(1):101 [FREE Full text] [CrossRef] [Medline]
- Martin L, Hutchens M, Hawkins C, Radnov A. How much do clinical trials cost? Nat Rev Drug Discov 2017 Jun;16(6):381-382. [CrossRef] [Medline]
- Kakkar AK, Sarma P, Medhi B. mHealth technologies in clinical trials: opportunities and challenges. Indian J Pharmacol 2018;50(3):105-107 [FREE Full text] [CrossRef] [Medline]
- Kotz D, Fu K, Gunter C, Rubin A. Security for mobile and cloud frontiers in healthcare. Commun ACM 2015 Jul 23;58(8):21-23. [CrossRef]
- Sims HP, Faraj S, Yun S. When should a leader be directive or empowering? How to develop your own situational theory of leadership. Bus Horiz 2009;52(2):149-158. [CrossRef]
- Treweek S, Littleford R. Trial management- building the evidence base for decision-making. Trials 2018 Jan 08;19(1):11 [FREE Full text] [CrossRef] [Medline]
- Campbell MK, Snowdon C, Francis D, Elbourne D, McDonald AM, Knight R, STEPS group. Recruitment to randomised trials: strategies for trial enrollment and participation study. The STEPS study. Health Technol Assess 2007 Nov;11(48):iii, ix-105 [FREE Full text] [CrossRef] [Medline]
- Fogel DB. Factors associated with clinical trials that fail and opportunities for improving the likelihood of success: a review. Contemp Clin Trials Commun 2018 Aug 7;11:156-164 [FREE Full text] [CrossRef] [Medline]
- Grant KP, Graham TS, Heberling ME. The project manager and project team involvement: implications for project leadership. J Leadersh Organ Stud 2001 Jan 1;7(4):32-42. [CrossRef]
- Mitchell E, Goodman K, Hartley S, Hickey H, McDonald AM, Meadows HM, UK Trial Managers’ Network Executive Group. Where do we go from here? - Opportunities and barriers to the career development of trial managers: a survey of UK-based trial management professionals. Trials 2020 May 06;21(1):384 [FREE Full text] [CrossRef] [Medline]
- Yardley L, Choudhury T, Patrick K, Michie S. Current issues and future directions for research into digital behavior change interventions. Am J Prev Med 2016 Nov;51(5):814-815. [CrossRef] [Medline]
- Michie S, Yardley L, West R, Patrick K, Greaves F. Developing and evaluating digital interventions to promote behavior change in health and health care: recommendations resulting from an international workshop. J Med Internet Res 2017 Jun 29;19(6):e232 [FREE Full text] [CrossRef] [Medline]
- Mohr DC, Schueller SM, Montague E, Burns MN, Rashidi P. The behavioral intervention technology model: an integrated conceptual and technological framework for eHealth and mHealth interventions. J Med Internet Res 2014 Jun 5;16(6):e146 [FREE Full text] [CrossRef] [Medline]
- Dybå T, Dingsøyr T. Empirical studies of agile software development: a systematic review. Inf Softw Technol 2008 Aug;50(9-10):833-859. [CrossRef]
- Cockburn A, Highsmith J. Agile software development, the people factor. Computer 2001 Nov;34(11):131-133. [CrossRef]
- Guo C, Ashrafian H, Ghafur S, Fontana G, Gardner C, Prime M. Challenges for the evaluation of digital health solutions-A call for innovative evidence generation approaches. NPJ Digit Med 2020 Aug 27;3(1):110 [FREE Full text] [CrossRef] [Medline]
- Patrick K, Hekler EB, Estrin D, Mohr DC, Riper H, Crane D, et al. The pace of technologic change: implications for digital health behavior intervention research. Am J Prev Med 2016 Nov;51(5):816-824. [CrossRef] [Medline]
- Igelström H, Hauffman A, Alfonsson S, Sjöström J, Cajander A, Johansson B. User experiences of an internet-based stepped-care intervention for individuals with cancer and concurrent symptoms of anxiety or depression (the U-CARE AdultCan trial): qualitative study. J Med Internet Res 2020 May 19;22(5):e16604 [FREE Full text] [CrossRef] [Medline]
- Norlund F, Wallin E, Olsson EM, Wallert J, Burell G, von Essen L, et al. Internet-based cognitive behavioral therapy for symptoms of depression and anxiety among patients with a recent myocardial infarction: the U-CARE heart randomized controlled trial. J Med Internet Res 2018 Mar 08;20(3):e88 [FREE Full text] [CrossRef] [Medline]
- Rondung E, Ternström E, Hildingsson I, Haines HM, Sundin O, Ekdahl J, et al. Comparing internet-based cognitive behavioral therapy with standard care for women with fear of birth: randomized controlled trial. JMIR Ment Health 2018 Aug 10;5(3):e10420 [FREE Full text] [CrossRef] [Medline]
- Sjömark J, Parling T, Jonsson M, Larsson M, Skoog Svanberg A. A longitudinal, multi-centre, superiority, randomized controlled trial of internet-based cognitive behavioural therapy (iCBT) versus treatment-as-usual (TAU) for negative experiences and posttraumatic stress following childbirth: the JUNO study protocol. BMC Pregnancy Childbirth 2018 Oct 01;18(1):387 [FREE Full text] [CrossRef] [Medline]
- Riley WT, Glasgow RE, Etheredge L, Abernethy AP. Rapid, responsive, relevant (R3) research: a call for a rapid learning health research enterprise. Clin Transl Med 2013 May 10;2(1):10 [FREE Full text] [CrossRef] [Medline]
|LICBT: low-intensity cognitive behavioral therapy|
|MRC: Medical Research Council|
|PRP: parent research partner|
|RCT: randomized controlled trial|
Edited by G Eysenbach; submitted 04.12.20; peer-reviewed by MDG Pimentel; comments to author 19.02.21; revised version received 16.04.21; accepted 02.05.21; published 08.10.21Copyright
©Joanne Woodford, Mathilda Karlsson, Josefin Hagström, Ylva Hägg Sylvén, Kajsa Norbäck, Helena Grönqvist, Louise von Essen. Originally published in JMIR Formative Research (https://formative.jmir.org), 08.10.2021.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.