Published on in Vol 5, No 5 (2021): May

Preprints (earlier versions) of this paper are available at, first published .
Utilizing Lean Software Methods To Improve Acceptance of Global eHealth Initiatives: Results From the Implementation of the Basic Emergency Care App

Utilizing Lean Software Methods To Improve Acceptance of Global eHealth Initiatives: Results From the Implementation of the Basic Emergency Care App

Utilizing Lean Software Methods To Improve Acceptance of Global eHealth Initiatives: Results From the Implementation of the Basic Emergency Care App

Original Paper

1Department of Emergency Medicine, Stanford University School of Medicine, Palo Alto, CA, United States

2Department of Emergency Medicine, University of California, San Francisco, San Francisco, CA, United States

3Department of Emergency Medicine, University of Cape Town/Stellenbosch University, Cape Town, South Africa

4Department of Emergency Medicine, Muhimbili University of Health and Allied Sciences, Dar es Salaam, United Republic of Tanzania

Corresponding Author:

Christian Rose, MD

Department of Emergency Medicine

Stanford University School of Medicine

900 Welch Road

Suite 350

Palo Alto, CA, 94304

United States

Phone: 1 4159159585


Background: Health systems in low- and middle-income countries face considerable challenges in providing high-quality accessible care. eHealth has had mounting interest as a possible solution given the unprecedented growth in mobile phone and internet technologies in these locations; however, few apps or software programs have, as of yet, gone beyond the testing phase, most downloads are never opened, and consistent use is extremely rare. This is believed to be due to a failure to engage and meet local stakeholder needs and the high costs of software development.

Objective: World Health Organization Basic Emergency Care course participants requested a mobile point-of-care adjunct to the primary course material. Our team undertook the task of developing this solution through a community-based participatory model in an effort to meet trainees’ reported needs and avoid some of the abovementioned failings. We aimed to use the well-described Lean software development strategy—given our familiarity with its elements and its ubiquitous use in medicine, global health, and software development—to complete this task efficiently and with maximal stakeholder involvement.

Methods: From September 2016 through January 2017, the Basic Emergency Care app was designed and developed at the University of California San Francisco. When a prototype was complete, it was piloted in Cape Town, South Africa and Dar es Salaam, Tanzania—World Health Organization Basic Emergency Care partner sites. Feedback from this pilot shaped continuous amendments to the app before subsequent user testing and study of the effect of use of the app on trainee retention of Basic Emergency Care course material.

Results: Our user-centered mobile app was developed with an iterative participatory approach with its first version available within 6 months and with high acceptance—95% of Basic Emergency Care Course participants felt that it was useful. Our solution had minimal direct costs and resulted in a robust infrastructure for subsequent assessment and maintenance and allows for efficient feedback and expansion.

Conclusions: We believe that utilizing Lean software development strategies may help global health advocates and researchers build eHealth solutions with a process that is familiar and with buy-in across stakeholders that is responsive, rapid to deploy, and sustainable.

JMIR Form Res 2021;5(5):e14851



eHealth in Low- and Middle-Income Countries

Health systems in low- and middle-income countries continue to face considerable challenges in providing high-quality affordable accessible care [1]. Simultaneously, they have experienced an unprecedented increase in the number of users of mobile phone and internet technologies, as well as a decline in the price of devices and services [2]. Furthermore, most medical professionals report that smartphone and mobile technologies are useful for training and education [3]. As a result, many health program implementers and policy makers are exploring the extent to which eHealth, defined as “the use of information and communication technologies for health” [4], can help address the challenges faced by resource-constrained health markets in terms of the availability, quality, and financing of health care [5-10].

However, few of these initiatives succeed as hoped or progress past the pilot phase. Although eHealth has the potential to strengthen health systems worldwide, consistent use is extremely rare and the evidence base is immature [11]. Consequently, the opportunities to advance knowledge remain limited in scope [12]. Limitations in adoption are likely multifactorial, but they often fail to address the needs of diverse stakeholders [13]. eHealth apps often have unique end-users, and health care scientists and researchers are not often well-versed in how software engineers operate as traditional scientific measures may be too rigid to identify the nuanced differences between user subgroups [14]. Lastly, software and apps are expensive to develop, averaging between a cost of US $23,000-28,000 (with minimal features) to $270,000 (for robust apps)—a cost many community-based projects simply cannot afford [15-17].

These limitations in software development are not new, however. Early in the history of software product development and the expansion of the internet, there was concern over excessive amounts of money being spent to develop technologies that did not meet user needs. Years were sometimes spent building apps which would later be found to have been based on false assumptions, were uselessly developed for an outdated technology, or the demand for a solution disappeared by the end of the software production cycle. Development cycles were too long and rarely had early stakeholder buy-in or feedback. In response, developers looked for strategies to remain responsive to an ever evolving market while building solutions as quickly and efficiently as possible.

One of the most commonly employed methods evolved from the Toyota Production System, later coined as Lean, a systematic method for waste reduction [18]. The aim of employing Lean in software development was to use this approach to build solutions that created value for customers, eliminated waste, empowered the developers, and allowed for continuous improvement. Not only has Lean methodology become commonplace in software development, its use has been growing in global health care and as a validated method to employ evidence-based practices and continuous quality improvement in diverse of practical settings [18-20].

Basic Emergency Care

The 2015 Global Burden of Disease Study [21] found that emergency medical diseases contributed to more than half of all years of life lost globally. While this continues to be a global burden, it is most acutely felt by low- and middle-income countries, which have 4.4 times the disease burden of high-income countries [21]. Moreover, it is estimated that over half of all deaths in these countries are potentially addressable by emergency care [22]. Well-organized emergency care systems play an important role in the delivery of emergency services to and the health outcomes of patients in low-resource settings [23]; however, frontline providers in low- and middle-income countries often lack the basic training to recognize and treat life-threatening conditions [24], and the accessibility of adequate emergency care in low-resource settings is limited [25]. The burden of acute diseases, coupled with the lack of emergency care training in low- and middle-income countries, begets unnecessary mortality.

Prioritizing an integrated approach to early recognition and resuscitation substantially reduces morbidity and mortality associated with emergent medical conditions [26,27]. As such, the World Health Organization developed an open-access Basic Emergency Care course to provide standardized training in basic assessment and life-saving techniques using a traditional lecture format. The course covers the Airway, Breathing, Circulation, Disability, and Exposure approach to pediatric and adult emergencies, trauma, respiratory distress, shock, pregnancy, and altered mental status. The Basic Emergency Care course is not, in itself, an eHealth intervention. It is traditional didactic training, consisting of a 5-day series of classes, aimed at teaching participants how to recognize and intervene in a number of emergent conditions (the most significant causes of morbidity and mortality

Participants are trained to conduct a primary survey of emergency situations they may encounter situation, using the mnemonic Airway, Breathing, Circulation, Disability, and Exposure, which is, in itself, a concise and standardized algorithm; however, many branch points and deviations as well as condition-specific interventions are possible. But while there are a large number of emergent medical conditions annually, each practitioner may only encounter a few of any particular type in a given year. Diagrammatic representations of the possible emergency situations that one might encounter contained multiple decision points and branches; thus, their complexity made them too unwieldy to be useful. Furthermore, not all emergent conditions take place in a hospital, with many occurring in the field and being managed by emergency medical technicians. Thus, a mobile point-of-care reference was suggested as a means to help individuals recall Basic Emergency Care course training.

While there are many software development models to choose from, such as Waterfall or Scrum, we believe that the breadth of use and familiarity of Lean across health care administration, the software industry, and global health make it the ideal model to be used to address some of the inherent difficulties of global health software development and implementation (Figure 1) [28,29]—each is necessary for developing eHealth interventions and each has robust familiarity with this methodology [19,30,31]. Global health experts and their stakeholders are necessary to understand the landscape of issues needing to be addressed within a community. Software developers need to be able to communicate with global health experts to understand those needs and the scope of technological solutions. Finally, health care administrators must be able to respond to their community and clinician needs as well as implement the developed solutions to work within their institution.

Figure 1. Stakeholder overlap with Lean methodologies.
View this figure

Given that the Lean process makes use of limited resources, empowers local stakeholders, and facilitates remote development, it may prove to be a successful model for wide-scale implementations [30]. We aimed at employing a standardized Lean software development strategy to quickly develop and iterate a community-based global eHealth initiative for diverse stakeholders.

Lean Strategy

Lean software development has 7 core elements [32]: seeing the whole, empowering the team, building integrity in, amplifying learning, deciding as late as possible, delivering as fast as possible, and eliminating waste (Figure 2).

Figure 2. Lean workflow in global eHealth software development.
View this figure
Seeing the Whole

Complex systems are composed of networks of interconnected components that influence each other, often in a nonlinear fashion [33]. Health care, specifically global health care, is a complex system. Problems are rarely simple and rarely have clear boundaries [34,35]. In order to develop broad and novel solutions, teams must be able to see the whole and avoid a hammer-looking-for-nails approach. In global health care, this is often exacerbated by a tendency to focus on squeaky wheels [36] that distract from the true underlying issue—which may be more difficult to address. To address this problem, Lean systems use root-cause analyses. While there are many different types of analyses, one simple approach is called The 5 Whys (Figure 3) [37,38]. This is an iterative interrogation technique used to explore the cause-and-effect relationships underlying a particular problem with the goal of determining the root cause by repeating the question “Why?”

Figure 3. The 5 Whys for the Basic Emergency Care app.
View this figure

Initial project planning using this approach began in early 2016 with input from health care providers and software developers. Because groups tend to overestimate the importance of their particular skill set, solutions to a root-cause analysis must be derived from stakeholders without a priori assumptions. For the Basic Emergency Care app, it was participants who referenced their access and regular use of mobile phones for medical purposes which guided the solution.

Empowering the Team

Team empowerment strategies have been shown to improve many elements of health care, from individual management of disease, public health strategies, or improving partnership and sustainability in global health [39,40]. Team empowerment requires: (1) enhancing meaningfulness of work, (2) fostering participation in decision making, (3) facilitating goal accomplishment, and (4) providing autonomy and freedom from bureaucratic restraints that can cause unnecessary delays and bottlenecks [41].

For the Basic Emergency Care app, team empowerment began in mid-2016 with a shared vision of both the problem and solution. Developers, who lacked the experience of emergency department care or global health initiatives, spent time with those who had, in the form of structured meetings and ongoing email communication, throughout the app development and testing process. Efforts were made to make videoconferencing available to developers whenever possible during testing so that helpful features and hurdles to use could be identified and addressed in real time. Participation across the development team was fostered through a variety of free and secure resources, such as Slack (a collaboration hub for projects which allows document sharing and conferencing) and GitHub (a version control system widely used in software engineering which allows for shared to-do lists and assignment of tasks). By fostering shared decision making through open communication and regular feedback, team members were able to develop a plan for addressing their tasks, which was then shared with the team, and suggest changes as necessary based on their role in the team. This allowed for participation in decision making and autonomy in areas of subject-matter expertise.

Building Integrity In

In the software engineering, integrity is described as a freedom from flaw, defect, or decay [42]; a solution should have internal and external consistency. That is, it should make intuitive sense to the end user how to interact with the solution (ie, perceived integrity), and the internal functioning of the solution should be the appropriate use of engineering and organization resources (ie, conceptual integrity). Software is usually expected to evolve gracefully as it adapts to the future. Solutions with integrity have a coherent architecture; score highly on usability and fitness for their purpose; and are maintainable, adaptable, and extensible. Research has shown that integrity comes from wise leadership, relevant expertise, effective communication, and healthy discipline [32]. Thus, integrity is achieved through excellent detailed information flow—from users to the development team and between the development team members (Figure 4) [43].

Figure 4. Integrity through information flow [30].
View this figure

Achieving integrity for the Basic Emergency Care app first required an empowered team capable of communicating and receiving feedback as described above. During app development from September 2016 through January 2017, all team conversations and amendments to software were recorded using Slack and GitHub, allowing for easy sharing with new members as they joined the team smoothing onboarding and preventing delays when team members transitioned. Furthermore, feedback channels for remote team members and site leads or prototype testers were maintained to easily provide feedback and suggestions to optimize the solution’s fit.

Amplifying Learning

Gathering information is an important step in solving new problems. For complex problems, the preferred approach to a solution is to use the scientific method: observe, create a hypothesis, devise an experiment to test the hypothesis, run the experiment, and see if the results are consistent with the hypothesis [32,44]. Today it is widely accepted that software design is a problem-solving process that involves discovering solutions through short repeated cycles of investigation, experimentation, and results validation. In Lean development, this is considered the sum of 2 other steps—waiting to as long as possible to finalize a decision or acting quickly and soliciting feedback.

Learning from past interventions has been difficult in prior eHealth initiatives. This is sometimes due to rigid statistical measures that attempt to measure the effect of the software itself rather than how it corresponds to the principles on which it is based [45]. Furthermore, many initiatives do not even make it past the pilot stages, thus limiting the broad awareness of their successes and failures due to publication bias [7,45]. Implementation science measures such as reach, effectiveness, or acceptance are rarely reported, making it difficult to learn from prior endeavors or predict successful strategies. Incorporation of theory-informed frameworks for understanding factors associated with adoption and nonadoption are essential for evaluating eHealth software and generating knowledge that can be applied to other settings [46].

Given that we expected he Basic Emergency Care app to be used in many different settings, we aimed to develop the app with avenues of concomitant feedback that would allow for product improvements and implementation improvements. This included readily available and standardized survey methods and a grounded theory approach to leading focus groups. We conducted these sessions, in Tanzania and South Africa from January 11, 2017 through February 5, 2017. Feedback from each session was used to modify the app; iterative changes to the app were tested in subsequent groups.

Deciding as Late as Possible

Development practices that provide for late decision making are effective in domains that involve uncertainty because they provide an option-based approach. A key strategy for delaying commitments when developing a complex solution is to build a capacity for change into the system. Delayed decisions are valuable because they can be made based on facts rather than speculation. This is also sometimes referred to as concurrent development or just-in-time production [47,48] Concurrent development allows for a breadth-first approach, to discover big costly problems early on in the development process (before it is too late). Concurrent development, as opposed to sequential development, means a team starts programming the highest value features as soon as high-level conceptual designs have been determined; detailed requirements may still be undergoing investigation. In addition to ensuring against costly mistakes, concurrent development is considered to be the best method for dealing with changing requirements [32]. This is because not only are the big decisions deferred while considering all options, but the little decisions are deferred as well. When change is expected, concurrent development reduces delivery time and overall cost, while improving the performance of the final product. But concurrent software development also has costs. It requires having developers with enough expertise in a particular domain (ie, mobile, machine learning, etc) to anticipate where the emerging design is likely to lead and having collaboration between customers and analysts [32].

During initial user feedback sessions for the Basic Emergency Care app, some participants suggested only allowing the user to progress linearly through the entire pathway thus forcing users to complete the entirety of the primary survey every time they use the app, but a particular subgroup requested the ability to simply search course material as soon as they identified a problem. Given that whether users would be allowed to jump through the app or progress linearly fashion would require significantly different interfaces and functionality (ie, a search bar and the ability to bypass certain checkpoints), it was necessary to test both options and delay decision, until the options had been vetted by domain experts or one had to be deployed. During the first iteration of this process in February 2017, engineers examined the product roadmap to determine whether this feature would even be possible, while end users provided feedback about feature necessity and value.

Delivering as Fast as Possible

While seemingly contradictory, delivering as fast as possible complements deciding as late as possible. The faster a change or feature can be delivered, the longer decisions can be delayed before testing. Furthermore, without speed, it is not possible to have reliable feedback. In software development, the discovery cycle—design, implement, feedback, improve—is critical for learning. The shorter these cycles, the more can be learned.

For the Basic Emergency Care app navigation, a software solution that made both options possible, was chosen early during app development (in late 2016). Team members tested this solution for bugs. Both options were later deployed to determine the impact of each option on user satisfaction, app use, or recall of the course material.

Eliminating Waste

As the name implies, eliminating waste is the core concept of the Lean development strategy. Waste can come in many forms; from monetary to time. Anything that does not directly add value to the end-user is a waste. There are some particular methods that have been described to see and eliminate waste in software development, called the 7 wastes: partially done work, extra processes, extra features, task switching, waiting, motion, and defects [49]. Eliminating waste is a central tenant of the Lean development strategy. Most authors suggest starting with an approach to eliminate waste. However, in global health, eliminating waste requires a holistic view of the problem, an understanding of what is possible, and stakeholder involvement as well as feedback before assumptions are made. Thus, we have this pivotal step at the end as a reminder to assess the prior work in context. Many of the strategies to eliminate waste are addressed by the other 6 core elements of Lean. Solutions with integrity limit motion and task-switching by allowing for bidirectional feedback from users to the engineering team so that workflows may be taken into account in the solution. Partially done work is limited when decisions are made only when absolutely necessary. Waiting is minimized by delivering as fast as possible. Defects are identified by amplifying learning. Finally, extra features and processes are only identified by seeing the whole and understanding the purpose or utility of each within the greater context of addressing the problem at hand.

For the Basic Emergency Care app, waste could be considered in 2 dimensions: cost and time. Waste in both of these dimensions could result from unnecessary features or delays in feedback or delivery of requested features. Thus, we utilized continuous feedback from stakeholders and iterative testing to ensure that we were not expanding the scope or necessary elements of our solution. This helped us to limit engineering demands and to give directed instructions to developers, which helped to limit costs and time. We were able to create our pilot app (Figure 5) for the cost of a single Android and iOS app developer (who was hired for a per-project fee of $3000) and web-hosting ($96 annually). Development took 6 months (from initial scope description until pilot completion).

Figure 5. Screenshot of the Basic Emergency Care app.
View this figure

In a convenience sample of participants from across all levels of training, the majority (56/59, 95%) believed that the Basic Emergency Care app was “useful for their practice.” Responses included that it was an effective means to help “remind (me) to complete the primary survey on all patients” and prevented “missing steps.” A subgroup of physicians trained in emergency medicine (9/30, 30%) reported that they would prefer if the tool allowed “searching conditions” or references for “specific interventions” rather than a rigid algorithm as “I already know what the problem is, but now I want to know how to address it.” This feedback was used to design the subsequent versions of the app to improve acceptance and usability for all target providers.


Our goal was to develop a mobile app through the use of a model that was well-known to each participant stakeholder in a global health initiative: global health, health care administration, and software engineering. We found that in this setting, the Lean software development model was effective for limiting costs, addressing stakeholder needs and local practices, and bringing solutions to scale quickly.

Online calculators estimated a cost between $11,700 to $71,000 to develop an app such as the Basic Emergency Care app with similar data and infrastructure needs [50,51]. However, we were able to build our pilot app for a fraction of that cost at total monetary cost of $3096 for software development and annual web-hosting. Of course, we recognize that there are other costs associated with development that must be taken into account. While the monetary cost of our app was low, it took much longer to develop than the estimates given for professional app development. But we believe that this speed was likely balanced by the benefit of continuous feedback in a participatory model which helped to empower the team and build integrity [52-55]. The participatory model emphasized by the Lean software development model is particularly helpful in global health app development where needs span geographic, linguistic, and technological barriers.

There is a poignant need for mobile solutions in low- and middle-income countries. Many health care providers there have mobile devices, and there has been an increase in the development of apps to support care in these locations [2]. However, global health systems are complex and have different practices, system needs, and user demographics [13]. Thus, meaningful resources should not be built entirely external to the primary stakeholders, as many have been to date. However, access to software engineers can be limited for global health interventions that do not often have long-term scalability, need for full-time engineering services, or typical funding. When professional engineering services are employed, time with the engineers is often limited, due to cost or access, and familiarizing them with local needs can take time, limiting their contributions.

However, we do not believe this must necessarily be the case. Given that the Lean model allows for robust stakeholder involvement, rapid iteration, and a holistic approach to problems while sharing a common language for global health, software development, and health care administration, we believe that it should be the model of choice for software development in low- and middle-income countries.


This study had several limitations that need to be considered. First, the communities of Tanzania and South Africa may not be representative of other low- and middle-income countries outside of sub-Saharan Africa. The goal is for the Basic Emergency Care app to be available worldwide, though it is currently only available in English-speaking countries. However, part of utilizing a model such as Lean is to promote the practice and skills necessary to generalize solutions to any community. We believe that the process of seeing the whole, in conjunction with the other core elements, make this model particularly well-suited to communities that may be different from those in which we tested our model and software.

Additionally, while our primary university affiliation is in the heart of a major metropolitan software and technology development area (ie, Silicon Valley), other programs may not find it as easy to engage a rounded team of health professionals, engineers, and stakeholders locally in their community. The Lean iterative process requires the timely engagement of team members to effectively suggest, implement, and revise eHealth software. This can be especially challenging when team members operate in different time zones and with different cultural backgrounds. However, as training and education for computer engineers continues to increase, with accessibility in-person and online, there continues to be growth of local engineering capabilities across the globe. The model utilized above is meant to be agnostic to the problem being addressed and the particulars of coding languages.

Similarly, new software development strategies, such as Scrum, continue to gain popularity, and newer models evolve all the time. It’s possible that these strategies will prove to be even more efficient than Lean for producing and iterating software products. However, Lean has gained broad acceptability in a multitude of industries across borders and specialties since it was first introduced. This makes it particularly well-suited to maintaining the robust source of knowledge and integrity necessary for consistent global health interventions.


Lean processes have become a standard in the software development industry, but the methodology also has a proven track and has gained broader acceptance in the health care administration and global health communities. Thus, the Lean methodology presents a particularly suited framework which is already available to most of the parties involved in developing global eHealth solutions. Furthermore, its ideals—such as empowering the team, eliminating waste, and having a big-picture view, are implicitly aligned with those of global health interventions. We believe Lean strategies applied to software development for global health initiatives, particularly for low- and middle-income countries, may address some of the concerns regarding the prior limitations of these interventions.


The authors gratefully acknowledge the dedication and input of all participating general practices and trial participants. CR was supported in part by a Veteran’s Affairs Office of Academic Affiliations Advanced Fellowship in Informatics.

Conflicts of Interest

None declared.


  1. Mills A. Health care systems in low- and middle-income countries. N Engl J Med 2014 Feb 06;370(6):552-557. [CrossRef] [Medline]
  2. Beratarrechea A, Lee AG, Willner JM, Jahangir E, Ciapponi A, Rubinstein A. The impact of mobile health interventions on chronic disease outcomes in developing countries: a systematic review. Telemed J E Health 2014 Jan;20(1):75-82 [FREE Full text] [CrossRef] [Medline]
  3. Robinson T, Cronin T, Ibrahim H, Jinks M, Molitor T, Newman J, et al. Smartphone use and acceptability among clinical medical students: a questionnaire-based study. J Med Syst 2013 Jun;37(3):9936. [CrossRef] [Medline]
  4. Digital health. World Health Organization.   URL: [accessed 2019-05-13]
  5. Nilsen W, Kumar S, Shar A, Varoquiers C, Wiley T, Riley WT, et al. Advancing the science of mHealth. J Health Commun 2012;17 Suppl 1:5-10. [CrossRef] [Medline]
  6. Boudreaux ED, Waring ME, Hayes RB, Sadasivam RS, Mullen S, Pagoto S. Evaluating and selecting mobile health apps: strategies for healthcare providers and healthcare organizations. Transl Behav Med 2014 Dec;4(4):363-371 [FREE Full text] [CrossRef] [Medline]
  7. Mechael P, Batavia H, Kaonga N, Searle S, Kwan A, Goldberger A, et al. Barriers and gaps affecting mHealth in low and middle income countries: policy white paper. Global Health Delivery Online. 2010.   URL: [accessed 2021-05-04]
  8. mHealth for development: the opportunity of mobile technology for health care in the developing world. Relief Web. 2009.   URL: https:/​/reliefweb.​int/​sites/​​files/​resources/​A695C439809709CF49257689001019B9-UNF_Feb2009.​pdf [accessed 2021-05-04]
  9. Germanakos P, Mourlas C. A mobile agent approach for ubiquitous and personalized eHealth information systems. 2005 Presented at: UM05 - Personalization for eHealth; July 29; Edinburgh, Scotland   URL:
  10. Kirwan M, Vandelanotte C, Fenning A, Duncan MJ. Diabetes self-management smartphone application for adults with type 1 diabetes: randomized controlled trial. J Med Internet Res 2013 Nov 13;15(11):e235 [FREE Full text] [CrossRef] [Medline]
  11. Peiris D, Miranda JJ, Mohr DC. Going beyond killer apps: building a better mHealth evidence base. BMJ Glob Health 2018;3(1):e000676 [FREE Full text] [CrossRef] [Medline]
  12. Sondaal SFV, Browne JL, Amoakoh-Coleman M, Borgstein A, Miltenburg AS, Verwijs M, et al. Assessing the effect of mhealth interventions in improving maternal and neonatal care in low- and middle-income countries: a systematic review. PLoS One 2016;11(5):e0154664 [FREE Full text] [CrossRef] [Medline]
  13. Ben-Zeev D, Schueller SM, Begale M, Duffecy J, Kane JM, Mohr DC. Strategies for mHealth research: lessons from 3 mobile intervention studies. Adm Policy Ment Health 2015 Mar;42(2):157-167 [FREE Full text] [CrossRef] [Medline]
  14. Mohr DC, Weingardt KR, Reddy M, Schueller SM. Three problems with current digital mental health research . . . and three things we can do about them. Psychiatr Serv 2017 May 01;68(5):427-429 [FREE Full text] [CrossRef] [Medline]
  15. Zhang M, Cheow E, Ho CS, Ng BY, Ho R, Cheok CCS. Application of low-cost methodologies for mobile phone app development. JMIR Mhealth Uhealth 2014 Dec 09;2(4):e55 [FREE Full text] [CrossRef] [Medline]
  16. How much does app development cost? Business of Apps.   URL: [accessed 2019-03-11]
  17. Neagu A. Figuring the costs of mobile app development. Formotus.   URL: [accessed 2019-03-11]
  18. Black JR, Miller D. The Toyota Way to Healthcare Excellence: Increase Efficiency and Improve Quality with Lean. Chicago: Health Administration Press; Oct 03, 2008.
  19. D'Andreamatteo A, Ianni L, Lega F, Sargiacomo M. Lean in healthcare: a comprehensive review. Health Policy 2015 Sep;119(9):1197-1209. [CrossRef] [Medline]
  20. Womack J, Byrne A, Fiume O. Going lean in health care. American Academy of Otolaryngology-Head and Neck Surgery.: Institute for Healthcare Improvement   URL: [accessed 2021-05-04]
  21. Razzak J, Usmani M, Bhutta Z. Global, regional and national burden of emergency medical diseases using specific emergency disease indicators: analysis of the 2015 Global Burden of Disease Study. BMJ Glob Health 2019;4(2):e000733 [FREE Full text] [CrossRef] [Medline]
  22. Debas HT, Donkor P, Gawande A, Jamison DT, Kruk ME, Mock CN. Essential surgery. In: Disease Control Priorities. Washington, DC: World Bank Publications; 2015.
  23. Smith J, Haile-Mariam T. Priorities in global emergency medicine development. Emerg Med Clin North Am 2005 Feb;23(1):11-29. [CrossRef] [Medline]
  24. Crisp N, Chen L. Global supply of health professionals. N Engl J Med 2014 Mar 06;370(10):950-957. [CrossRef] [Medline]
  25. Hodkinson P, Wallis L. Emergency medicine in the developing world: a Delphi study. Acad Emerg Med 2010 Jul;17(7):765-774 [FREE Full text] [CrossRef] [Medline]
  26. Reynolds T, Sawe H, Rubiano A, Shin S, Wallis L, Mock C. Strengthening health systems to provide emergency care. In: Disease Control Priorities: Improving Health and Reducing Poverty. Washington (DC): The International Bank for Reconstruction and Development/The World Bank; 2018.
  27. Hsia R, Razzak J, Tsai AC, Hirshon JM. Placing emergency care on the global agenda. Ann Emerg Med 2010 Aug;56(2):142-149. [CrossRef] [Medline]
  28. Balaji S, Murugaiyan M. Wateerfall vs v-model vs agile: a comparative study on SDLC. Int J Inform Tech Bus Manag 2012 Jun 29:26-30 [FREE Full text]
  29. Schwaber K, Beedle M. Agile Software Development With Scrum. Saddle River, NJ: Prentice Hall PTR; Oct 2001.
  30. Carter P, Desmond J, Akanbobnaab C, Oteng R, Rominski S, Barsan WG, et al. Optimizing clinical operations as part of a global emergency medicine initiative in Kumasi, Ghana: application of Lean manufacturing principals to low-resource health systems. Acad Emerg Med 2012 Mar;19(3):338-347 [FREE Full text] [CrossRef] [Medline]
  31. Vallon R, Wenzel L, Brüggemann ME, Grechenig T. An Agile and Lean process model for mobile app development: case study into Austrian industry. J Softw 2015 Nov;10(11):1245-1264 [FREE Full text] [CrossRef]
  32. Poppendieck M, Poppendieck T. Lean Software Development: An Agile Toolkit. Boston, MA: Addison-Wesley Professional; May 2003.
  33. Amaral L, Ottino J. Complex networks. Eur Phys J B 2004 Mar 1;38(2):147-162 [FREE Full text] [CrossRef]
  34. Hidalgo CA, Blumm N, Barabási AL, Christakis NA. A dynamic network approach for the study of human phenotypes. PLoS Comput Biol 2009 Apr 10;5(4):e1000353 [FREE Full text] [CrossRef] [Medline]
  35. Pourbohloul B, Kieny M. Complex systems analysis: towards holistic approaches to health systems planning and policy. Bull World Health Organ 2011 Apr 01;89(4):242-242 [FREE Full text] [CrossRef] [Medline]
  36. Senge P. The fifth discipline, the art and practice of the learning organization. Nonprofit Manag Leadersh 1991 May;30(5):37-37 [FREE Full text] [CrossRef]
  37. Ohno T. Toyota Production System: Beyond Large-Scale Production. Boca Raton, FL: CRC Press; 1988.
  38. Serrat O. The five whys technique. In: Knowledge Solutions: Tools, Methods, and Approaches to Drive Organizational Performance. Singapore: Springer Singapore; 2017:307.
  39. Leffers J, Mitchell E. Conceptual model for partnership and sustainability in global health. Public Health Nurs 2011;28(1):91-102. [CrossRef] [Medline]
  40. Wallerstein N. What is the evidence on effectiveness of empowerment to improve health? World Health Organization Regional Office for Europe. 2006.   URL: [accessed 2021-05-04]
  41. MacPhee M, Chang L, Lee D, Spiri W. Global health care leadership development: trends to consider. J Healthc Leadersh 2013 Jun:21 [FREE Full text] [CrossRef]
  42. Sterling C. Building integrity in. Getting Agile.   URL: [accessed 2021-05-04]
  43. Fujimoto T, Clark K. Product Development Performance: Strategy, Organization, and Management in the World Auto Industry. Cambridge, MA: Harvard Business School; 1991.
  44. Reinertsen D. Managing the Design Factory. New York, NY: Free Press; 1997.
  45. Geng EH, Peiris D, Kruk ME. Implementation science: relevance in the real world without sacrificing rigor. PLoS Med 2017 Apr 25;14(4):e1002288 [FREE Full text] [CrossRef] [Medline]
  46. Greenhalgh T, Wherton J, Papoutsi C, Lynch J, Hughes G, A'Court C, et al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. J Med Internet Res 2017 Nov 01;19(11):e367 [FREE Full text] [CrossRef] [Medline]
  47. Adachi T, Enkawa T, Shih LC. A concurrent engineering methodology using analogies to just-in-time concepts. Int J Prod Res 2007 May 03;33(3):587-609 [FREE Full text] [CrossRef]
  48. Sohlenius G. Concurrent engineering. CIRP Ann Manuf Technol 1992;41(2):645-655 [FREE Full text] [CrossRef]
  49. Poppendieck M. Principles of lean thinking. Software Engineering Group Universidad Nacional de San Luis. 2002.   URL: [accessed 2021-05-04]
  50. Estimate my app. Oozou.   URL: [accessed 2021-05-04]
  51. How much does it cost to build an app. Epic AI.   URL: [accessed 2021-05-04]
  52. Cornwall A, Jewkes R. What is participatory research? Soc Sci Med 1995 Dec;41(12):1667-1676 [FREE Full text] [CrossRef]
  53. Wallerstein NB, Duran B. Using community-based participatory research to address health disparities. Health Promot Pract 2006 Jul;7(3):312-323. [CrossRef] [Medline]
  54. Horowitz CR, Robinson M, Seifer S. Community-based participatory research from the margin to the mainstream: are researchers prepared? Circulation 2009 May 19;119(19):2633-2642 [FREE Full text] [CrossRef] [Medline]
  55. Rowe-Roberts D, Cercos R, Mueller F. Preliminary results from a study of the impact of digital activity trackers on health risk status. Stud Health Technol Inform 2014;204:143-148. [Medline]

Edited by G Eysenbach; submitted 29.05.19; peer-reviewed by A Joshi, A Kanter; comments to author 14.02.20; revised version received 30.11.20; accepted 13.04.21; published 26.05.21


©Christian Rose, Taylor Nichols, Daniel Hackner, Julia Chang, Steven Straube, Willem Jooste, Hendry Sawe, Andrea Tenner. Originally published in JMIR Formative Research (, 26.05.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on, as well as this copyright and license information must be included.