This was a first-stage feasibility trial for a novel app called MMApp. MMApp was designed to identify patients as multimorbid, defined by the presence of a Qualifying Comorbidity Set. Qualifying Comorbidity Sets are single, double, and triple combinations of acute and chronic medical conditions identified by Silber et al [5] to be associated with a 2-fold increased risk of in-hospital mortality after general surgery, and our group has found this to extend to the EGS setting [11,12]. We chose the EGS setting to test the feasibility of MMApp, as our previous retrospective investigations regarding multimorbidity have examined EGS populations [11,12]. If MMApp is feasible for use in the EGS setting, this would support prospective investigations of similar populations. Furthermore, time is limited in the emergency setting; thus, if MMApp is feasible for use here, then it would be reasonable to speculate that it is feasible for use in an elective setting as well, and further validation could be conducted. At this point, MMApp is purely investigational and not available for public use.

Qualifying Comorbidity Sets were developed using Centers for Medicare and Medicaid (CMS) claims [5]. The medical conditions that comprise these sets are defined using the standard Medicare Hierarchical Condition Category (HCC) coding system (version 22) [13-26] and indicators for disability with Current Procedural Terminology (CPT) codes. See Table S1 in Multimedia Appendix 1, which shows the full list of comorbid conditions and associated International Classification of Disease Ninth/Tenth Revision Clinical Modification (ICD 9/10-CM), CPT, and Health care Common Procedure Coding System (HCPCS) codes; see Table S2 in Multimedia Appendix 1, for Qualifying Comorbidity Sets.

A total of 5 physicians (CBR, SER, CF, JT, and RRK) and 1 data analyst (CW) translated the codes to clinical terms using a modified Delphi technique [27,28]. Four physicians independently analyzed the full list of ICD 9/10-CM, CPT, and HCPCS codes used to define each comorbid condition present in any Qualifying Comorbidity Set. Each physician developed a representative clinical definition for each claims-defined comorbid condition, along with specific included and excluded diagnoses needed to differentiate each comorbid condition from any other. Subsequently, the 5 physicians (CBR, SER, CF, JT, and RRK) and the data analyst (CW) met jointly. The group discussed one claims-defined comorbid condition at a time and compared it to the proposed clinical definitions. The group also considered the relevant inclusion and exclusion diagnoses for each clinical definition before developing a consensus agreement regarding the best possible clinical definition of each claims-defined comorbid condition. Any disagreements regarding the best possible clinical corollary were discussed, and the selection of the clinical definition was determined by a unanimous vote with equal veto power for all involved. If a veto was used by any member, further discussion and revision were required to achieve a unanimous vote of agreement. There were 3 rounds of discussion over the course of 5 weeks before a final list of clinical definitions, included diagnoses, and excluded diagnoses was determined for each comorbid condition. (See Table S3 in Multimedia Appendix 1 for clinical definition, included diagnoses, excluded diagnoses, and comparable Silber- or claims-defined comorbid condition for all comorbid conditions present in any qualifying comorbidity set.)

The computer scientist on the team (HS) engineered the iOS app using the Flutter framework (Google LLC), leveraging best practices for software development such as source control management using git and GitHub (GitHub, Inc). Comorbid conditions and Qualifying Comorbidity Sets were coded into JavaScript Object Notation files included in the app bundle, which are then compared against the user’s answers to determine multimorbidity status. The app was distributed directly to study team members’ phones using Apple’s beta distribution tools, including Xcode (Apple Inc), App Store Connect, and TestFlight (Apple Inc). Graphic design (SS) and button association were created using Figma (Figma Inc; Figures 1 and 2). See Figures 1 and 2 for examples of graphic design within MMapp. (See also Table S4 in Multimedia Appendix 1 for progression through MMApp for a patient with multimorbidity, scenario D; see Table S4 in Multimedia Appendix 1 for progression through MMapp for a patient with multimorbidity, scenario E.) Comorbid conditions were organized by organ system based on input from physician team members (CBR, SER, CF, JT, and RRK).

A total of 5 hypothetical patient scenarios were created by the research team and written in the History and Physical note style of the study institution (see Section 3 in Multimedia Appendix 1 for patient scenarios). Participants were presented with each patient scenario and asked to complete the MMApp. In order to benchmark the time required to use the MMApp against a commonly used clinical tool, participants were also instructed to apply each scenario patient to the American College of Surgeons National Surgical Quality Improvement Program Surgical Risk Calculator (ACS-NSQIP-SRC) [29]. Participants were instructed to select Exploratory Laparotomy with CPT 49000 as the procedure of choice for all patients when using the ACS-NSQIP-SRC and to fill in all other data points using only the information offered in the hypothetical patient’s History and Physical notes. This was done on a computer using the ACS NSQIP website [29].

To amass 50 total trials of MMApp (using 5 hypothetical scenarios), a total of 10 surgical residents from a single academic, tertiary care hospital were recruited for participation in feasibility testing using convenience sampling. None of the recruited surgical residents were part of the app development team in any way. Residents were contacted through text message to request participation and were not compensated. All included participants were in good academic standing at the institution involved and at least in postgraduation year 2 (PGY-2) or above. The study institution has a 7-year surgical residency (PGY-1 through PGY-7), with research years taken during PGY-4 and PGY-5. PGY-1 residents were excluded as they may not have adequate clinical experience with the initial evaluation of EGS patients.

The time to complete MMApp and ACS-NSQIP-SRC (in seconds) for each scenario was recorded by a stopwatch, controlled by a study team member administering the questionnaire and trial of MMApp (CBR). Time to complete each scenario with both MMApp and the ACS NSQIP-SRC was averaged across all participants and compared using paired t tests. For MMApp, comorbidities selected for each scenario were recorded, along with the number of comorbidities correctly chosen, incorrectly chosen, and missed for each scenario. The sensitivity and specificity of identifying a patient as multimorbid using MMApp were calculated using composite data from all scenarios. Usability and satisfaction with MMApp were assessed using an 18-item questionnaire administered immediately after completing all 5 scenarios, adapted from similar surveys used to assess mobile apps (see Section 4 in Multimedia Appendix 1 for the questionnaire) [30,31]. Participants rated agreement or disagreement with each questionnaire statement on a 5-point Likert-type scale: strongly disagree (1 point), disagree (2 points), neutral (3 points), agree (4 points), and strongly agree (5 points). The mean response score and SD across all participants were calculated. Open feedback from participants was clustered into themes of positive feedback and negative or formative feedback and analyzed for trends and comparison.

More than 50% of EGS patients are over 60 years old, and one-third are older than 70 years [32], many with multiple medical problems and at high risk for adverse outcomes [33-35]. However, the best and most efficient way to determine if an older patient is “high risk” in the clinical setting is yet to be clearly determined. Multimorbidity, defined by the presence of a Qualifying Comorbidity Set, has been used in retrospective, claims-based analyses to identify patients at particularly high risk for adverse outcomes after EGS [11,12], but translation for easy clinical use and validation had not been possible before the creation of MMApp. Ho et al [36] found functional limitations to be a very important risk factor for long-term survival after admission for an EGS condition. Frailty, though recognized to be associated with adverse clinical outcomes, has no gold-standard definition [13] and the frailty phenotype requires physical assessment of patients’ baseline clinical status, like grip strength, endurance, and walking speed [14], which can be difficult in an emergency setting or with an acutely ill patient. In that vein, Qualifying Comorbidity Sets, and thus MMApp should importantly include functional status indicators, like home oxygen or wheelchair use, which could serve as indicators of both functional limitation and frailty as contributing factors toward multimorbidity. Furthermore, a large prospective cohort study in the United Kingdom has found multimorbidity to be associated with a higher risk of mortality than frailty or disability [15]. Future work is necessary toward the validation of MMApp in further trials and could include a direct comparison of MMApp to frailty-based apps to predict the risk of adverse outcomes.

MMApp was designed for ease of use. In this study, most participants reported a disinterest in using the ACS-NSQIP-SRC [29] when evaluating EGS patients, in part due to the detailed information required for use, which can be difficult to find in the chart at the time of consultation (eg, the American Society of Anesthesiologists [ASA] classification). Alternative tools, such as the Emergency General Surgery Frailty Index (EGSFI), also require information that is difficult to find in the chart regarding the patient’s baseline functional status (eg, whether the patient needs assistance with specific activities of daily living, including toileting, housework, and grooming) [16]. As such, it is important to limit the amount of time, effort, and information input required by the clinician trying to determine if their patient is at a particularly high risk for adverse outcomes. MMApp uses information easily found in the patient’s medical record, standard history, and physical examination. In the EGS setting, we have demonstrated multimorbidity to be a comprehensive mechanism to stratify risk among EGS patients [11,12]. Based on this study’s findings, MMApp can be used to efficiently translate clinical findings into a diagnosis of multimorbidity, meriting future validation and investigation.

Multimorbidity is found in a higher proportion of ethnic minority groups [17] and is earlier in onset among non-Hispanic Black American than among non-Hispanic White individuals [18]. These disparate patterns of multimorbidity have been identified by the NIH as an area of interest [6]. The use of MMApp has the potential to advance health equity by allowing clinicians to easily classify patients as multimorbid. Given that non-Hispanic Black patients have lower participation rates in clinical trials [19-21] and higher rates of mortality and complications after EGS than White patients [22,23], it is important to understand the impact of multimorbidity on these disparities.

The efficient patient population identification made possible by MMApp could be generalized to other research and clinical settings. This could include facilitation of eventual risk prediction and goal setting for treatment plans or recruitment for clinical trials. Streamlined recruitment is especially important when trying to enroll subjects who are not easy to “see” or when there is complexity in the exclusion or inclusion criteria. Mobile apps are now being used and studied at all levels of research, from identifying subclinical atrial fibrillation in at-risk populations [24], to streamlined questionnaire administration for patient use [25], to at-home self-assessments to improve patient participation in interventions outside of the clinical setting [26]. However, many of these mobile apps are aimed toward patient use. Beyond MMApp itself, a simple, physician-facing app to quickly screen patients for enrollment in prospective clinical trials could boost invitations for participation, especially in the emergency setting when there is limited time. The methodology used to generate MMApp could be applied to generate similar apps to be used by research teams aiming to increase the efficiency of study recruitment.