To establish and demonstrate a transferable, dual-perspective evaluation framework for mHealth research instruments in high-burden occupational settings, this study used an iterative design and development methodology coupled with multitiered usability and feasibility evaluations, using the newly developed “Nurses’ Work-Life and Health” mobile app as an empirical exemplar. The app was designed to collect daily work- and health-related data from shift-work nurses over a 14-day period. An iterative, user-centered design was adopted, consisting of 3 sequential phases: conceptual framework and needs assessment (phase 1); app design and development involving alpha testing (internal usability testing) with nurse researchers (n=5) to refine system logic and detect technical bugs, followed by beta testing (user-based usability evaluation) with practicing clinical nurses (n=16) to evaluate real-world UI/user experience (UX) ergonomics and standardized usability (phase 2); and a 14-day pilot feasibility study with nurse end users (n=5) as a methodological proof-of-concept, complemented by qualitative process evaluation with an expert nurse researcher (phase 3; Figure 1). This 3-phase structure was specifically aligned to evaluate the feasibility, adherence, temporal integrity, and user burden associated with app-based longitudinal data capture in clinical settings.

Insights from both user- and researcher-oriented assessments indicated that existing, off-the-shelf survey tools were insufficient for the constraints of shift-work nursing, directly shaping the specifications for a bespoke mHealth app. From the user perspective, key design priorities included shift-synchronized notification logic with adaptive follow-up prompts, concise questionnaires designed for completion within 3 to 4 minutes, and an ergonomic visual layout that minimizes cognitive burden during postshift fatigue. From the researcher perspective, key design priorities included a secure back-end infrastructure capable of generating time-stamped logs and a cloud-based dashboard for real-time monitoring of data completeness.

During initial planning, the app’s architecture was systematically mapped using a hierarchical tree structure to define relationships among disparate data domains (eg, shift timing, sleep metrics, and daily fatigue) and optimize screen navigation flows. This hierarchical organization ensured that complex multidomain entry forms were decomposed into intuitive, low-cognitive-burden visual modules before software engineering commenced. Preliminary screen layouts and core functionalities were outlined to ensure intuitive navigation and minimal user burden. A detailed development plan was created using Microsoft PowerPoint, specifying screen layouts, UI elements, and functional requirements. To enhance usability, selected design features from existing calendar-based apps were incorporated, allowing users to visualize daily entries relative to work schedules. Sample screens were developed using SketchUp (Trimble Inc) on an iPad to provide a visual prototype. Throughout development, contemporary literature on diary methods and mHealth interventions was consulted to ensure that the app was theoretically grounded and methodologically sound.

Alpha testing was conducted to evaluate the initial prototype’s stability, functional integrity, and visual ergonomics from a multidisciplinary perspective before clinical exposure. Consistent with Nielsen and Landauer’s [20] recommendation that testing with 5 users can identify approximately 85% of usability problems, 5 experts—including 2 mobile app developers, 2 nurse researchers, and 1 clinical nurse—were purposively recruited and participated in testing. In this phase, participants primarily functioned as technical and structural evaluators tasked with identifying system bugs, workflow errors, and UI/UX layout inconsistencies.

A mixed methods approach was adopted incorporating a quantitative survey instrument adapted from the technology acceptance model [21] and the system usability scale [22] to systematically assess technical functionality and user-centered design. Quantitatively, participants rated 17 items across six domains using a 4-point Likert scale (1=“strongly disagree” and 4=“strongly agree”): (1) ease of use (eg, navigation of the calendar and timeline), (2) functionality (eg, data modification and information display), (3) user satisfaction, (4) technical performance, (5) perceived usefulness, and (6) intention to use. The 4-point scale was specifically selected to avoid neutral responses and elicit more definitive feedback. Qualitative data collection integrated domain-specific open-ended questionnaire items (eg, “If you encountered any issues or gave a low rating, please describe the experience in detail”), providing granular qualitative feedback on technical bugs, system crashes, or navigation barriers. The combined findings directly informed back-end system revisions and UI refinements prior to beta testing.

Following alpha testing refinements, beta testing was conducted to evaluate usability and feasibility in a real-world clinical context with 16 clinical shift-work nurses across diverse hospital units (eg, medical-surgical ward, intensive care unit [ICU], and emergency department [ED]). Participants were recruited via snowball sampling. Inclusion criteria for nurse participants required being a registered nurse with at least 6 months of clinical nursing experience, working a rotating shift schedule, and owning an Android-based smartphone. Nurses on extended medical or maternity leave, those working fixed day/night schedules, or those assigned exclusively to nonclinical administrative units were excluded. The focus was on the integration of the app into the demanding and irregular workflows of nursing professionals.

Similar to alpha testing, a mixed methods approach was used to evaluate clinical usability and UX in real-world nursing environments. Quantitative data collection used a structured survey instrument adapted from the technology acceptance model [21] and the system usability scale [22]. Participants completed a 15-item survey across six domains: (1) ease of use (eg, intuitive data entry during high-fatigue states), (2) functionality (eg, flexibility of data modification and app environment stability), (3) user satisfaction (eg, visual legibility of icons and colors), (4) technical performance (eg, app loading speed), (5) perceived usefulness (eg, effectiveness in monitoring the relationship between shift-work and health behaviors), and (6) user burden and intention to use (eg, minimal interference with clinical duties and willingness for 14-day continuous use). All items were rated on a 4-point Likert scale (1=“strongly disagree” and 4=“strongly agree”).

To capture contextual nuances specifically regarding operational barriers to app use, qualitative data collection integrated domain-specific open-ended survey questions (eg, “If you encountered difficulties during a specific shift, please describe the circumstances in detail”). This allowed researchers to identify how specific clinical circumstances and temporal constraints (such as high-intensity patient handovers or emergency admissions) directly impeded app interaction and timely data entry. Addressing these user-reported difficulties, qualitative feedback directly guided functional app refinements, such as extending permitted completion windows and optimizing dynamic push notification timing, thereby minimizing user friction and potential participant attrition prior to the 14-day feasibility study.

Through these iterative alpha and beta testing cycles, the UI and UX were optimized for clinical shift-work environments (Figure 2). Specifically, in addition to the main dashboard displaying daily summary metrics and timeline status (Figure 2A), the shift schedule configuration interface was designed to allow nurses to seamlessly set their rotating shift patterns (day, evening, night, off, or others), which automatically synchronizes dynamic postshift notification triggers (Figure 2B). Furthermore, to minimize cognitive friction during postshift fatigue, the daily recovery and health journal interface was refined using high-legibility visual sliders (eg, a 1‐10 scale for measuring sleep quality, subjective health, and daily stress levels) and streamlined inputs, enabling rapid and intuitive data capture without text-entry burden (Figure 2C).

The study protocol was reviewed and approved by the institutional review board of Chung-Ang University (approval number 1041078‐20240131-HR-023) prior to participant recruitment and data collection. Formal written informed consent was obtained from all individual participants prior to enrollment in each phase of the study. The consent process outlined the study objectives, data collection procedures, confidentiality safeguards, and the right to withdraw at any time without any professional or administrative penalty. To protect participant privacy and maintain confidentiality, all electronic research data were strictly deidentified. Upon registration, the back-end authentication system automatically assigned a nonidentifiable user identifier to decouple research datasets from personal identity information. Data transmissions were encrypted in transit via SSL/HTTPS protocols, and all stored records within Cloud Firestore were encrypted at rest using the AES-256 standard, with administrative access restricted strictly to primary investigators. To acknowledge their time and effort, clinical nurses who completed the 14-day feasibility protocol received a mobile electronic gift voucher valued at approximately US $35 (50,000 KRW), while participants in the alpha and beta testing phases received nominal token compensation (coffee coupons valued at US $5‐10) upon survey completion.

The 14-day feasibility evaluation demonstrated strong protocol feasibility and high temporal fidelity. Across the 70 total participant-days (5 participants × 14 d), objective adherence reached 94.3% (66/70 participant-days; mean 13.2, SD 1.1 d/nurse) with 100% participant retention (Table 3). Of the 70 expected daily journal entries, participants successfully completed 62 entries, yielding an overall completion rate of 88.5% (62/70). The analysis of server-side UTC time stamps confirmed that 83.9% (52/62) of completed entries were submitted within the strict 4-hour completion window following dynamic shift-matched notifications, supporting the feasibility of near–real-time data capture in fast-paced clinical environments. The remaining 16.1% (10/62) of entries were submitted beyond the 4-hour window and flagged by the back end as delayed entries, reflecting periods of acute workload surges or extended clinical handovers. System logs showed that the average time spent per entry was 3.4 (SD 0.8) minutes, which was rated as highly acceptable (mean 3.70, SD 0.50) and associated with low perceived interference with daily routines (mean 3.65, SD 0.45), despite demanding shift schedules.

A semistructured interview with the participating expert nurse researcher, who used the app daily over the 14-day period, highlighted several points:

Overall, the 14-day feasibility and process evaluation suggested that the app has practical value for both research data collection and self-monitoring, while also identifying opportunities for further refinement in technical performance and workflow design.

This study developed and evaluated the “Nurses’ Work-Life and Health” app to address the methodological challenges of longitudinal data collection in shift-working nursing populations. Beyond demonstrating the technical viability of the app, the principal scientific contribution of this study lies in its transferable, dual-perspective evaluation framework. By integrating subjective end-user experiences, objective back-end system logs, and in-depth qualitative scrutiny from an expert researcher who directly engaged in the protocol, this framework bridges the critical gap between technological functionality and research utility. When interpreting these outcomes, it is essential to distinguish between context-specific pilot findings and the broadly transferable aspects of our methodology. Given the pilot scale of the feasibility evaluation (n=5), specific quantitative indicators—such as the 100% retention rate, 94.3% adherence rate, and 3.4-minute entry time—represent context-specific feasibility metrics bounded by this particular clinical sample. In contrast, the broader scientific contribution resides in the transferable design mechanisms and methodological principles established through this work. Specifically, integrating shift-synchronized notification logic with low-cognitive timeline interfaces demonstrates how mHealth tools can minimize reporting burden in high-fatigue settings, while the 3-tiered evaluation protocol offers a replicable strategy for digital health investigators seeking to validate data collection tools in complex occupational environments.

The high level of engagement achieved in this study may be primarily attributed to the app’s ease of use and efficiency. Although initial qualitative feedback from alpha and beta testing highlighted challenges with navigation for high-fatigue users, iterative refinements—including ergonomic “thumb-friendly” adjustments and enhanced visual differentiation—appeared to mitigate these concerns. Objective system logs showed that the average time required for each entry was 3.4 minutes, suggesting minimal time burden for repeated reporting. This low entry burden is meaningful in the context of repeated mobile diary reporting and EMA research, where participant compliance and sustained engagement are closely related to the practical burden of data entry [4,23]. The expert nurse researcher further corroborated these findings, noting that recall bias was significantly reduced by the app compared to paper diaries by allowing near–real-time data capture. By reducing the effort required for data entry, the app may reduce the back-filling of data common in retrospective reports, thereby enhancing the ecological validity of the longitudinal data [24].

Beyond its role as a data collection instrument, the app demonstrated potential as a self-monitoring tool. The participating expert nurse researcher reported increased self-awareness regarding health behaviors, such as hydration and sleep patterns. Although this behavioral reactivity could introduce bias in observational studies [25], it also highlights the app’s potential for self-monitoring and behavior reflection. While this study used the Job Demands–Resources and Effort–Recovery models as instrumental guides for variable selection, the high completion rates across daily work and recovery domains support the practical utility of these frameworks for structuring longitudinal mHealth diaries. Moving forward, future iterations could incorporate personalized feedback dashboards or real-time alerts, extending the app from a passive data collection tool into a closed-loop intervention platform [26,27]. Integration with wearable sensors could validate subjective diary data against objective physiological metrics (eg, heart rate variability) to provide a more nuanced understanding of the work-recovery process.

However, the expert process evaluation also emphasized the importance of methodological safeguards for broader implementation. Key concerns included delayed data entry during high-intensity shifts and the risk of satisficing (choosing the easiest response) under extreme fatigue [28]. To mitigate these risks and support data integrity, the app could incorporate smart-validation mechanisms and dynamic skip logic to filter irrelevant items and prevent inconsistencies in self-reported data [29]. The integration of context-aware prompts (eg, shift-synchronized notifications or GPS-based reminders) could ensure timely, accurate entries without disrupting clinical duties. Additionally, expanding the tool to include more diverse contextual variables (eg, specific patient safety incidents or unit-level staffing) would increase its utility for broader epidemiological or organizational research. By evolving into a more adaptive longitudinal data collection platform, the app can support the synthesis of high-fidelity evidence critical for nursing research.

The evaluation framework established in this study offers broader methodological utility for digital health research. In mHealth and EMA research, tools are frequently deployed without rigorous prevalidation of their impact on participant burden or data integrity [16]. Our 3-tiered evaluation strategy—combining iterative usability testing, objective compliance logging, and firsthand expert process evaluation—provides a standardized, replicable roadmap for researchers developing digital tools for other high-stress, shift-work, or clinical populations (eg, physicians and emergency responders). Adopting this dual-perspective assessment ensures that digital data collection instruments are not only technologically robust and user-friendly but also methodologically capable of minimizing recall and reporting biases.

Several limitations should be acknowledged. First, the app was developed exclusively for the Android platform, which may initially limit accessibility. However, this platform-specific focus facilitated the stabilization of the back-end architecture before cross-platform expansion. Second, while the sample sizes for usability testing (n=16) and feasibility evaluation (n=6) align with established usability engineering standards and pilot study guidelines [20,30], the quantitative implementation outcomes reported here are inherently preliminary and context-specific. Furthermore, the reliance on nonprobability sampling methods (purposive and snowball sampling) across testing phases introduces potential selection bias. Participants in this pilot may represent early adopters with higher digital literacy, greater intrinsic health awareness, or higher research engagement, which could overestimate adherence and satisfaction compared to broader clinical populations. In addition, while participants represented major high-burden units (medical-surgical wards, ICU, and ED), certain specialty areas—such as outpatient clinics or psychiatric units—were not covered, limiting the applicability of findings across all clinical contexts. Finally, while the title and conceptual framework emphasize nurses’ health, this 14-day pilot study primarily evaluated the feasibility, user acceptability, and temporal fidelity of the data collection platform rather than directly assessing long-term clinical health outcomes. Although the daily journal captures key health-related behaviors and recovery metrics, the lack of direct evaluation of health outcome trajectories represents a main limitation of this feasibility study. Accordingly, our conclusions are bounded by these pilot data, and the specific metrics should not be directly generalized as definitive population-level estimates. Rather, the primary value of this pilot lies in demonstrating the proof-of-concept for the dual-perspective evaluation framework and identifying structural refinements required for scalability. Future research must deploy this tool across larger, more heterogeneous nursing cohorts using probabilistic sampling and extended follow-up windows to evaluate long-term durability and generalizable implementation outcomes. To build upon this platform, subsequent iterations should also incorporate daily self-reported journal entries with objective physiological markers—such as wearable actigraphy or heart rate variability—to empirically evaluate the direct impact of shift work on nurses’ long-term clinical health outcomes.

This study establishes and demonstrates a transferable, dual-perspective evaluation framework for assessing mHealth data collection tools in high-burden, shift-work occupational settings. By combining subjective end-user feasibility metrics, objective back-end usage logs, and qualitative process evaluation by an expert researcher, this multitiered approach bridges the gap between technical usability and methodological rigor. Applied to the newly developed “Nurses’ Work-Life and Health” app, the framework supported the feasibility of high temporal fidelity longitudinal data capture in complex clinical environments without imposing prohibitive cognitive burden on participants, as evidenced by high adherence (94.3%) and a brief 3.4-minute entry time.

Beyond the immediate evaluation of this specific app, this integrated evaluation strategy provides a replicable protocol for digital health investigators seeking to validate mHealth and EMA tools prior to full-scale deployment. While the framework demonstrated the app’s readiness for real-time monitoring, it also effectively identified crucial safeguards needed for future scalability, including dynamic skip logic, context-aware reminders, and automated data validation. Ultimately, this evaluation framework serves as a robust methodological foundation for future mHealth research, facilitating the collection of high-quality evidence needed to design data-driven interventions and policies that support nurses’ well-being and patients’ safety.