This paper is in the following e-collection/theme issue:

Development and Evaluation of Research Methods, Instruments and Tools (511) Artificial Intelligence (1421) Big Data (245) Digital Science, Open Science (79)

Published on in Vol 8 (2024)

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/55496, first published .
Accessible Ecosystem for Clinical Research (Federated Learning for Everyone): Development and Usability Study

Accessible Ecosystem for Clinical Research (Federated Learning for Everyone): Development and Usability Study

Accessible Ecosystem for Clinical Research (Federated Learning for Everyone): Development and Usability Study

Authors of this article:

Ashkan Pirmani1, 2, 3, 4 Author Orcid Image ;   Martijn Oldenhof1 Author Orcid Image ;   Liesbet M Peeters2, 3, 4 Author Orcid Image ;   Edward De Brouwer1 Author Orcid Image ;   Yves Moreau1 Author Orcid Image
Article Authors Cited by (1) Tweetations Metrics

    Original Paper

    1ESAT-STADIUS, KU Leuven, Leuven, Belgium

    2Data Science Institute, Hasselt University, Diepenbeek, Belgium

    3University Multiple Sclerosis Center, Hasselt University, Diepenbeek, Belgium

    4Biomedical Research Institute, Hasselt University, Diepenbeek, Belgium

    *these authors contributed equally

    Corresponding Author:

    Yves Moreau, PhD

    ESAT-STADIUS

    KU Leuven

    Kasteelpark Arenberg 10

    Leuven, 3001

    Belgium

    Phone: 32 16 32 86 45

    Email: Yves.Moreau@esat.kuleuven.be


    Background: The integrity and reliability of clinical research outcomes rely heavily on access to vast amounts of data. However, the fragmented distribution of these data across multiple institutions, along with ethical and regulatory barriers, presents significant challenges to accessing relevant data. While federated learning offers a promising solution to leverage insights from fragmented data sets, its adoption faces hurdles due to implementation complexities, scalability issues, and inclusivity challenges.

    Objective: This paper introduces Federated Learning for Everyone (FL4E), an accessible framework facilitating multistakeholder collaboration in clinical research. It focuses on simplifying federated learning through an innovative ecosystem-based approach.

    Methods: The “degree of federation” is a fundamental concept of FL4E, allowing for flexible integration of federated and centralized learning models. This feature provides a customizable solution by enabling users to choose the level of data decentralization based on specific health care settings or project needs, making federated learning more adaptable and efficient. By using an ecosystem-based collaborative learning strategy, FL4E encourages a comprehensive platform for managing real-world data, enhancing collaboration and knowledge sharing among its stakeholders.

    Results: Evaluating FL4E’s effectiveness using real-world health care data sets has highlighted its ecosystem-oriented and inclusive design. By applying hybrid models to 2 distinct analytical tasks—classification and survival analysis—within real-world settings, we have effectively measured the “degree of federation” across various contexts. These evaluations show that FL4E’s hybrid models not only match the performance of fully federated models but also avoid the substantial overhead usually linked with these models. Achieving this balance greatly enhances collaborative initiatives and broadens the scope of analytical possibilities within the ecosystem.

    Conclusions: FL4E represents a significant step forward in collaborative clinical research by merging the benefits of centralized and federated learning. Its modular ecosystem-based design and the “degree of federation” feature make it an inclusive, customizable framework suitable for a wide array of clinical research scenarios, promising to revolutionize the field through improved collaboration and data use. Detailed implementation and analyses are available on the associated GitHub repository.

    JMIR Form Res 2024;8:e55496

    doi:10.2196/55496

    Keywords

    federated learning; multistakeholder collaboration; real-world data; integrity; reliability; clinical research; implementation; inclusivity; inclusive; accessible; ecosystem; design effectiveness 


    Real-World Data in Health Care: Toward Federated Learning Solutions

    Real-world data (RWD), which encompass clinical and health data, are a fundamental asset in health care, providing invaluable insights for research and clinical applications [1,2]. However, for privacy, legal, collaborative, or practical reasons, RWD are typically fragmented into different data sources, such as electronic health records, patient registries, and other clinical repositories. This fragmentation, characterized by the dispersion of RWD across numerous health care systems and geographic locations, hampers the Findability, Accessibility, Interoperability, and Reusability (“FAIRness”) of the RWD [3]. Thus, the capacity to harness these fragmented data sources for evidence-based health care decision-making is hindered and must be improved [4,5].

    While limited data sharing impedes the clinical research progress in general, the impact is particularly significant for research into low-prevalence diseases, where data scarcity already poses a challenge [6]. This issue becomes especially critical in urgent situations, such as the COVID-19 pandemic [7,8], where rapid data sharing is vital to quickly understand the impact of a novel disease.

    Nevertheless, many challenges stemming from RWD fragmentation are systemic and can be seen as constraints that are difficult to eliminate. Legal data protection frameworks, ethical considerations, and organizational policies of different countries often limit the dissemination of such data both within and across countries [9-12]. These challenges are further complicated by different data registries using different data formats, standards, and infrastructures to reflect their specific needs [13]. This context raises the need for a global data sharing framework to facilitate the secure and efficient sharing of RWD, while accommodating most of the complexities of the clinical data ecosystem.

    A key component of such a global data sharing framework is federated learning (FL), which offers a solution by training machine learning models on distributed data sets without centralization [14,15]. Nevertheless, current FL packages crucial for implementing this system fail to cater to the multifaceted requirements of health care stakeholders.

    Related Works

    The FL framework ecosystem is characterized by its heterogeneity, prompted by contributions from both the open-source and industry communities, with each framework casted to address specific facets of FL’s inherent challenges. From enhancing security and privacy to ensuring scalability and promoting framework agnosticism to facilitating ease of use and customization, these frameworks collectively foster a conducive environment for the flourishing of FL applications.

    Industry solutions such as IBM FL [16], the Clara Training Framework [17], and Sherpa.ai [18] offer specialized features but suffer from restricted accessibility due to their proprietary nature, limiting widespread application and innovation. Open-source FL frameworks, despite their appeal of broader accessibility, grapple with challenges unique to their design and intended use cases. Syft, also known as PySyft [19], in the context of offering advanced privacy-enhancing technologies, focuses on secure and private data analysis across various domains. Despite it emphasizes structured transparency systems, the complex setup process associated with Syft could impede its broad adoption. As a pioneering framework in the realm of privacy technology, Syft has undergone substantial revisions across its iterations. While aimed at improvement, these continuous overhauls may pose challenges for early adopters in terms of staying abreast with the framework’s evolution.

    Frameworks such as FATE [20] and OpenFL [21] are distinguished by their strong emphasis on security and privacy, which are critical in sectors such as health care, where data confidentiality is essential. FATE is noted for its robust security and privacy features. However, it requires substantial infrastructural investment and has a steep learning curve, as there are some discrepancies in documentation clarity and availability across different parts of the framework. OpenFL also offers Intel-enabled trusted execution environment known as Safe Guard and mutual Transport Layer Security for all communications to ensure maximum privacy. However, implementing this framework on a broader scale is hindered and may necessitate specific hardware, software, or the activation of certain services [22]. TensorFlow Federated [23], while offering a specialized toolkit for TensorFlow models, limits its use to those committed to a singular modeling framework, potentially sidelining a broader audience that operates across diverse machine learning frameworks.

    Flower [24] and FedLab [24] are recognized for their framework agnosticism and scalability. Flower emerges as a user-friendly and lightweight option, albeit still progressing through its developmental stages with real-world applications yet to be fully demonstrated. This reflects a common developmental trajectory among open-source FL frameworks, where achieving a balance between accessibility, ease of use, and broad applicability poses a significant challenge.

    FedML [25] distinguishes itself through its versatility and recent architectural improvements, positioning it alongside frameworks such as Plato [26], known for supporting a diverse array of computing paradigms. This dual focus on facilitating FL research and the development of practical applications underscores its strengths. However, integrating significant updates may necessitate substantial maintenance for applications developed on its platform. Additionally, developers may need to reacquaint themselves with the framework’s updated features, presenting a potential challenge in adaptability and ongoing engagement.

    Innovative approaches are evident in PaddleFL [27] and Galaxy FL [28], with Galaxy FL pioneering blockchain technology integration for improved data privacy and model confidentiality. This innovation broadens the FL framework landscape, introducing novel data protection and utilization aspects. However, blending FL with blockchain remains a debated topic within the literature, especially concerning the significant computation and communication overheads [29].

    Lastly, FLSim [30] specializes in FL simulations, providing a comprehensive framework that incorporates differential privacy and secure aggregation scenarios. Similarly, Leaf [31] focuses primarily on simulation studies. While this specialization in simulation offers valuable insights, it may also narrow their applicability, potentially restricting their direct translation into real-world deployments.

    For a more in-depth discussion of the FL frameworks, we guide readers to Multimedia Appendix 1, where Table S1 in Multimedia Appendix 1 [32-51] offers a comprehensive summary of the highlights and key features of various FL frameworks. Additionally, Table S2 in Multimedia Appendix 1 details the developmental progress and recent advancements within these frameworks, providing a clear snapshot of their evolution and current state.

    Navigating the Path Forward: The Need for an Accessible Ecosystem for Clinical Research

    Beyond the challenges already covered, a recent review of FL in the biomedical domain reveals that only some research studies have provided reusable or reproducible source code alongside their research [52]. This highlights a common issue: the tendency to start a unique project for each separate data sharing initiative. This practice not only consumes valuable time and resources but also hinders the evolution of a universal ecosystem. Furthermore, the opportunity to build upon and contribute to an existing initiative is often overlooked.

    Despite recent progress in the field, there is a critical demand for a more inclusive, user-friendly, and holistic framework in health care collaborative research [53,54]. A framework that empowers stakeholders from diverse backgrounds to fully engage in collaborative learning opportunities is essential. This requires FL solutions characterized by intuitive interfaces, streamlined workflows, comprehensive documentation, and a design process that actively seeks and incorporates feedback from a wide range of stakeholders. By prioritizing usability and inclusivity in developing FL solutions, we can significantly lower technical barriers. This approach enables wider participation and amplifies the contributions of collaborative health care research.

    In this work, we present Federated Learning for Everyone (FL4E)—a dynamic, data-centric framework designed with inclusivity and adaptability at its core. FL4E is dedicated to efficiently converting RWD into actionable evidence, facilitating engagement from a broad spectrum of stakeholders, and enhancing collaborative learning experiences. Its hierarchical architecture offers a customizable data sharing ecosystem, facilitating a tunable balance between centralization and federation. With its modular approach, FL4E is general and provides an ecosystem-based environment that can support a wide range of collaborative research projects. This modularity further allows all stakeholders to contribute to the project by selectively engaging in the relevant part of the data sharing pipeline.

    Key Contributions

    • We propose a unique FL framework designed to cater to the diverse needs of health care stakeholders and to adapt to various study designs. This design is further enhanced by offering a tunable balance between centralization and federation.
    • We formalize this balance by introducing the concept of “degree of federation.”
    • Our evaluation of the framework, using 2 real-world data sets, underscores the value of federation. Specifically, hybrid experiments conducted at different degrees of federation deliver performance comparable to fully federated setups.

    Overview

    In this section, we rigorously investigate the fundamental principles underlying FL4E, systematically articulating its core concepts. This entails a meticulous examination of the pivotal modules and essential components that constitute FL4E, with a particular focus on their functionalities and interrelationships. Moreover, this section highlights the architectural nuances of FL4E, comprehensively detailing its structural configuration and the dynamics of its operational framework.

    Key Concepts of FL4E

    First, the concept of “degree of federation” within the FL4E framework introduces a novel, adaptable, and pragmatic approach to collaborative data analysis designed to meet stakeholders’ varied needs and limitations. This feature enables participants to customize their level of involvement and data sharing based on their specific infrastructural capacities, regulatory requirements, and privacy considerations.

    FL4E acknowledges that while federated models offer significant privacy advantages, their practicality may vary due to different constraints. Therefore, FL4E is designed to accommodate various levels of participation for stakeholders, ensuring flexibility if one data sharing stream proves unfeasible.

    Therefore, the choice between a centralized or federated model within FL4E is presented not as a dichotomy but as a continuum. On the one hand, centralization may simplify operations and enhance efficiency, but it could compromise privacy to some degree. Conversely, full federation prioritizes privacy, possibly adding operational and analytical complexity. FL4E’s design empowers stakeholders to find a balance that best suits their needs, governance frameworks, and regulatory environments.

    This adaptable strategy has proven effective in accommodating diverse data sharing needs, as demonstrated by the Global Data Sharing Initiative for Multiple Sclerosis and COVID-19 [55,56]. The deployment of a versatile hybrid data acquisition system, merging traditional data sharing with federated methods, has not only broadened inclusivity and flexibility but also led to a notable (3527/7757, 45.47%) increase in data collection volume compared to initial efforts [57]. These results underscore the “degree of federation” practical value, showcasing the advantages of integrating such adaptability into data acquisition and sharing frameworks. Different configurations within the degree of federation are depicted in Figure 1.

    ‎
    Figure 1. The degree of federation characterizes the balance between fully centralized and fully federated setups, leading to hybrid solutions where some of the stakeholders centralize their data while others prefer a federated approach. FL4E: Federated Learning for Everyone.

    However, it is essential to emphasize that the framework fosters regulatory compliance across all participation methods. Whether stakeholders prefer the simplicity and direct oversight of a more centralized model or the enhanced privacy of a federated approach, stakeholders are reminded of their collective obligation to adhere to prevailing data protection regulations.

    Second, ecosystem-based collaborative learning: FL4E framework introduces a paradigm shift from conventional project-specific FL to an “ecosystem-based” approach, underscoring a strategic pivot toward creating a unified, adaptable platform tailored for conducting various analyses centered on specific diseases or health conditions. At the heart of the “ecosystem-based” methodology is its holistic perspective, aiming to formulate a comprehensive and synergistic environment for data sharing and collaborative research.

    An ecosystem, in this context, is envisioned not just as a collection of data or series of disjointed projects but as a dynamic, collaborative network of stakeholders. This includes researchers, health care professionals, patients, and policy makers, all united by a common goal: understanding and addressing a particular health issue. By fostering such an ecosystem, FL4E encourages the pooling of diverse resources, expertise, and perspectives. This collaborative efficiency breaks down traditional silos, allowing for more efficient use of resources through shared data and findings.

    Moreover, encompassing a wide array of stakeholders and research questions enables the ecosystem to provide a more nuanced and comprehensive understanding of the health condition under investigation. The diversity of perspectives and expertise enriches the research outcomes, offering holistic insights far beyond what could be achieved in isolation.

    This approach is inherently adaptable and capable of evolving in response to new research findings, emerging health challenges, and shifts in the regulatory landscape. Such flexibility ensures the long-term sustainability and relevance of the ecosystem, enabling it to continue generating valuable insights over time.

    Regarding its structure, FL4E features a nested design. Within each ecosystem, several studies are carried out, each focusing on specific research questions related to the overarching theme. These studies, while operating under their unique regulatory and consent protocols, incorporate the degree of federation. They can include a data set, data quality scripts, and analyses tailored to their specific research queries.

    FL4E’s structure also incorporates simplicity and accessibility in its design. The intuitive user interface encourages contributions from stakeholders, fostering a collaborative environment that is integral to the ecosystem’s success. This systematic approach to collaboration and data sharing is visually depicted in Figure 2.

    The FL4E framework builds upon the foundational concepts of the Flower framework [32]. Flower’s flexibility, support for multinode execution, and agnosticism to both machine learning frameworks and programming languages significantly contribute to the overall adaptability and versatility of FL4E.

    ‎
    Figure 2. FL4E’s nested approach schema: the schema presents the ecosystem approach of the FL4E, facilitating the incorporation of diverse studies framed by different research questions. It further highlights the support for a wide range of analyses, leading to the execution of statistical experiments. FL4E: Federated Learning for Everyone.

    Modules and Functionality

    Overview

    This section provides an in-depth overview of the core modules within the FL4E framework: the Study Center, Repository Center, Model Center, and Data Center. Each module’s role and anticipated functionality are detailed, showcasing how the division into distinct centers contributes to effectively deploying the overarching ecosystem. Figure 3 complements this by visually representing the architectural arrangement of server and client components. By detailing the anticipated interactions among stakeholders and illustrating the operational flow within the framework, the architecture provides a solid foundation for understanding the potential and practical application of FL4E in advancing health care data-driven research and insights.

    ‎
    Figure 3. The high-level architecture of FL4E: This figure showcases a comprehensive framework designed to accommodate the nuanced interactions of diverse stakeholders. The architecture includes the expected user stories across 3 primary categories of participants: data providers, data scientists, and downstream users. Each stakeholder group engages with FL4E platform through distinct pathways. The architecture diagram features a table outlining the unique interactions of the stakeholders, mapping their respective roles and activities within the FL4E framework. This detailed mapping clearly explains how each stakeholder contributes to and benefits from the FL4E, highlighting the platform's versatility and user-centric design. At the core of our architecture lie 3 fundamental components: the server, client, and executor machine, each vital to the execution of FL tasks. The diagram we provide elucidates their interconnected roles, showcasing the seamless flow of data, scripts, and analytical results across the system. The server acts as the orchestrator for FL tasks, hosting the primary web application within a Docker container as an ASP.NET application. It securely manages the platform's data, housed on an SQL server in a dedicated hosting environment. This component is crucial for coordinating tasks and uses a Python environment to manage secure data sharing and preprocessing of “data center” module of the framework. On the client side, implemented as a Docker-based image, it runs on the data contributor's machine. This component is essential for integrating RWD into the FL process. Developed using Python and ASP.NET for web applications, the client-side component establishes a connection to the executer machine. On the other hand, the executor machine plays a crucial role in conducting the analysis. It is designed to receive client updates. This adaptable component configuration allows data scientists to tailor it according to their specific analytical needs and preferences. FL: federated learning; FL4E: Federated Learning for Everyone; RWD: real-world data.
    Study Center

    The Study Center is a critical piece in the ecosystem design of the framework. It coordinates research activities via the Study Catalogue and the Analysis Center, highlighting the interconnectivity and synergy within the system.

    The Study Catalogue functions as a cooperative repository where various stakeholders can share and access overarching information or metadata about different studies. Collaborations can be initiated by a data scientist with an intriguing research question but needing more data or by a data provider possessing the required data but lacking the analytical tools. Downstream users can further enrich the catalogue by contributing scientific articles or proposing research questions relevant to other stakeholders. Any new entry to the catalogue should include the title, a brief description, the study lead or author, and the current status of the entry (ie, active or not). If the entry involves any data, this should also be noted, as the data would be subsequently managed within the Data Center.

    In contrast, the Analysis Center is primarily designed for data scientists. Here, they can create an analysis entry derived from a study catalogue record. While doing so, they must furnish comprehensive details, including the exact analysis title, a descriptive summary, and the privacy status of the analysis (private or public). In a private setting, data scientists can selectively grant access to specific users (data providers). This means only authorized users can access these entries.

    In the FL4E framework, data scientists play a crucial role in orchestrating the analysis by providing essential scripts for client-side execution. Given that the FL4E backend is built upon the Flower framework, these scripts must align with this framework’s requirements. The machine learning–agnostic nature of this foundation offers a substantial flexibility to data scientists, allowing them to craft these scripts within the ambit of their expertise and the specific needs of the analysis. This flexibility is further augmented by Flower’s support for secure channels, enabling the addition of privacy-preserving configurations to the analysis execution. The integration of Flower’s capabilities within FL4E not only simplifies the technical implementation of FL projects but also empowers data scientists to innovate and experiment with various analytical models and privacy configurations. This approach ensures that FL4E remains a versatile and secure platform for FL, accommodating a wide range of research objectives and data privacy standards.

    To ensure transparency, the process of sharing scripts among the stakeholders is not automated. Clients may review and, if agreed, run these scripts locally using the FL4E-client component. In keeping with the principles of modularity and customization, the server does not include a built-in compiler to execute the analysis. Data scientists should disclose the IP and port address of the designated machine to data providers for analysis execution, enabling FL process participation. It is vital to note that no analysis should proceed in the Analysis Center without explicit consent from participating centers. If a particular study is associated with any data on central server, data scientists should incorporate the data as a centralized client into the federated analysis, effectively acting as an additional presumptive federated client in the data scientist’s machine.

    Repository Center

    The Repository Center serves as a warehouse for all scripts and models essential for stakeholders’ research and collaboration. It serves both as a repository to disseminate the scripts and models to the relevant stakeholders, and as an archive that stores past analyses and provides examples facilitating the creation of new studies. It seamlessly integrates with the Study Center and Model Center, enabling direct and organized access to all shared scripts and models across these centers.

    Model Center

    The Model Center is designed to facilitate the coordination and sharing of trained models. Similar to the Study Center, the Model Center is structured around 2 main elements: the Model Catalogue and the Model Repository.

    The Model Catalogue serves as a database for high-level information about various trained models. This can include the title, description, type of model, study lead, and status. The Model Catalogue is particularly useful for downstream users and data scientists who come across an interesting model and wish to share it within the ecosystem, or for those who want to share the outcome of an analysis conducted within the Study Center.

    Furthermore, the Model Repository is where detailed information about each trained model is stored. This includes the relationship of the model to a specific Model Catalogue entry or Analysis Center entry. Like the Study Center, the Model Center also adopts authorization layers to ensure that only authorized users can access certain models. A typical scenario is a trained model that would only be shared among the participants that contributed with data.

    However, if needed, the Model Center can also be used to disseminate a model more broadly. Indeed, users can attach trained models to the results of an analysis. This means that a user who may not have been able to join an analysis can still benefit from the trained models, download them, and test them on their local data. This is especially useful for data providers who want to validate models against their data sets or for data scientists who wish to use them as pretrained models in a similar domain.

    Furthermore, the Model Center allows data scientists to provide a decision support system as a result of a trained model. They can inject an HTML web page into the Model Repository entry, serving as a web-based dashboard. This user-friendly interface enables downstream users, even those without expert knowledge, to engage with the system effectively.

    Data Center

    The Data Center plays a crucial role in actualizing the concept of the “degree of federation” within the FL4E framework. Beyond providing data sharing capabilities, it also enables data integration and aggregation, meaning this module stores the data in a central server. Data providers and data scientists are central to the sharing process within the Data Center, using a meticulous 3-step routine to ensure data quality, along with effective data sharing and aggregation. The first step is (1) the raw data sharing scheme, followed by (2) the data cleaning and enhancement, and (3) the data sharing and aggregation.

    Raw Data Sharing Scheme

    The process initiates when the data provider shares a raw data scheme via a data dictionary, a static file describing the data structure. This dictionary must be attached to an existing study catalogue entry along with details like title, description, and data type (public or private). For private data, the provider must list authorized users.

    Data Cleaning and Enhancement Script

    Once the data scheme is established, data scientists must create a Python script for data cleaning and enhancement. This script ensures data quality by handling duplicates, missing values, and validating value ranges based on information of the data schema. This script should be uploaded to the Data Center and linked to the data dictionary.

    Data Sharing and Aggregation

    After setting up the data dictionary and cleaning script, data sharing can occur. Each data share triggers the cleaning script execution, with the outcomes appended to a core dictionary and each upload saved as a distinct file. This setup permits various data providers to share data in line with the data dictionary, with the script guaranteeing aggregation into a single file. Data scientists can then access this aggregated file for analytical use if access is granted to them.

    Implementation of the Framework

    The FL4E framework is divided into server-side and client-side implementations. Server-side, the framework is a microservice-based web application, designed using ASP.NET Core [58]. It adheres to the N-tier architecture and uses the Model-View-Controller design, along with a repository pattern, leading to a scalable and maintainable application. The application is segmented into 4 layers: Application, Model, Utility, and Data Access. Each layer performs distinct tasks, such as managing user authorization, defining data structures, handling requests, and interacting with the database. ASP.NET Core Identity module is used for authentication and authorization, while an admin control panel supervises user registrations in the interface layer of the framework [58].

    Client-side, the framework aims for simplicity and ease of use, also leveraging the ASP.NET Core framework. It incorporates 3 primary interfaces for data mounting, script uploading, and script execution. The data mounting interface facilitates user selection and mounting of data files; the script uploading interface enables users to upload scripts acquired from the server; and the script execution interface allows for the running of these scripts, using the Flower framework to secure the communication between the client and server.

    Ensuring the seamless operation and accessibility of the FL4E platform, both the client and server components are encapsulated as Docker containers. This strategic choice underpins the platform’s commitment to ease of deployment, consistency across different environments, and portability, thereby, significantly enhancing the user experience for participants in FL projects.

    A fully functional prototype of the FL4E server component has been successfully deployed to Microsoft Azure, showcasing the platform’s robustness [59]. This deployment not only demonstrates FL4E’s capability to leverage cloud infrastructure but also its readiness to support FL projects.

    To further acquaint potential users and stakeholders with FL4E’s functionalities and user interface, a comprehensive video demonstration has been made available [60]. This visual guide serves as an invaluable resource for understanding the platform’s operational flow, features, and the simplicity of initiating and participating in FL studies.

    For those interested in a deeper exploration or customization of FL4E, the source code for both the server and client components is readily accessible [61]. This openness ensures transparency, fosters a community of collaboration, and enables continuous improvement of the platform by allowing researchers, developers, and data scientists to contribute to its development.

    Ethical Considerations

    This study conducted secondary analyses of openly licensed repository data sets for demonstration purposes in the results section. No sensitive data were used for the cocreation of the framework. All data set sources are cited and adhere to open licensing agreements. For more detailed information about the data sets, please refer to the original sources.


    Data Sets

    We evaluated our framework using 2 real-world clinical data sets from FLamby [62]: Fed-Heart-Disease and Fed-TCGA-BRCA. The Fed-Heart-Disease data set contains 740 records from 4 centers, detailing 13 clinical features and a binary heart disease indicator for each patient. We used a logistic regression model for prediction. The Fed-TCGA-BRCA data set, sourced from The Cancer Genome Atlas’s Genomics Data Commons portal, has data from 1088 patients with breast cancer across 6 centers. With 39 clinical features and each patient’s time of death, we used a Cox model to predict the risk of death.

    Additionally, the Wisconsin Breast Cancer data set was incorporated into the GitHub repository as an extra use case [61,63]. It serves as a set of foundational scripts that can be generalized and act as guidelines for hybrid experiment settings in the FL4E framework.

    Degree of Federation Scenarios

    To demonstrate how FL4E adapts to various research questions, and how it modulates the degree of federation, we conducted 3 experiments using these data sets:

    Fully Federated Experiment

    In this scenario, every client participates in the FL process with each one contributing to the global model using their respective local data sets. This scenario aligns with the conventional FL setup. The data flow for this experiment is depicted in Figure 3.

    Hybrid Experiment

    In the hybrid setup, while all clients are part of the training process, some of them contribute their data centrally to the server. For instance, with the Fed-Heart-Disease data set, clients 1 and 2 were randomly selected to centrally contribute their data, while the rest participated as federated clients. For the Fed-TCGA-BRCA data set, clients 2, 3, and 5 joined as centralized clients.

    Centralized Experiment

    For this experiment, all clients send their data to the central server, and the model is trained centrally. This design aims to provide a benchmark for comparing the performance of the federated and hybrid approaches with the centralized model.

    Experimental Findings

    We evaluated our FL models using area under the receiver operating characteristic curve (ROC-AUC) and accuracy metrics for the Fed-Heart-Disease data set and the concordance index for the Fed-TCGA-BRCA data set. These metrics were chosen according to their ability to assess model performance in binary classification and survival analysis task, while accounting for potential class imbalance. For federated experiments, we implemented multiple FL strategies: FedAVG [14], FedOpt [64] (including FedAdagrad and FedYogi), and FedProx [65]. Hyperparameter tuning was performed using grid search on a centralized setting and applied to each experiment of the data sets. To ensure consistency, each experiment was repeated 5 times. The results of these experiments are summarized in Table 1.

    In our analysis of the Fed-Heart-Disease data set, as illustrated in Multimedia Appendix 2 and detailed in Table 1, notable behavior patterns emerge from clients 2 and 3. These patterns indicate the effects of class imbalance and quantity skew, suggesting that smaller clients with imbalanced data sets may benefit more from participating in an FL environment than operating in isolation. This insight is harmonious with the foundational principles of the FL paradigm, which aspires to harness the potential of diverse data sources to enhance model robustness and generalizability.

    To clarify the evaluation process, we introduce the notation where K represents the total number of clients participating in the FL process, and ni denotes the size of the data set for the i-th client, where i ranges from 1 to K. Furthermore, Ei signifies the evaluation metric (such as accuracy or ROC-AUC) obtained from assessing the global model by the i-th client. The sum of the data set sizes across all clients is represented by N, calculated as . Therefore, the overall evaluation metric E for the global FL model is calculated using the formula , offering a comprehensive measure of performance across the federated network.

    This approach significantly impacts the evaluation process by prioritizing the characteristics of individual clients over the generalizability of the FL model. This provides a more comprehensive understanding of model performance across diverse environments. However, the potential for skewness in federated evaluation, often stemming from imbalanced clients, arises from its reliance on a weighted average of individual client metrics. This introduces challenges, particularly when directly comparing it to centralized performance.

    It is important to note that for a direct comparison between federated and centralized evaluations, the ideal scenario involves assessing the FL model using a separate global test set on the server side. While this approach aligns more closely with the methodology used in centralized evaluations, it is not always feasible, especially in real-world settings where maintaining a separate test set on the server can be impractical due to data governance and privacy concerns.

    Contrary to initial expectations that centralized models would outperform others, our analysis unveiled that federated and hybrid models exhibit superior performance based on the ROC-AUC measure, even though the accuracy metrics might suggest a different narrative. To ensure the reliability and consistency of our findings, we have conducted rigorous comparisons with the original benchmark [62]. These comparisons demonstrate that our observations, especially concerning the performance metrics of centralized and federated models such as accuracy, are in line with existing research. This alignment reinforces the validity of our analysis despite the inherent challenges in conducting direct comparisons between federated and centralized performance evaluations [62].

    Similarly, the Fed-TCGA-BRCA study substantiates the advantage of FL. The heterogeneity of clients in the data set analysis, as highlighted by FLamby [62], indicates that this diversity significantly influences training performance, underscoring the potential benefits of FL.

    Notably, the hybrid experiment yields a crucial insight: it offers performance that matches fully federated models while significantly outperforming local settings for smaller clients. Importantly, this is achieved with considerably less complexity and overhead associated with managing federation across multiple data centers. This setup enables centralized management of in-house data, providing analysts with direct access (within regulatory boundaries) and flexibility for advanced analyses. By having data information at their disposal, analysts can develop more effective scripts, enabling advanced analyses with greater flexibility.

    Table 1. Experiments carried out using FL4Ea with the Fed-Heart-Disease and Fed-TCGA-BRCA data sets, demonstrating adaptability to varying degrees of federation.
    Experiment and parameterFed-Heart-Disease, mean (SD)Fed-TCGA-BRCA, mean (SD)

    		ROC-AUCbAccuracyC-indexc
    Fully federated

    		FedAvg0.846 (0.002)0.733 (0.007)0.732 (0.030)

    		FedAdagrad0.841 (0.029)0.726 (0.05)0.748 (0.016)

    		FedYogi0.803 (0.051)0.715 (0.032)0.745 (0.037)

    		FedProx0.846 (0.003)0.741 (0.006)0.725 (0.007)
    Hybrid experiment

    		FedAvg0.825 (0.004)0.740 (0.054)0.656 (0.06)

    		FedAdagrad0.821 (0.008)0.741 (0.005)0.776 (0.036)

    		FedYogi0.794 (0.013)0.710 (0.039)0.726 (0.041)

    		FedProx0.822 (0.004)0.737 (0.012)0.439 (0.227)
    Local experiment

    		Client 00.842 (0.009)0.753 (0.011)0.668 (0.064)

    		Client 10.882 (0.007)0.800 (0.013)0.445 (0.237)

    		Client 20.546 (0.271)0.550 (0.199)0.635 (0.166)

    		Client 30.542 (0.054)0.559 (0.096)0.570 (0.140)

    		Client 4—d—0.851 (0.078)

    		Client 5——0.666 (0.001)

    		Clients 1 and 20.819 (0.003)0.752 (0.008)—

    		Clients 2, 3, and 5——0.578 (0.125)
    Centralized0.812 (0.003)0.753 (0.007)0.609 (0.207)

    aFL4E: Federated Learning for Everyone.

    bROC-AUC: area under the receiver operating characteristic curve.

    cC-index: concordance index.

    dNot applicable.


    FL4E presents a novel approach to collaborative learning in a health care context, by innovatively balancing centralization and federation. This allows health care organizations to engage in FL according to their infrastructure and policy capabilities. Composed of the Study Center, Repository Center, Model Center, and Data Center, FL4E’s modular, ecosystem-oriented design caters to a wide variety of stakeholders, thereby facilitating multiparty clinical research collaborations.

    However, FL4E is not alone in promoting deployment ease and inclusivity in FL environments. FeatureCloud [66] advances FL by offering a platform that simplifies deployment across health care and research sectors. Aimed at democratizing multi-institutional analyses, FeatureCloud ensures that advanced programming skills or complex setups do not impede participation. The AI Store module, facilitating algorithm sharing and access, highlights FeatureCloud’s commitment to broadening FL adoption.

    In parallel, the emergence of UniFed [67] underscores a similar commitment to simplifying FL’s application by unifying diverse open-source frameworks. UniFed’s approach standardizes FL experimentation and deployment, offering a configuration-based, schema-enforced task specification that eases the distributed execution of FL projects. By supporting various FL frameworks, UniFed addresses the challenges of workflow diversity, data format discrepancies, and interpretation variations, thereby streamlining the FL experimentation process.

    What distinguishes FL4E in the evolving landscape is its focus on the “degree of federation” and the adaptability it brings, ensuring the framework is inclusive, adaptable, and can host hybrid experiments. Unlike FeatureCloud, which was developed from scratch, FL4E builds on the well-regarded Flower framework, providing a robust, accessible foundation and using this base, an adaptive, ecosystem-based model, designed to cater to diverse stakeholder preferences and requirements. FL4E extends the democratization vision of FL, incorporating a modular, customizable design that enhances engagement and ensures versatility across various applications. Additionally, FL4E adheres to key coding principles such as N-tier architecture and the repository pattern. These design choices promote ease of use, flexibility for modifications, and future-proofing the framework, further distinguishing FL4E by making it more accessible, maintainable, and adaptable to advancements in FL.

    As the FL field continues to mature, platforms such as FeatureCloud and UniFed play crucial roles in lowering technical barriers and fostering an environment conducive to collaborative innovation. Their contributions, along with FL4E’s nuanced understanding of stakeholder engagement, collectively propel the FL ecosystem toward a future where data-driven insights are accessible to a broader spectrum of the health care and research community, thereby amplifying the impact of FL in advancing health care outcomes and research endeavors.

    Nonetheless, FL4E still presents some limitations. Version control can pose challenges due to the Flower package dependency, and significant version changes may require internal updates. Moving from technical concerns to regulatory issues, ensuring adherence to regional data privacy regulations necessitates diligence from initiators and study leads, who must establish proper agreements for data sharing. Additionally, the framework’s dependence on manual communication among stakeholders could potentially impede its universal accessibility. This last point is particularly critical, as the system’s effective functioning heavily relies on the presence of a pre-established network of collaborators where members have a certain level of familiarity or knowledge of each other.

    Although FL4E and other FL systems offer potential technical solutions to collaborative learning in health care, the influence of societal barriers should not be underestimated. Hence, continuous education, stakeholder engagement, and proof-of-concept demonstrations within controlled environments are crucial for successful deployment.

    Future development path to FL4E, the demand for advanced privacy measures becomes increasingly apparent. While FL introduces a novel approach to leveraging patient-level data across decentralized sources, it is essential to recognize that this method is not without its vulnerabilities [68-70]. The process of sharing impersonal statistics during FL can inadvertently encode sensitive information, thus introducing potential risks and challenges to data privacy [68,71,72].

    The architecture of FL, which is designed to enhance privacy by eliminating the need for direct data sharing, is nevertheless susceptible to sophisticated attacks and privacy breaches. These vulnerabilities include inference attacks, where malicious entities may attempt to reconstruct personal data from model updates, as well as direct cyber threats that target the integrity and confidentiality of the FL process [68]. The body of literature on FL privacy highlights the critical importance of acknowledging these risks, with studies identifying various attack vectors that could potentially compromise the privacy guarantees of federated model sharing [72].

    Therefore, despite FL’s significant reduction in the risk of direct data exposure, it introduces new complexities in safeguarding against indirect data inference and cyberattacks. This nuanced understanding necessitates a cautious approach in the implementation of FL, particularly when handling sensitive health care information. The design of FL workflows, including the development of infrastructure and study plans, must incorporate robust security measures and privacy-preserving techniques.

    To this end, it is crucial to leverage the latest advancements in secure computing, differential privacy, and encryption technologies to strengthen the FL process against potential vulnerabilities [73-75]. Acknowledging these risks and adequately preparing for them is paramount, ensuring that the advantages of using decentralized data through FL do not come at the expense of patient privacy or data security. The implementation of industry-scale privacy measures is critically important in FL applications, as emphasized by recent studies [74]. However, these measures must be carefully balanced to ensure that data protection does not compromise the computational efficiency of FL analysis.

    To ease deployment, recent research in practical FL points toward promising techniques, such as soft computing’s fusion with FL [76] and the use of FL in cloud-edge collaborative architectures [77]. Such strategies can address concerns over data privacy, high communication costs, and data management difficulties, offering a promising direction for future investigations.

    While the manual steps (for instance, script exchanges) in FL4E are designed for transparency, automating these steps could improve the framework’s adaptability and usability. Additionally, transitioning from the current ASP.Net to Python could amplify the framework’s appeal, especially given Python’s strong foothold in the data science realm. As health care data expand in volume and complexity, frameworks such as FL4E, with further advancements and community involvement, stand to be vital tools in catalyzing transformative progress and collaboration in the domain.

    Acknowledgments

    YM, MO, and AP are funded by VLAIO PM: Augmenting Therapeutic Effectiveness through Novel Analytics (HBC.2019.2528); Research Council KU Leuven: Symbiosis 4 (C14/22/125) and Symbiosis3 (C14/18/092); and CELSA—Active Learning (CELSA/21/019). YM, MO and AP are affiliated to Leuven.AI and received funding from the Flemish Government (AI Research Program), FWO SBO (S003422N), and ELIXIR Belgium (I002819N). EDB was funded by an FWO-SB grant.

    Data Availability

    The data sets used for the analysis purposes of this study are available at the FLamby repository on GitHub [78]. FLamby serves as a benchmark for cross-silo data sets for federated learning applications, with a focus on health care application [62].

    Authors' Contributions

    EDB and YM contributed equally.

    Conflicts of Interest

    EDB is an employee of Genentech, Inc.

    Multimedia Appendix 1

    Landscape analysis: prominent frameworks for federated learning.

    DOCX File , 60 KB
    Multimedia Appendix 2

    Delineating client heterogeneity in real-world applications: exploration of class imbalance in the Fed-Heart-Disease analysis.

    PNG File , 103 KB
    1. Sherman RE, Anderson SA, Dal Pan GJ, Gray GW, Gross T, Hunter NL, et al. Real-world evidence - what is it and what can it tell us? N Engl J Med. 2016;375(23):2293-2297. [CrossRef] [Medline]
    2. Wu J, Roy J, Stewart WF. Prediction modeling using EHR data: challenges, strategies, and a comparison of machine learning approaches. Med Care. 2010;48(6 Suppl):S106-S113. [CrossRef] [Medline]
    3. Wilkinson MD, Dumontier M, Aalbersberg IJ, Appleton G, Axton M, Baak A, et al. The FAIR guiding principles for scientific data management and stewardship. Sci Data. 2016;3:160018. [FREE Full text] [CrossRef] [Medline]
    4. Jensen PB, Jensen LJ, Brunak S. Mining electronic health records: towards better research applications and clinical care. Nat Rev Genet. 2012;13(6):395-405. [CrossRef] [Medline]
    5. Weiskopf NG, Weng C. Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. J Am Med Inform Assoc. 2013;20(1):144-151. [FREE Full text] [CrossRef] [Medline]
    6. Kodra Y, Weinbach J, Posada-de-la-Paz M, Coi A, Lemonnier S, van Enckevort D, et al. Recommendations for improving the quality of rare disease registries. Int J Environ Res Public Health. 2018;15(8):1644. [FREE Full text] [CrossRef] [Medline]
    7. Moorthy V, Henao Restrepo AM, Preziosi M, Swaminathan S. Data sharing for novel coronavirus (COVID-19). Bull World Health Organ. 2020;98(3):150. [FREE Full text] [CrossRef] [Medline]
    8. Peeters LM, Parciak T, Walton C, Geys L, Moreau Y, De Brouwer E, et al. COVID-19 in people with multiple sclerosis: a global data sharing initiative. Mult Scler. 2020;26(10):1157-1162. [FREE Full text] [CrossRef] [Medline]
    9. Aitken M, de St Jorre J, Pagliari C, Jepson R, Cunningham-Burley S. Public responses to the sharing and linkage of health data for research purposes: a systematic review and thematic synthesis of qualitative studies. BMC Med Ethics. 2016;17(1):73. [FREE Full text] [CrossRef] [Medline]
    10. Holm S, Ploug T. Big data and health research-the governance challenges in a mixed data economy. J Bioeth Inq. 2017;14(4):515-525. [CrossRef] [Medline]
    11. Taruscio D, Mollo E, Gainotti S, Posada de la Paz M, Bianchi F, Vittozzi L. The EPIRARE proposal of a set of indicators and common data elements for the European platform for rare disease registration. Arch Public Health. 2014;72(1):35. [FREE Full text] [CrossRef] [Medline]
    12. Dreyer NA, Blackburn SC, Mt-Isa S, Richardson JL, Thomas S, Laursen M, et al. Direct-to-patient research: piloting a new approach to understanding drug safety during pregnancy. JMIR Public Health Surveill. 2015;1(2):e22. [FREE Full text] [CrossRef] [Medline]
    13. Kahn MG, Callahan TJ, Barnard J, Bauck AE, Brown J, Davidson BN, et al. A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data. EGEMS (Wash DC). 2016;4(1):1244. [FREE Full text] [CrossRef] [Medline]
    14. McMahan B, Moore E, Ramage D, Hampson S, Arcas BAY. Communication-efficient learning of deep networks from decentralized data. 2017. Presented at: 20th International Conference on Artificial Intelligence and Statistics; April 20-22, 2017:1273-1282; Fort Lauderdale, FL. URL: https://proceedings.mlr.press/v54/mcmahan17a.html
    15. Konečný J, McMahan HB, Yu FX, Richtárik P, Suresh AT, Bacon D. Federated learning: strategies for improving communication efficiency. arXiv. Preprint posted online on October 30, 2017. [CrossRef]
    16. Ludwig H, Baracaldo N, Thomas G, Zhou Y, Anwar A, Rajamoni S, et al. IBM Federated Learning: an enterprise framework white paper V0.1. arXiv. Preprint posted online on July 22, 2020. [CrossRef]
    17. Wen Y, Li W, Roth H, Dogra P. Federated Learning powered by NVIDIA Clara. NVIDIA Technical Blog. Dec 1, 2019. URL: https://developer.nvidia.com/blog/federated-learning-clara/ [accessed 2023-12-15]
    18. Rodríguez-Barroso N, Stipcich G, Jiménez-López D, Ruiz-Millán JA, Martínez-Cámara E, González-Seco G, et al. Federated learning and differential privacy: software tools analysis, the Sherpa.ai FL framework and methodological guidelines for preserving data privacy. Information Fusion. Dec 2020;64:270-292. [CrossRef]
    19. Ziller A. PySyft: a library for easy federated learning. Studies in Computational Intelligence. 2021;965:111-139.
    20. Yang L, Tao F, Tianjian C, Qian X, Qiang Y. FATE: an industrial grade platform for collaborative learning with data protection. J Mach Learn Res. 2021;22(226):1-6. [FREE Full text]
    21. Foley P, Sheller MJ, Edwards B, Pati S, Riviera W, Sharma M, et al. OpenFL: the open federated learning library. Phys Med Biol. 2022;67(21):214001. [FREE Full text] [CrossRef] [Medline]
    22. Try federated learning with OpenFL. Intel. URL: https:/​/www.​intel.com/​content/​www/​us/​en/​developer/​articles/​technical/​try-federated-learning-with-openfl.​html [accessed 2024-03-18]
    23. TensorFlow federated: machine learning on decentralized data. TensorFlow. URL: https://www.tensorflow.org/federated [accessed 2023-12-15]
    24. Zeng D, Liang S, Hu X, Wang H, Xu Z. FedLab: a flexible federated learning framework. arXiv. Preprint posted online on April 22, 2022. [CrossRef]
    25. He C, Li S, So J, Zeng X, Zhang M, Wang H, et al. FedML: a research library and benchmark for federated machine learning. arXiv. Preprint posted online on November 8, 2020. [CrossRef]
    26. Li B, Su N, Ying C, Wang F. Plato: an open-source research framework for production federated learning. 2023. Presented at: ACM TURC '23: ACM Turing Award Celebration Conference - China 2023; July 28-30, 2023:1-2; Wuhan, China. [CrossRef]
    27. PaddlePaddle/PaddleFL: federated deep learning in PaddlePaddle. GitHub. URL: https://github.com/PaddlePaddle/PaddleFL [accessed 2024-03-18]
    28. Liu L, Wu C, Xiao J. Blockchain-based platform for distribution AI. EasyChai. Preprint posted online on February 2, 2019. [CrossRef]
    29. Qu Y, Uddin MP, Gan C, Xiang Y, Gao L, Yearwood J. Blockchain-enabled federated learning: a survey. ACM Comput Surv. Nov 21, 2022;55(4):1-35. [CrossRef]
    30. Li L, Wang J, Xu CZ. FLSim: an extensible and reusable simulation framework for federated learning. Social-Informatics and Telecommunications Engineering. 2021;369:350-369. [CrossRef]
    31. Caldas S, Duddu SMK, Wu P, Li T, Konečný J, McMahan HB, et al. LEAF: a benchmark for federated settings. arXiv. Preprint posted online on December 9, 2019. [CrossRef]
    32. Beutel DJ, Topal T, Mathur A, Qiu X, Fernandez-Marques J, Gao Y, et al. Flower: a friendly federated learning research framework. arXiv. Preprint posted online on March 5, 2022. [CrossRef]
    33. PySyft. OpenMined Blog. URL: https://blog.openmined.org/tag/pysyft/ [accessed 2024-03-18]
    34. Ziller A, Trask A, Lopardo A, Szymkow B, Wagner B, Bluemke E, et al. PySyft: a library for easy federated learning. In: Rehman MH, Gaber MM, editors. Federated Learning Systems. Studies in Computational Intelligence, vol 965. Cham, Switzerland. Springer; Jun 12, 2021:111-139.
    35. FATE. FedAI. URL: https://fate.fedai.org/ [accessed 2024-03-18]
    36. FederatedAI/FATE-Community. GitHub. URL: https://github.com/FederatedAI/FATE-Community/blob/master/FATE_Project_Technical_Charter.pdf [accessed 2024-03-18]
    37. FederatedAI/FATE. GitHub. URL: https://github.com/FederatedAI/FATE [accessed 2024-03-18]
    38. He C, Li S, So J, Zhang M, Zeng X, Wang H, et al. FedML: a research library and benchmark for federated machine learning. MIT Media Lab. Dec 12, 2020. URL: https:/​/www.​media.mit.edu/​publications/​fedml-a-research-library-and-benchmark-for-federated-machine-learning/​ [accessed 2024-03-18]
    39. The production AI platform for federated learning at scale. FedML. URL: https://www.fedml.ai/home [accessed 2024-03-18]
    40. Flower: a friendly federated learning framework. Flower. URL: https://flower.ai/ [accessed 2024-03-18]
    41. alibaba/FederatedScope. GitHub. URL: https://github.com/alibaba/FederatedScope [accessed 2024-04-24]
    42. OpenFL: an open source framework for federated learning. OpenFL. URL: https://openfl.io/ [accessed 2024-03-18]
    43. SMILELab-FL/FedLab. GitHub. URL: https://github.com/SMILELab-FL/FedLab [accessed 2024-03-18]
    44. Stripelis D, Anastasiou C, Toral P, Asghar A, Ambite JL. arXiv. Preprint posted online on November 13, 2023. [CrossRef]
    45. Roth HR, Cheng Y, Wen Y, Yang I, Xu Z, Hsieh YT, et al. NVIDIA FLARE: federated learning from simulation to real-world. arXiv. Preprint posted online on April 28, 2023. [CrossRef]
    46. PFL: Paddle federated learning. PaddleFL. URL: https://paddlefl.readthedocs.io/en/latest/ [accessed 2024-03-18]
    47. Lai F, Dai Y, Singapuram S, Liu J, Zhu X, Madhyastha H, et al. FedScale: benchmarking model and system performance of federated learning at scale. arXiv. Preprint posted online on June 18, 2022. [FREE Full text] [CrossRef]
    48. TL-System/plato. GitHub. URL: https://github.com/TL-System/plato [accessed 2024-03-18]
    49. Zhuang W, Gan X, Wen Y, Zhang S. EasyFL: a low-code federated learning platform for dummies. IEEE Internet Things J. 2022;9(15):13740-13754. [CrossRef]
    50. GalaxyLearning/GFL. GitHub. URL: https://github.com/GalaxyLearning/GFL [accessed 2024-03-18]
    51. facebookresearch/FLSim. GitHub. URL: https://github.com/facebookresearch/FLSim [accessed 2024-03-18]
    52. Crowson MG, Moukheiber D, Arévalo AR, Lam BD, Mantena S, Rana A, et al. A systematic review of federated learning applications for biomedical data. PLOS Digit Health. May 2022;1(5):e0000033. [FREE Full text] [CrossRef] [Medline]
    53. Rudrapatna VA, Butte AJ. Opportunities and challenges in using real-world data for health care. J Clin Invest. 2020;130(2):565-574. [FREE Full text] [CrossRef] [Medline]
    54. Hiramatsu K, Barrett A, Miyata Y, PhRMA Japan Medical Affairs Committee Working Group 1. Current status, challenges, and future perspectives of real-world data and real-world evidence in Japan. Drugs Real World Outcomes. 2021;8(4):459-480. [FREE Full text] [CrossRef] [Medline]
    55. Simpson-Yap S, De Brouwer E, Kalincik T, Rijke N, Hillert JA, Walton C, et al. Associations of disease-modifying therapies with COVID-19 severity in multiple sclerosis. Neurology. 2021;97(19):e1870-e1885. [CrossRef] [Medline]
    56. Simpson-Yap S, Pirmani A, Kalincik T, De Brouwer E, Geys L, Parciak T, et al. Updated results of the COVID-19 in MS Global Data Sharing Initiative: anti-CD20 and other risk factors associated with COVID-19 severity. Neurol Neuroimmunol Neuroinflamm. Aug 29, 2022;9(6):e200021. [FREE Full text] [CrossRef] [Medline]
    57. Pirmani A, de Brouwer E, Geys L, Parciak T, Moreau Y, Peeters LM. The journey of data within a global data sharing initiative: a federated 3-layer data analysis pipeline to scale up multiple sclerosis research. JMIR Med Inform. 2023;11:e48030. [FREE Full text] [CrossRef] [Medline]
    58. ASP.NET Core. Microsoft. URL: https://dotnet.microsoft.com/en-us/apps/aspnet [accessed 2024-12-30]
    59. FL4E-working prototype. Azure. URL: https://fl-server.azurewebsites.net/ [accessed 2024-12-30]
    60. Federated learning for everyone. Ashkan Pirmani YouTube page. May 14, 2024. URL: https://www.youtube.com/watch?v=K8n7kffpvQ0&ab_channel=AshkanPirmani [accessed 2024-05-14]
    61. ashkan-pirmani/FL4E-Analysis. GitHub. URL: https://github.com/ashkan-pirmani/FL4E-Analysis [accessed 2024-12-15]
    62. du Terrail JO, Ayed SS, Cyffers E, Grimberg F, He C, Loeb R, et al. FLamby: datasets and benchmarks for cross-silo federated learning in realistic healthcare settings. arXiv. Preprint posted online on May 5, 2023. [CrossRef]
    63. Breast Cancer Wisconsin (Prognostic). UC Irvine Machine Learning Repository. URL: https://archive.ics.uci.edu/dataset/16/breast+cancer+wisconsin+prognostic [accessed 2024-12-15]
    64. Reddi S, Charles Z, Zaheer M, Garrett Z, Rush K, Konečný J, et al. Adaptive federated optimization. arXiv. Preprint posted online on September 8, 2021. [CrossRef]
    65. Li T, Sahu AK, Zaheer M, Sanjabi M, Talwalkar A, Smith V. Federated optimization in heterogeneous networks. arXiv. Preprint posted online on April 21, 2020. [CrossRef]
    66. Matschinske J, Späth J, Bakhtiari M, Probul N, Kazemi Majdabadi MM, Nasirigerdeh R, et al. The FeatureCloud platform for federated learning in biomedicine: unified approach. J Med Internet Res. 2023;25:e42621. [FREE Full text] [CrossRef] [Medline]
    67. Liu X, Shi T, Xie C, Li Q, Hu K, Kim H, et al. UniFed: all-in-one federated learning platform to unify open-source frameworks. arXiv. Preprint posted online on December 30, 2023. [FREE Full text] [CrossRef]
    68. Kairouz P, McMahan HB, Avent B, Bellet A, Bennis M, Nitin Bhagoji A, et al. Advances and Open Problems in Federated Learning. Foundations and Trends in Machine Learning. 2021;14(1–2):1-210. [FREE Full text] [CrossRef]
    69. Nasirigerdeh R, Torkzadehmahani R, Baumbach J, Blumenthal DB. On the privacy of federated pipelines. 2021. Presented at: SIGIR 2021 - the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval; July 11-15, 2021:1975-1979; Virtual event, Canada. [CrossRef]
    70. Hartebrodt A, Röttger R, Blumenthal DB. Federated singular value decomposition for high-dimensional data. Data Min Knowl Disc. 2023;38(3):938-975. [CrossRef]
    71. Tolpegin V, Truex S, Gursoy ME, Liu L. Data poisoning attacks against federated learning systems. Berlin, Heidelberg.; 2020. Presented at: 25th European Symposium on Research in Computer Security, ESORICS 2020; September 14-18, 2020:480-501; Guildford, United Kingdom. [CrossRef]
    72. Bagdasaryan E, Veit A, Hua Y, Estrin D, Shmatikov V. How to backdoor federated learning. arXiv. Preprint posted online on August 6, 2019. [CrossRef]
    73. Dwork C. Differential privacy. 2006. Presented at: 33rd International Colloquium, ICALP 2006; July 10-14, 2006:1-12; Venice, Italy. [CrossRef]
    74. Park J, Lim H. Privacy-preserving federated learning using homomorphic encryption. Applied Sciences. 2022;12(2):734. [CrossRef]
    75. Wood A, Najarian K, Kahrobaei D. Homomorphic encryption for machine learning in medicine and bioinformatics. ACM Comput Surv. 2020;53(4):1-35. [CrossRef]
    76. Supriya Y, Gadekallu TR. A survey on soft computing techniques for federated learning- applications, challenges and future directions. J. Data and Information Quality. 2023;15(2):1-28. [CrossRef]
    77. Huang Y, Yang X, Guo J, Cheng J, Qu H, Ma J, et al. A high-precision method for 100-day-old classification of chickens in edge computing scenarios based on federated computing. Animals (Basel). 2022;12(24):3450. [FREE Full text] [CrossRef] [Medline]
    78. owkin/FLamby. GitHub. URL: https://github.com/owkin/FLamby [accessed 2024-05-14]

    ‎
    FL: federated learning
    FL4E: Federated Learning for Everyone
    ROC-AUC: area under the receiver operating characteristic curve
    RWD: real-world data

    Edited by A Mavragani; submitted 15.12.23; peer-reviewed by D Blumenthal, X Zhang; comments to author 23.02.24; revised version received 25.04.24; accepted 15.05.24; published 17.07.24.

    Copyright

    ©Ashkan Pirmani, Martijn Oldenhof, Liesbet M Peeters, Edward De Brouwer, Yves Moreau. Originally published in JMIR Formative Research (https://formative.jmir.org), 17.07.2024.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.