This paper is in the following e-collection/theme issue:

Formative Evaluation of Digital Health Interventions (2410) Design and Evaluation of Medical Education Materials (30) Innovations in Cancer Diagnostic and Decision Support (58) Clinical Cancer Research (83) Computer-Aided Diagnosis (128) e-Learning and Medical Education (691) Tools, Programs and Algorithms (266)

Published on in Vol 5, No 3 (2021): March

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/19408, first published .
e-Learning for Instruction and to Improve Reproducibility of Scoring Tumor-Stroma Ratio in Colon Carcinoma: Performance and Reproducibility Assessment in the UNITED Study

e-Learning for Instruction and to Improve Reproducibility of Scoring Tumor-Stroma Ratio in Colon Carcinoma: Performance and Reproducibility Assessment in the UNITED Study

e-Learning for Instruction and to Improve Reproducibility of Scoring Tumor-Stroma Ratio in Colon Carcinoma: Performance and Reproducibility Assessment in the UNITED Study

Authors of this article:

Marloes A Smit1 Author Orcid Image ;   Gabi W van Pelt1 Author Orcid Image ;   Elisabeth MC Dequeker2 Author Orcid Image ;   Raed Al Dieri3 Author Orcid Image ;   Rob AEM Tollenaar1 Author Orcid Image ;   J Han JM van Krieken4 Author Orcid Image ;   Wilma E Mesker1 Author Orcid Image ;   UNITED Group1
Article Authors Cited by (11) Tweetations (1) Metrics

    Original Paper

    1Department of Surgery, Leiden University Medical Center, Leiden, Netherlands

    2Department of Public Health and Primary Care, Biomedical Quality Assurance Research Unit, University of Leuven, Leuven, Belgium

    3European Society of Pathology, Brussels, Belgium

    4Department of Pathology, Radboud University Medical Center, Nijmegen, Netherlands

    Corresponding Author:

    Wilma E Mesker, PhD

    Department of Surgery

    Leiden University Medical Center

    Albinusdreef 2

    Leiden, 2333 ZA

    Netherlands

    Phone: 31 715264005

    Email: w.e.mesker@lumc.nl


    Background: The amount of stroma in the primary tumor is an important prognostic parameter. The tumor-stroma ratio (TSR) was previously validated by international research groups as a robust parameter with good interobserver agreement.

    Objective: The Uniform Noting for International Application of the Tumor-Stroma Ratio as an Easy Diagnostic Tool (UNITED) study was developed to bring the TSR to clinical implementation. As part of the study, an e-Learning module was constructed to confirm the reproducibility of scoring the TSR after proper instruction.

    Methods: The e-Learning module consists of an autoinstruction for TSR determination (instruction video or written protocol) and three sets of 40 cases (training, test, and repetition sets). Scoring the TSR is performed on hematoxylin and eosin–stained sections and takes only 1-2 minutes. Cases are considered stroma-low if the amount of stroma is ≤50%, whereas a stroma-high case is defined as >50% stroma. Inter- and intraobserver agreements were determined based on the Cohen κ score after each set to evaluate the reproducibility.

    Results: Pathologists and pathology residents (N=63) with special interest in colorectal cancer participated in the e-Learning. Forty-nine participants started the e-Learning and 31 (63%) finished the whole cycle (3 sets). A significant improvement was observed from the training set to the test set; the median κ score improved from 0.72 to 0.77 (P=.002).

    Conclusions: e-Learning is an effective method to instruct pathologists and pathology residents for scoring the TSR. The reliability of scoring improved from the training to the test set and did not fall back with the repetition set, confirming the reproducibility of the TSR scoring method.

    Trial Registration: The Netherlands Trial Registry NTR7270; https://www.trialregister.nl/trial/7072

    International Registered Report Identifier (IRRID): RR2-10.2196/13464

    JMIR Form Res 2021;5(3):e19408

    doi:10.2196/19408

    Keywords

    colon cancer; tumor-stroma ratio; validation; e-Learning; reproducibility study; cancer; tumor; colon; reproducibility; carcinoma; prognosis; diagnostic; implementation; online learning 


    The prediction of disease outcome for an individual patient as part of personalized medicine is becoming routine practice in the management of cancer patient treatment. Staging of colon cancer by pathologists is based on hematoxylin and eosin (H&E)-stained sections of the primary tumor. The tumor-node-metastasis (TNM) classification is used as the main selection criterion for additional treatment, along with noting of characteristics such as depth of invasion and differentiation grade [1], according to the American Joint Committee staging algorithm. However, conventional H&E sections provide more information than previously recognized.

    In the last decade, research has not only focused on the tumor and its characteristics but increasingly also on the tumor microenvironment. The tumor microenvironment consists of the stromal background with a variety of cells such as fibroblasts, endothelial cells, and lymphocytes. The tumor-stroma ratio (TSR) is the amount of tumor relative to the amount of stroma in the primary tumor [2-4]. Patients with a high amount of stroma (stroma-high) have a worse prognosis compared to those harboring tumors with a low amount of stroma (stroma-low) in multiple types of cancer [5-13].

    Scoring the TSR is performed on H&E-stained sections in only 1-2 minutes, with good to excellent interobserver agreement [3]. This implies that TSR scoring is easy to learn and useful in daily practice.

    The Uniform Noting for International Application of the Tumor-Stroma Ratio as an Easy Diagnostic Tool (UNITED) study was developed to prepare for implementation of the TSR as an additional high-risk indicator along with traditional TNM classification. As part of the UNITED study, an instruction protocol and reproducibility study were initiated using an e-Learning module as described in the published research protocol [14].

    Digital pathology is increasingly being implemented in daily diagnostic practice as well as for teaching. Digital pathology for instruction, most commonly used for instruction of students, has multiple advantages: more students can be reached because it is web-based; all students look at exactly the same case; annotations can be shared with the teacher, resulting in direct feedback; and students can complete the course when and where it suits them [15-18].

    The use of e-Learning for education has been adopted in different medical specialties worldwide. An example of e-Learning used in pathology is a module developed for Dutch pathologists [19,20]. The module focuses on decreasing the variation in grading dysplasia in adenomas and increasing the consistency of scoring serrated lesions. Two separate studies have shown that e-Learning is a good method to improve performance [19,20].

    e-Learning to instruct professionals has also been confirmed in specialties other than pathology [21,22]. For instance, based on a systematic review for surgical training (students, residents, and surgeons), Maertens et al [23] concluded that e-Learning is as effective as other methods for training.

    The aim of this study was to confirm the high reproducibility of scoring the TSR using an e-Learning module to train a variety of pathologists.


    Case Selection

    The e-Learning module was based on H&E-stained sections of stage II and III colon cancer resection specimens. The cases were randomly selected from the archives of the Pathology Department of Leiden University Medical Centre (LUMC). The number of cases with very low stroma (ie, 10% or 20% stroma area) were limited from the analysis to increase the number of cases that are generally more difficult to score for pathologists and are therefore more suitable for training purposes. In the e-Learning, 55% of the cases were stroma-low (≤50% stroma) and 45% were stroma-high (>50% stroma). None of the patients had received neoadjuvant treatment at the time of sample collection. The sample size was based on a workable amount of cases to maintain quality without the case load being too high.

    Slides were scanned using a 20× objective with the Panoramic 250 scanner (3D Histech) or with the IntelliSite Digital pathology slide scanner (Philips).

    Participants

    Pathologists and pathology residents from all over the world could participate in the UNITED study and in the e-Learning. The UNITED study started in 2018, and pathologists were invited to start (and complete) the e-Learning in the period of December 2017 to April 2019. Data collection ended in April 2019. Multimedia Appendix 1 provides an overview of participating countries, and the numbers of participating pathologists and residents.

    TSR Scoring Method

    The previously published protocol for scoring the TSR was used in this study [3,4]. In brief, the section of the deepest part of the tumor, usually the section used for determining the T-stage, was chosen. The area with the highest amount of stroma was selected and scored at 100× magnification in increments of 10%. A field should contain tumor cells on four opposite edges of the field of evaluation.

    e-Learning

    The e-Learning module was developed in PathXL Tutor version 6.1.1.1. (Philips, Belfast, UK). This software uses digital images, and was developed to easily share and teach a network of pathology students, or in this case pathologists. The PathXL software allows the pathologist to analyze the slide in a manner comparable to using a microscope. The e-Learning was prepared to resemble real-life microscopy as far as possible by using round annotations with a fixed size of 3.4 mm2. This represents the size of the field of vision of microscopes, even from different brands, when using 100× magnification.

    Participants were blinded to clinical data of the sections and were only informed that the patients did not receive neoadjuvant treatment.

    Before starting the first set of the e-Learning module, participants were asked to watch the instruction video on the study website [24]. The training set consisted of 40 cases. This set started with 5 multiple choice questions where annotations were placed upfront in different areas of the section (Multimedia Appendix 2). Participants were asked to select the correctly placed annotation and determine the percentage of stroma. In the other 35 cases (and in the other sets), the participant was asked to place the annotations themselves at the most optimal position and to determine the stroma percentage.

    The test set also consisted of 40 cases, including 37 (93%) new cases. To determine the intraobserver agreement of scoring the TSR, the test set was repeated (repetition set) after a 2-month washout period with the same cases placed in a different order.

    The answer model used for evaluating the results was established by two experienced observers (GP, MS) together with a pathologist (VS) of the LUMC. The coordinators of the UNITED study checked all finished e-Learning sets for stroma percentage and for correct placement of the annotation. The answers of the participants were compared with the answer model. Continuing with the second set was allowed when an interobserver agreement (κ) of ≥0.7 with predefined scores was reached. In the case in which a participant did not pass a set due to a κ score below 0.7, the same set had to be rescored (see Multimedia Appendix 3 for the flowchart of the e-Learning module [14]) after feedback was given.

    Statistical Analysis

    Data collected in this study comprised: (1) whether the participant is a pathologist or resident, (2) participating country, (3) the stroma percentage of the different questions, and (4) whether or not the participant considered a question to be difficult. In this study, a possible bias could be that participating pathologists/residents are generally more motivated for participation.

    Stroma percentages were classified as stroma-low (≤50% stroma) or as stroma-high (>50% stroma) [3,4,25]. This dichotomous output and the outcome of whether or not the annotation was placed correctly were used for measuring observer agreement. Cohen κ coefficient was used to measure inter- and intraobserver agreement. This score is quantitative and was used as a noncontinuous variable. Histograms were used to visualize the distribution of the κ scores for each set.

    Nonnormally distributed continuous variables are described with the median and range (minimum and maximum values). For the median κ scores of a set, the first κ score of a participant of each set was used. The Wilcoxon signed-rank test was used to compare paired nonnormally distributed continuous variables (eg, measuring the progress between different e-Learning sets by participants).

    Ethical Considerations

    The UNITED study protocol has been approved by the Medical Research Ethics Committee of the LUMC (study number p17.302). All samples were handled in accordance with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards.


    Participants

    In total, 63 participants (49 pathologists and 14 residents) were registered for e-Learning. However, 14 participants were nonresponsive after registration, and thus 36 pathologists and 13 residents from 19 countries started the e-Learning; these 49 participants were used for analysis. All participating pathologists had gastrointestinal pathology as a subspecialty; however, most of them had more than one subspecialty. The residents had not yet chosen a pathology subspecialty. Thirty-six (73%) participants (28 pathologists and 8 residents) finished the training set and continued with the test set. In total, 31 (63%) participants finished the whole cycle (3 sets) of the e-Learning. A complete overview and reasons for participants to drop out are shown in Figure 1. The participants who quit the study were left out of the analysis.

    ‎
    Figure 1. Overview of the number of participants of the e-Learning and the reasons for not finishing the e-Learning cycle.
    View this figure

    Observer Agreement

    After finishing the training set of the e-Learning, the observer agreement was determined. Twenty-four (67%) participants (19 pathologists and 5 residents) passed the training set at the first attempt (ie, κ≥0.7). Two participants (both residents) needed a third chance to pass the training set. The median κ score for the training set was 0.72. The test set was passed the first time by 31 (91%) participants (23 pathologists and 8 residents) and 3 (9%) pathologists had to repeat the test set. The median κ score for the test set was 0.77. After a 2-month washout period, 28 (90%) participants (21 pathologists and 7 residents) directly passed the repetition set, with a median κ score of 0.76 (Table 1, Figure 2). A significant improvement was observed from the training set to the test set (P=.002). No significant changes of the κ scores were observed between the test set and the repetition set (P=.30, Figure 3). Intraobserver agreement was measured for the 31 participants who finished the repetition set. The median κ score of the intraobserver agreement was 0.77 (Table 1). Scoring results from pathologists showed significant improvement from the training set to the test set (P=.006) and no fall back (P=.74) after a washout period of 2 months. The scoring results of the residents showed no significant changes between sets (P=.26 and P=.13). Details are shown in Multimedia Appendix 4.

    Table 1. Distribution of κ scores per set.
    SetCompleted set, Nκ, median (range)
    Training set360.72 (0.21-0.90)
    Test set340.77 (0.51-0.97)
    Repetition set310.76 (0.60-0.89)
    Intraobserver agreement310.77 (0.61-0.95)
    ‎
    Figure 2. Distribution of κ scores for each set of the e-Learning: (A) training set, (B) test set, (C) repetition set, and (D) intraobserver agreement.
    View this figure
    ‎
    Figure 3. Progress of κ scores per participant during the e-Learning module.
    View this figure

    Difficulty of the Questions

    For each case, participants were asked whether scoring the TSR of the section was easy, normal, or difficult. If answered as difficult, a reason could be given. A case was classified as difficult when at least 40% of the participants agreed with this assessment. Eleven cases (9 different cases) were classified as difficult (4 in the training set, 2 in the test set, and 5 in the repetition set). The cases classified as difficult in the test set remained difficult in the repetition set. The three other cases of the repetition set were more difficult than average in the test set. As expected, most of the difficult cases were those close to the cut-off value of 50%, mucinous tumors, tumors with a lot of necrosis, and tumors in which the distinction between the stroma and the smooth muscle was difficult (see Multimedia Appendix 5 for examples of difficult cases). Overall, the 11 cases were more often answered wrong (29% of the answers) by participants who classified (one of) these cases as difficult compared to 19% of wrong answers for cases that were assessed as not difficult (see Multimedia Appendix 6 for the subdivision per case).

    Drawing Annotations

    A few cases were used in both the training and test sets. Analysis of these cases showed progress of scoring at the hotspot, as the annotations were more centered (Figure 4). Furthermore, in stroma-low cases, annotations were more widely spread over the entire tumor area, whereas one or two hotspots were more often identified in stroma-high cases (Figure 5).

    ‎
    Figure 4. Example of improvement of selecting the most optimal area for scoring the tumor-stroma ratio. Diffuse spread of annotations was seen in the training set (A) compared to less variation of the scoring area in the test set (B).
    View this figure
    ‎
    Figure 5. Examples of the distribution of annotations. In stroma-low cases (A), a wide distribution was more frequent, whereas in stroma-high cases, a hotspot (or sometimes more) formed (B).
    View this figure

    This study shows that e-Learning is an accurate method to instruct pathologists and pathology residents for scoring the TSR. A κ score >0.7 was used to define the reproducibility of the method. The median κ score improved and the minimum value increased after completing the consecutive sets for scoring the TSR. Significant progress (P=.002) was observed from the training to the test set. If a set was not passed the first time, feedback was given to the participant before they repeated the same set; however, participants did not get insight into their precise mistakes. The feedback was personalized, but not case-specific.

    A decrease was observed between the number of created accounts, the number of participants who started the e-Learning, and the number of participants who completed the whole module. There are multiple reasons for this decrease. Most of the participants were registered by the principal investigator of an institute; however, not everyone responded to their registration. Another reason for dropout was withdrawal of the center from the UNITED study.

    In daily diagnostics, H&E-stained sections are used for determining cancer stage. To ensure the high quality of the sections used for e-Learning, the original H&E-stained slides of the cases used for diagnostic purposes were scanned. Furthermore, all slides were scanned at the same magnification to avoid differences in the quality of the digital images.

    When reviewing the e-Learning sets, in some cases it seemed as if the participants had scored the tumor percentage instead of the stroma percentage. This might be explained by the fact that pathologists are accustomed to scoring the neoplastic cell percentage for molecular analysis. Another explanation might be more related to semantics. The amount of tumor is in the numerator, which might be a source of confusion. In these doubtful cases, the participant was asked to reevaluate the case as well as some others. Thus, scoring the percentage of stroma remains a point of attention.

    Overall, participants were well able to choose the right area for scoring the stroma percentage and to estimate whether a section was stroma-high or stroma-low. A common misinterpretation was scoring at the invasive front instead of looking for an area with as much stroma as possible within the section. This might be explained by the fact that pathologists are accustomed to scoring the tumor budding at the invasive front [26]. Furthermore, as the scoring protocol describes the use of the section from the deepest part of the tumor (usually the section used for determining the T-status) [3], the distinction between scoring the TSR in the whole tumor area and not necessarily at the invasive front might not have been made clear enough in the instructions. When comparing the results of pathologists and residents, pathologists showed significant improvement from the training to the test set, whereas residents did not. A possible explanation is the small group of residents (n=8) or the fact that pathologists are more experienced than residents.

    Research performed on other pathology biomarkers used in daily practice such as lymphovascular invasion [27], tumor grading [28,29], classification and grading of colorectal polyps or adenomas [19,20,30-33], and the estimation of tumor cell percentage [34] has shown weak to moderate interobserver agreement. With three median κ scores above 0.7, the interobserver agreement for scoring the TSR in this study was found to be good. Although no comparison arm was included in this study, the median κ values obtained in this study are lower than those reported previously. This can be explained by the fact that the scores were low in the training set, which improved in the test set.

    Digital pathology is increasingly entering pathology practice, although most pathology departments are not yet (fully) digitalized. In the future, a digital image analysis program will be useful for more accurate scoring and even better reproducibility. Digital pathology for teaching goals has some advantages and disadvantages. The advantages include easy distribution of samples and being able to score a set at the same time while reaching a worldwide group of pathologists, and disadvantages include possible software flaws or bugs when using a digitized workflow. In this study, the placed annotations were not always saved correctly, which sometimes made it difficult to analyze the results for a participant. In these particular cases, the participant was asked to reevaluate the case.

    In conclusion, this study showed that e-Learning is a good and effective method to instruct pathologists and residents in scoring the TSR. Furthermore, this study confirmed the reproducibility of the scoring method.

    Acknowledgments

    The authors would like to thank Professor Vincent Smit for scoring the cases in the e-Learning module to develop an answer model, Professor Hein Putter for advice on statistical analysis, and Ms. Cleo Keppens and Ms. Kelly Dufraing for their help with the e-Learning software. The UNITED study is funded by The Dutch Cancer Society (KWF Kankerbestrijding project 10174) and the Stichting Fonds Oncologie Holland.

    Conflicts of Interest

    None declared.

    ‎
    Multimedia Appendix 1

    Overview of the participating countries and number of participating pathologists or residents per country.

    PDF File (Adobe PDF File), 112 KB
    ‎
    Multimedia Appendix 2

    Example of a multiple choice question of the e-Learning. Each participant was asked to select the annotation at the most optimal position following the criteria mentioned in the scoring protocol.

    PDF File (Adobe PDF File), 2755 KB
    ‎
    Multimedia Appendix 3

    Flowchart for the instruction of participating pathologists using the e-Learning module.

    PDF File (Adobe PDF File), 168 KB
    ‎
    Multimedia Appendix 4

    Progress of κ scores per participant separated for pathologists (A) and residents (B).

    PDF File (Adobe PDF File), 141 KB
    ‎
    Multimedia Appendix 5

    Examples of difficult cases for scoring the tumor-stroma ratio (TSR) as mentioned by the participants. (A) A case around the cut off-value of 50%. (B) Necrosis makes it difficult to select the most optimal site for the annotation. (C) A mucinous tumor can sometimes be difficult to score, because mucus has to be visually excluded from the scorings area. (D) Stromal tissue and smooth muscle tissue.

    PDF File (Adobe PDF File), 244 KB
    ‎
    Multimedia Appendix 6

    Subdivision of the 11 questions classified as difficult, showing the proportion of wrong answers per question.

    PDF File (Adobe PDF File), 108 KB
    1. Labianca R, Nordlinger B, Beretta G, Mosconi S, Mandalà M, Cervantes A, ESMO Guidelines Working Group. Early colon cancer: ESMO Clinical Practice Guidelines for diagnosis, treatment and follow-up. Ann Oncol 2013 Oct;24(Suppl 6):vi64-vi72 [FREE Full text] [CrossRef] [Medline]
    2. Mesker WE, Liefers G, Junggeburt JMC, van Pelt GW, Alberici P, Kuppen PJK, et al. Presence of a high amount of stroma and downregulation of SMAD4 predict for worse survival for stage I-II colon cancer patients. Cell Oncol 2009;31(3):169-178 [FREE Full text] [CrossRef] [Medline]
    3. van Pelt GW, Kjær-Frifeldt S, van Krieken JHJM, Al Dieri R, Morreau H, Tollenaar RAEM, et al. Scoring the tumor-stroma ratio in colon cancer: procedure and recommendations. Virchows Arch 2018 Oct 20;473(4):405-412 [FREE Full text] [CrossRef] [Medline]
    4. Mesker WE, Junggeburt JMC, Szuhai K, de Heer P, Morreau H, Tanke HJ, et al. The carcinoma-stromal ratio of colon carcinoma is an independent factor for survival compared to lymph node status and tumor stage. Cell Oncol 2007;29(5):387-398 [FREE Full text] [CrossRef] [Medline]
    5. Wu J, Liang C, Chen M, Su W. Association between tumor-stroma ratio and prognosis in solid tumor patients: a systematic review and meta-analysis. Oncotarget 2016 Oct 18;7(42):68954-68965 [FREE Full text] [CrossRef] [Medline]
    6. Zhang R, Song W, Wang K, Zou S. Tumor-stroma ratio (TSR) as a potential novel predictor of prognosis in digestive system cancers: A meta-analysis. Clin Chim Acta 2017 Sep;472:64-68. [CrossRef] [Medline]
    7. Kramer CJH, Vangangelt KMH, van Pelt GW, Dekker TJA, Tollenaar RAEM, Mesker WE. The prognostic value of tumour-stroma ratio in primary breast cancer with special attention to triple-negative tumours: a review. Breast Cancer Res Treat 2019 Jan 9;173(1):55-64 [FREE Full text] [CrossRef] [Medline]
    8. Wang K, Ma W, Wang J, Yu L, Zhang X, Wang Z, et al. Tumor-stroma ratio is an independent predictor for survival in esophageal squamous cell carcinoma. J Thorac Oncol 2012 Sep;7(9):1457-1461 [FREE Full text] [CrossRef] [Medline]
    9. Dekker TJA, van de Velde CJH, van Pelt GW, Kroep JR, Julien J, Smit VTHBM, et al. Prognostic significance of the tumor-stroma ratio: validation study in node-negative premenopausal breast cancer patients from the EORTC perioperative chemotherapy (POP) trial (10854). Breast Cancer Res Treat 2013 Jun 25;139(2):371-379. [CrossRef] [Medline]
    10. Eriksen AC, Sørensen FB, Lindebjerg J, Hager H, dePont Christensen R, Kjær-Frifeldt S, et al. The prognostic value of tumour stroma ratio and tumour budding in stage II colon cancer. A nationwide population-based study. Int J Colorectal Dis 2018 Aug 21;33(8):1115-1124 [FREE Full text] [CrossRef] [Medline]
    11. Peng C, Liu J, Yang G, Li Y. The tumor-stromal ratio as a strong prognosticator for advanced gastric cancer patients: proposal of a new TSNM staging system. J Gastroenterol 2018 May 16;53(5):606-617 [FREE Full text] [CrossRef] [Medline]
    12. Kemi N, Eskuri M, Herva A, Leppänen J, Huhta H, Helminen O, et al. Tumour-stroma ratio and prognosis in gastric adenocarcinoma. Br J Cancer 2018 Aug 30;119(4):435-439 [FREE Full text] [CrossRef] [Medline]
    13. Pongsuvareeyakul T, Khunamornpong S, Settakorn J, Sukpan K, Suprasert P, Intaraphet S, et al. Prognostic evaluation of tumor-stroma ratio in patients with early stage cervical adenocarcinoma treated by surgery. Asian Pac J Cancer Prev 2015;16(10):4363-4368 [FREE Full text] [CrossRef] [Medline]
    14. Smit M, van Pelt G, Roodvoets A, Meershoek-Klein Kranenbarg E, Putter H, Tollenaar R, et al. Uniform noting for international application of the tumor-stroma ratio as an easy diagnostic tool: protocol for a multicenter prospective cohort study. JMIR Res Protoc 2019 Jun 14;8(6):e13464 [FREE Full text] [CrossRef] [Medline]
    15. Parker EU, Reder NP, Glasser D, Henriksen J, Kilgore MR, Rendi MH. NDER: a novel web application for teaching histology to medical students. Acad Pathol 2017 Feb 10;4:2374289517691061 [FREE Full text] [CrossRef] [Medline]
    16. Helle L, Nivala M, Kronqvist P, Gegenfurtner A, Björk P, Säljö R. Traditional microscopy instruction versus process-oriented virtual microscopy instruction: a naturalistic experiment with control group. Diagn Pathol 2011 Mar 30;6(Suppl 1):S8 [FREE Full text] [CrossRef] [Medline]
    17. Sahota M, Leung B, Dowdell S, Velan GM. Learning pathology using collaborative vs. individual annotation of whole slide images: a mixed methods trial. BMC Med Educ 2016 Dec 12;16(1):311 [FREE Full text] [CrossRef] [Medline]
    18. Samulski TD, Taylor LA, La T, Mehr CR, McGrath CM, Wu RI. The utility of adaptive eLearning in cervical cytopathology education. Cancer Cytopathol 2018 Feb 20;126(2):129-135. [CrossRef] [Medline]
    19. IJspeert JEG, Madani A, Overbeek LIH, Dekker E, Nagtegaal ID. Implementation of an e-learning module improves consistency in the histopathological diagnosis of sessile serrated lesions within a nationwide population screening programme. Histopathology 2017 May 08;70(6):929-937. [CrossRef] [Medline]
    20. Madani A, Kuijpers CCHJ, Sluijter CE, Von der Thüsen JH, Grünberg K, Lemmens VEPP, et al. Decrease of variation in the grading of dysplasia in colorectal adenomas with a national e-learning module. Histopathology 2019 May 10;74(6):925-932. [CrossRef] [Medline]
    21. Nakanishi H, Doyama H, Ishikawa H, Uedo N, Gotoda T, Kato M, et al. Evaluation of an e-learning system for diagnosis of gastric lesions using magnifying narrow-band imaging: a multicenter randomized controlled study. Endoscopy 2017 Oct 21;49(10):957-967. [CrossRef] [Medline]
    22. Canty D, Barth J, Yang Y, Peters N, Palmer A, Royse A, et al. Comparison of learning outcomes for teaching focused cardiac ultrasound to physicians: A supervised human model course versus an eLearning guided self- directed simulator course. J Crit Care 2019 Feb;49:38-44. [CrossRef] [Medline]
    23. Maertens H, Madani A, Landry T, Vermassen F, Van Herzeele I, Aggarwal R. Systematic review of e-learning for surgical training. Br J Surg 2016 Oct;103(11):1428-1437. [CrossRef] [Medline]
    24. Tumor-stroma ratio. UNITED.   URL: http://watchstroma.com/ [accessed 2019-05-02] [WebCite Cache]
    25. Huijbers A, Tollenaar R, v Pelt G, Zeestraten E, Dutton S, McConkey C, et al. The proportion of tumor-stroma as a strong prognosticator for stage II and III colon cancer patients: validation in the VICTOR trial. Ann Oncol 2013 Jan;24(1):179-185 [FREE Full text] [CrossRef] [Medline]
    26. Lugli A, Kirsch R, Ajioka Y, Bosman F, Cathomas G, Dawson H, et al. Recommendations for reporting tumor budding in colorectal cancer based on the International Tumor Budding Consensus Conference (ITBCC) 2016. Mod Pathol 2017 Sep;30(9):1299-1311. [CrossRef] [Medline]
    27. Harris E, Lewin D, Wang H, Lauwers G, Srivastava A, Shyr Y, et al. Lymphovascular invasion in colorectal cancer: an interobserver variability study. Am J Surg Pathol 2008 Dec;32(12):1816-1821 [FREE Full text] [CrossRef] [Medline]
    28. Chandler I, Houlston RS. Interobserver agreement in grading of colorectal cancers-findings from a nationwide web-based survey of histopathologists. Histopathology 2008 Mar;52(4):494-499. [CrossRef] [Medline]
    29. Barresi V, Reggiani Bonetti L, Ieni A, Branca G, Tuccari G. Histologic prognostic markers in stage IIA colorectal cancer: a comparative study. Scand J Gastroenterol 2016 Mar;51(3):314-320. [CrossRef] [Medline]
    30. Jensen P, Krogsgaard MR, Christiansen J, Braendstrup O, Johansen A, Olsen J. Observer variability in the assessment of type and dysplasia of colorectal adenomas, analyzed using kappa statistics. Dis Colon Rectum 1995 Feb;38(2):195-198. [CrossRef] [Medline]
    31. Terry MB, Neugut AI, Bostick RM, Potter JD, Haile RW, Fenoglio-Preiser CM. Reliability in the classification of advanced colorectal adenomas. Cancer Epidemiol Biomarkers Prev 2002 Jul;11(7):660-663 [FREE Full text] [Medline]
    32. Denis B, Peters C, Chapelain C, Kleinclaus I, Fricker A, Wild R, et al. Diagnostic accuracy of community pathologists in the interpretation of colorectal polyps. Eur J Gastroenterol Hepatol 2009 Oct;21(10):1153-1160. [CrossRef] [Medline]
    33. Costantini M, Sciallero S, Giannini A, Gatteschi B, Rinaldi P, Lanzanova G, SMAC Workgroup. Interobserver agreement in the histologic diagnosis of colorectal polyps. the experience of the multicenter adenoma colorectal study (SMAC). J Clin Epidemiol 2003 Mar;56(3):209-214. [CrossRef] [Medline]
    34. Smits AJJ, Kummer JA, de Bruin PC, Bol M, van den Tweel JG, Seldenrijk KA, et al. The estimation of tumor cell percentage for molecular testing by pathologists is not accurate. Mod Pathol 2014 Feb;27(2):168-174. [CrossRef] [Medline]

    ‎
    H&E: hematoxylin and eosin
    LUMC: Leiden University Medical Centre
    TNM: tumor node metastasis
    TSR: tumor-stroma ratio
    UNITED: Uniform Noting for International Application of the Tumor-Stroma Ratio as an Easy Diagnostic Tool

    Edited by G Eysenbach; submitted 17.04.20; peer-reviewed by Y Chu, CS Wu; comments to author 25.09.20; revised version received 14.12.20; accepted 03.03.21; published 19.03.21

    Copyright

    ©Marloes A Smit, Gabi W van Pelt, Elisabeth MC Dequeker, Raed Al Dieri, Rob AEM Tollenaar, J Han JM van Krieken, Wilma E Mesker, UNITED Group. Originally published in JMIR Formative Research (http://formative.jmir.org), 19.03.2021.

    This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on http://formative.jmir.org, as well as this copyright and license information must be included.