This study was designed as a proof-of-principle study. AO images of 2 healthy participants (3 eyes; with no history of ocular diseases and no retinal abnormalities) and 6 patients (12 eyes) with genetically confirmed Stargardt disease were included in this study. The patients with Stargardt disease all had significant macular pathology, but no disease affecting the peripapillary region or spreading outside of the vascular arch. Two test sets of AO-FIO images made with the RTX1 AO retinal camera were created. Each test set included AO-FIO images from a total of 4 participants, including 1 healthy individual and 3 patients with Stargardt disease. The training set included 118 images (32 healthy); the test set contained 157 images (38 healthy).

The images were acquired during outpatient clinic visits. Images were taken starting at the optic nerve head, then moving temporally toward the foveal region and the temporal retina. Image acquisition settings were optimized for each individual image. No formal image quality criteria were used.

All images used in this study were derived from the Radboudumc ophthalmology biobank, which was approved by the Radboudumc Ethics Committee (Commissie Mensgebonden Onderzoek; CMO; 2017-3535). The study was conducted in adherence with the provisions of the Declaration of Helsinki, and all study participants gave written informed consent prior to inclusion in the study.

On AO-FIO images, photoreceptors generally appear as a mosaic of bright maxima [4], resulting in grayscale retinal images (4°×4°) consisting of bright blobs, that is, round groups of pixels with similar pixel intensities standing out against the generalized background gray level. Thus, retinal photoreceptors on AO images can be counted using blob detection analysis [22], which eventually allows estimation of the cone density within the examined area of the retina. Manual counting of the blobs is not practical, as there may be >10,000 blobs per image, resulting in a highly time-consuming analysis if performed by human graders. Automated analysis techniques, on the other hand, are expected to perform with much higher speed and possibly higher accuracy.

One of the authors (JAAHP) used GPT-4 (OpenAI; used between January 22, 2024, and February 19, 2024) to generate a script in the R programming language (version 4.3.2; R Foundation for Statistical Computing) for automated blob counting. ChatGPT was used through OpenAI’s hosted service. The underlying computational environment is managed by OpenAI and is not exposed to the user; therefore, detailed hardware and software specifications are not available. RStudio (version 2022.02.1; Posit Software PBC) was used to run the script. The following R packages were used in the script: EBImage (version 4.44.0) [23], imager (version 0.45.2) [24], abind (version 1.4-5) [25], and spatstat (version 3.0-7) [26]. Only textual instructions were given to the GenAI while generating the code. Using the input provided by GPT-4, preprocessing steps were built into the code for accurate analysis. After every preprocessing step, the code returned the processed image, allowing the researchers to verify whether all preprocessing steps were applied as intended to the image. The initial code was then tested using images from test set 1, and the programming errors that occurred were reported back to GPT-4. In this way, the model was facilitated to generate code while iteratively eliminating programming errors during the process.

To ensure the efficacy of the blob analysis, the language model was directed to generate code that highlighted the identified cones on the original images. Subsequently, the accurate location of the blobs on the altered image was manually verified to ensure that only genuine blob-like structures, and not vessels or background grayness, were indicated. This facilitated visual verification by the researchers (JAAHP, LCVDZ, and TT) that the code correctly identified blobs.

Next, another author (LCVDZ) ran the code using a test set of AO-FIO images different from the training data to evaluate the performance of the script on independent images.

To test the script’s performance, the cone counts of 5 AO images (Figure 1) obtained from 2 healthy eyes and 3 eyes of patients with Stargardt disease, selected from the test set, were manually assessed by 2 independent graders (JAAHP and YVL) and compared with those generated by the script and the AO box tool of Wooning et al [17].

Each AO image covered a 4°×4° field of view, which we considered too large for full manual annotation in a single step. Therefore, we extracted smaller regions of interest at random locations within each selected AO image. For the manual grading, the selected image snippets were uploaded to ImageJ (version 1.53t; National Institutes of Health). Using the multiselection tool, all round, bright blobs were counted by the 2 graders. Both graders had extensive experience (>1 year) with the AO RTX1 camera, including expertise in AO image interpretation. No formal grading training was conducted. Grading was performed independently, and the graders were blinded to the presence of pre-existing pathology. There was no adjudicated consensus label. Annotations were saved using the ROI Manager. The intraclass correlation coefficients between the graders, the script, and the AO box tool were calculated, and Bland-Altman analyses were performed. Additional validation of the script was not conducted, as the objective of this study was to show the proof of principle of the produced code.

In our current study, we used GenAI (GPT-4) to generate R-based image analysis code with the potential to perform quantitative analysis of retinal AO-FIO imaging. A key benefit of using GenAI in image analysis lies in its broad availability. This allows researchers to generate scripts for complex image analysis software without extensive pre-existing programming knowledge. However, it appears that using GenAI for image analysis still necessitates a basic understanding of image preprocessing, the principles of image analysis, and the use of GenAI (Multimedia Appendix 2). For instance, one should know the appropriate terminology for the appearance of a certain structure (here, photoreceptors as blobs) in image analysis. In both healthy patients and patients with Stargardt disease, the developed algorithm successfully identified blob-like structures most likely corresponding to photoreceptors throughout the imaging field. Comparing the script cone counts with those of the human graders demonstrated that the generated script counted more cones than the average count of the human graders in every image. In the healthy images A and B, which are depicted in Figure 4, it counted +6.9% (134 cones) and +10.8% (80 cones) more cones, respectively, than the human graders. The worst performance was observed on image D, where the script counted 136.1% more cones (352 cones) than the human graders, as it appears to have identified many false positives at the image border, as shown in Multimedia Appendix 3. The best apparent performance of the graded script was on image E, in which it counted +1.8% (4 cones) more than the average count of the graders.

The generated script counted fewer cones than the AO box tool in both images from healthy participants (506 and 257 fewer, respectively) and also counted fewer cones in Stargardt images C and E (991 and 830 fewer, respectively) in Multimedia Appendix 3, while counting more cones than the AO box in image D (299 more cones) in Multimedia Appendix 3. The intergrader agreement analysis demonstrated that the human graders had high agreement. The generated script also demonstrated high agreement with the mean of the human graders, but low agreement with the AO box. It should be noted that this analysis was performed on only 5 images; therefore, no conclusions can be drawn from these findings. However, it is noteworthy that the generated script had low agreement with the AO box. This may be because the AO box was trained only on healthy participants, whereas our script was generated and tuned using both healthy patients and patients with Stargardt disease.

Our script, for which the threshold values were manually determined using Stargardt images (excluding any of the tested images), was less likely to produce false positives when clear cones were visible. However, in images C and E in Multimedia Appendix 3, some of the selected cones appear to differ between the human graders and the generated script. Moreover, even the intergrader interpretation differed for these images. This is most likely due to the altered appearance of cone photoreceptors in Stargardt disease, which might be a major issue for cone counting. In this study, we used criteria similar to those applied in the AO box study by Wooning et al [17] for defining cone photoreceptors. We used the criterion that a cone photoreceptor should be a bright, round structure. They defined a photoreceptor according to the following criteria: the structure had to be round, exhibit an intensity peak, and have a similar size and spacing as neighboring structures. This last criterion was not used in our study. However, it is known that cone spacing changes in Stargardt disease [27], making this criterion for identifying a cone unreliable when counting cones in the pathological retina. Before AO can be used as an end point, or even as a secondary biomarker for clinical trials, there needs to be consensus on the definition of a cone in Stargardt disease to reliably count the number of cones.

This study has several limitations that warrant consideration when interpreting the results. The most significant limitation we encountered with R was the lack of options for watershed analysis. We circumvented this issue by estimating the cone count based on a predetermined cone pixel size definition. This option only works when all analyzed images have the same image resolution as the images in the training set. Nevertheless, watershed segmentation of the blobs would most likely have made the script more reliable [28]. Other quality measures that we wanted to implement in the script but that did not work were the width-to-height ratio for cones, as cones are often round, and Voronoi analysis to assess the cone mosaic [29]. In hindsight, R appeared not to have been the most logical choice of programming language for image analysis. We chose R because it was already available within our research environment and was more familiar to the team than other programming languages, such as Python. In retrospect, Python-based libraries may provide more mature and flexible tools for image analysis applications of this type.

Another weakness of the script we generated is the manually derived threshold value. For large-scale automated analysis of AO images, however, this value may need to be adapted to different datasets. Therefore, for the script to function reliably, an automated thresholding method will be required.

It should be noted that the script was generated using GPT-4. More advanced GenAI models are already available, potentially generating more sophisticated scripts. However, thorough testing and validation of such scripts will still be essential before using them in clinical research [30]. For now, we were able to generate a functional script for AO image analysis that was capable of detecting blob-like structures in retinal images. We did not develop or validate this script as a clinically robust research tool. Before such an approach could be considered suitable for clinical or large-scale research applications, the script would require substantial refinement, methodological validation, and review by experts in computer vision and biomedical image analysis to address issues related to generalizability, parameter optimization, and robustness across datasets. This study sought solely to demonstrate the feasibility of using GPT-4 for quantitative image analysis coding in AO imaging applications. The present study provides a preliminary evaluation of the generated script by comparing its output on 5 image snippets with counts obtained by human graders and the AO box tool. Although the script demonstrated strong agreement with the human graders, the small sample size and exploratory nature of this analysis preclude definitive conclusions regarding its accuracy, reliability, or generalizability. Therefore, a larger validation study using an improved version of the script and a substantially larger set of images is required before its performance can be comprehensively assessed and validated.

A general challenge in using GenAI is the occurrence of hallucinations [31]. Hallucinations, in the field of natural language processing, are defined as situations in which the generated product of GenAI is inconsistent with the given input or expected output environment, violates facts, or lacks meaning [32]. Hallucination in coding often appears when adding new coding snippets to perform a certain function in an already existing script or coding environment [33]. Adding medical images directly to ChatGPT, which we did not do, can also cause hallucinations, generating unreliable output [34]. While generating our AO image analysis script, we encountered some hallucination-related issues. As can be seen in Multimedia Appendix 2, GPT-4 claimed it could add watershed analysis and Voronoi analysis to the script, but this resulted only in a nonfunctioning script, which required us to reprompt using an older version of the script to stay on track. Extensive testing of the GenAI output is necessary to ensure that no hallucinated strings of code are present in a final script that will be used for formal analysis. Furthermore, researchers must be aware that GenAI may “forget” previous prompts, which may need reiteration, and that it may not be able to combine ideas from earlier failed steps with the current step. It has been shown that learning and applying prompt engineering [35] may lead to a reduced number of prompts needed. Another potential advantage of prompt engineering is that prompts should be designed in such a way that older generated code is not “forgotten” by GenAI. Standardized and improved prompting may result in more effective results with improved precision.

In conclusion, we were able to generate a functional script using GPT-4 that was able to perform blob detection in AO images and deliver plausible output compared to human graders and pre-existing software. However, before such a script can be used reliably in research, it should undergo extensive fine-tuning, testing, and validation to enhance its robustness. Future research should focus on more potent image analysis software coding and the validation of its results to ultimately assess the usability of cone counts measured on AO images as a biomarker for clinical trials.