The VNRC and the Crystalvue NFC-700 were used in this study. The VNRC is a nonmydriatic, macula-centered, 45° field imaging retinal camera system that uses a set of proprietary machine learning algorithms to generate high-fidelity composite color posterior chamber images of the eye by capturing and compiling a series of retinal images from each perceived single flash (<200 ms). The camera system is incorporated into a cloud-based software platform that accepts digitized images transferred from the VNRC and has the capacity to store, convert formats, display, and transfer data or imaging data between cameras (Figure S1 in Multimedia Appendix 1).

The Crystalvue NFC-700 camera [19,20] uses a conventional technique [21] to automatically adjust focus and capture the best quality image using a single annulus type of illumination. The image is captured by the built-in color complementary metal-oxide-semiconductor camera module.

This was an analysis of previously collected paired-image acquisitions (discussed further in this study). The objective of this study was to characterize the overall performance of the VNRC, articulated in 2 aims, that is, first, characterize the comparative performance of the VNRC (as an investigational device) against a reference instrument to generate clinically interpretable images, and second, evaluate the utility of the image quality control feature (IQCF) embedded in the camera system (discussed in section “Development and Description of a Retinal Imaging Camera for Primary Care”), by investigating the association between the outputs of this algorithm scoring the quality (ie, the capture status) of the images and image gradeability as determined by ophthalmologists.

The Verily Retinal Service was developed for use in primary care clinics. The VNRC is a lightweight (approximately 6 kg), 45° field imaging camera system consisting of custom electronics, optics, LEDs, and a retinal camera (Figures 1A and 1B). The VNRC has a range of pivot=0° to 45° to adjust for user height, posture, and comfort. The black face rest is light blocking, which enables retinal imaging in bright ambient lighting conditions, and the built-in handle allows for flexibility for camera placement within a clinic space.

The VNRC uses a stereo infrared camera system (pupil camera 1 and 2 in Figure 1B) to achieve proper pupil alignment and facilitate fast image acquisition (120 frames per second [fps]). Users can adjust an image to their best perceived focus with a focus knob, and an interactive game-like interface allows users to optimize proper pupil alignment using small eye or head movements within the face rest, while operators can simultaneously oversee this process to ensure proper eye location (Figure 1C). The focus motor controls both the retinal camera and microdisplay, allowing the user to view an interactive image and video inside the camera. This is similar to an experience in a virtual reality headset, but monocular.

After the initial focus step, users position their head into the VNRC face rest in order to align their eye with the camera lens. Once proper focal length is established by dialing the focus knob, users can confirm by pressing the top of the focus knob. Alternatively, for users unable to use the focus knob, the VNRC can also be brought into focus using an operator-driven setting controlled via the software graphical user interface.

The VNRC then uses a proprietary redundant illumination system to automatically collect a series of retinal images (up to 30). This is in contrast to conventional techniques that only capture 1 image frame per flash [21] and is made possible by the use of high-speed high-sensitivity image sensors (120 fps) deeply integrated with the system-on-chip. The flash duration is limited to ensure sufficient quality, minimizing dazzle. Here, the system-on-chip uses an LED array with the camera system to capture a series of full-field (>45°) images of the retina with various illuminations focused on different areas of the crystalline lens of the eye (Figure 2). The proprietary redundant illumination system dynamically adapts to specific eye characteristics, such as pupil size and position, and optimizes a series of retinal illumination configurations, regardless of corneal clarity.

This sequence of flashes, or “burst imaging,” illuminates the entire field of view, occurs within 200 milliseconds, and is perceived as a single flash. The acquisition time falls below the typical latency of the pupil’s response time [22], so that the imaging process is not expected to interfere with natural pupil constriction. The typical burst image set contains many fully illuminated images, with varying levels of artifacts on the images.

The VNRC uses a proprietary “Burst Reduce Algorithm” to generate a single high-fidelity retinal image (Figures 3A and 3B), by merging a variable number of frames. The total number of frames is typically 20 images, but is highly dependent on the presence of artifacts and the amount of eye motion during the camera flash. The algorithm draws similarities from pixel-level high dynamic range imaging [23] but incorporates an assignment of a score layer to separate photons due to retinal reflection from scattered photons due to cataract or cornea reflection, or lid or lash occlusion, after compensating for intra-acquisition gaze shift. It is notable that no fine spatial filtering was applied to enhance the contrast or segment-specific features or pathologies [24,25].

A subsequent algorithm, the IQCF algorithm, then performs a quality control assessment of the resulting composite image to determine if a recapture is needed (Figure 3C). The model was trained to compute areas where the retina is clearly visible, without impaired visibility due to (1) darkness, (2) saturation, (3) blur, or (4) haze. The IQCF algorithm produces a single score per image by calculating the probability that each pixel contributes to that of the retina signal, as represented by an image mask. The algorithm incorporates a fixed cutoff to determine if the image needs to be recaptured. If the score does not exceed this operating point, then the camera will output instructions to the operator that another image acquisition is required. Thus, the VNRC provides real-time image quality characterization information to help reduce the chances that a substandard image is used for clinical interpretation. Importantly, the IQCF does not score image gradeability.

The final output of the VNRC can integrate into a primary care workflow. It feeds into a cloud-based software platform that accepts digitized images transferred from the camera and has the capacity to transfer, store, convert formats, display, and transfer medical device data or medical imaging data between medical devices (Figure S1 in Multimedia Appendix 1). The VNRC also continuously collects metadata such as details on camera use, uptime, and operation in order to allow for real-time error handling. This passively collected metadata is uploaded along with the retinal images.

This was a retrospective study conducted in a subset of the participants in a prospective technical feasibility trial (Verily protocol 103535; approved by institutional review board [IRB] Western IRB, before initiation, IRB Protocol #20214693) [26]. All participants signed informed consent (Multimedia Appendix 2) approved by the IRB and received nominal compensation for their time ($25 for the screening procedure, an additional $75 for each completed study visit). Personal study-related data were managed in accordance with local data protection law.

Eye images (K=206) were captured from 108 participants (Table 1) with both the VNRC and the Crystalvue NFC-700 Camera.

The proportion of images of sufficient quality for clinical interpretation was 0.985 (203/206) and 0.971 (200/206) for the VNRC and Crystalvue NFC-700 cameras, respectively. The difference in proportion was 0.015 (95% CI –0.007 to 0.033; Table 2, examples in Figure 4).

Graders also computed the quality of the images across several specific criteria, for both the VNRC and the reference camera (Tables S1 and S2 in Multimedia Appendix 1), indicating whether the quality of specific image aspects was sufficient for clinical interpretation (note that these were not pooled for a single majority classification; there were 3 separate adjudications for each factor in each image). Graders evaluated the visualization of the optic disc and determined that it was adequate in a majority of images for both devices (in at least 90%, 225/248 of images by the VNRC and in at least 93%, 195/209 of the reference). There were similar results in the classification of macula visualization (deemed appropriate in at least 92%, 229/248 of VNRC images and 83%, 175/209 of reference images), and the visualization of the retinal vessels (appropriate in at least 94%, 233/248 of the VNRC images and 92%, 194/209 of reference images). Graders also found a majority of images to be of sufficient quality regarding key imaging features, such as adequate focus (in at least 91%, 228/248 of VNRC images and 88%, 185/209 of reference images), appropriate brightness (in at least 94%, 213/248 and 198/209 of images, for both), adequate field of view (in at least 91%, 228/248 and 191/209 of images, for both), no significant image defects (in at least 91%, 227/248 and 92%, 194/209), no small pupil interference (at least 89%, 222/248 and 87%, 183/209), and no ocular media opacity (in at least 73%, 183/248 and 85%, 178/209 of VNRC and reference images, respectively).

We found a moderate association (φ=0.58) between ophthalmologists’ assessments of a retinal image’s gradeability and the IQCF algorithm’s scoring of capture status (Table 3).

Overall, the IQCF scored 50 images as needing recapture, but the human assessment was “gradable” for 60% (30/50) of these. Conversely, the vast majority of the images scored by the IQCF as not needing recapture, 292 out of 293 (99.7%), were found “gradable” by human assessment. Refer to Figure 5 for examples of images with concordant and discordant IQCF scoring and human assessment.

There was agreement among 100% (15/15) of the participating users in the simulated clinic environment that they were able to have both eyes screened easily. In addition, they somewhat agreed or strongly agreed (rating of 5 to 7 on a 1‐7 Likert scale) with statements indicating that they were confident in knowing how to complete the screening after watching the video, that they found it intuitive to set themselves up with the camera properly, felt comfortable while completing the screening, and had a positive experience with the camera (Table 4, Table S3 in Multimedia Appendix 1).

All participating operators under simulated conditions (30/30, 100%), who had either health care degrees, licenses, or some training somewhat agreed or strongly agreed (ratings of 5 to 7 on a 1‐7 Likert scale) with statements indicating that they felt comfortable completing the screening, had a positive experience while using the camera, found training easy and useful to understand, found it easy to capture a retinal image with the camera, and found the camera easy to clean (Table 5, and Table S4 and S5 in Multimedia Appendix 1). A majority (23/30) felt that hands-on help for users was probably not needed, and approximately half (14/30) felt that they would not have to apply their relevant clinical training to complete specific tasks.

Most of the issues reported by users related to involuntary clicks back and forth in the user interface, particularly during the “focus image” steps. These issues, at most, caused delays (not failures) in the image capture process.

We report here the results of a series of analyses to characterize the performance and usability of a new retinal camera system aimed for implementation in primary care settings. The performance was satisfactory in 2 main aspects. First, the VNRC performed at comparable levels to a reference camera for the generation of quality retinal images; the numeric difference in the percentage proportion of images with quality to be clinically interpretable was low (0.015), and the CI for that difference straddled 0, indicating a likelihood that there was no difference between the 2 devices. Further reinforcing the main results, gradability across a variety of image quality metrics appeared numerically similar across VNRC and the reference camera. Second, the quality-scoring outputs of the quality control algorithm embedded in the system showed a moderate association (φ=0.58) with the classification of images as “gradable” or “ungradable” by human graders. Furthermore, operators and users (ie, individuals with and without previous health care training) found the system to be generally intuitive and approachable to use, allowing them to feel comfortable performing image captures on their own (at least 95%, 44/45 of survey participants agreed to the corresponding statements).

Our findings indicate that VNRC can perform at the level of a standard tabletop retinal camera system. Thus, the VNRC may provide a balance of features that could mitigate some of the reservations from primary care providers to the adoption of DR screening programs. Across both primary care and specialty clinics, tabletop equipment may be the highest quality option for retinal imaging [27]; however, high direct costs and the resource and space demands of optimal installation (ie, dedicated darkened room) act as deterrents, particularly in underresourced environments. While handheld and smartphone-based cameras overcome these aspects, reports are mixed regarding the quality of their image capture and their ease of successful use for DR screening (for instance, the skill and finesse required to obtain proper eye alignment [13,28], particularly when imaging is performed without pharmacologic pupil dilation [13,18,29-31]. Most of the images excluded from the comparative analysis (for lack of bilateral counterparts) were in the reference camera group (probably due to the inability of that camera to capture images for pupil sizes below 3.5 mm). The rates of ungradable images we observed are encouragingly low, considering that other studies with mydriatic devices (where they would be expected to be lower) have reported approximately 6% [32], or ranges from 0% to approximately 28% with mydriatic cameras (when grading was done via algorithm) [33]. A possible downside of producing a high rate of ungradable images (which can be due to a variety of causes, such as small pupil size, eye pathologies) may be an inflation of referrals that can overwhelm downstream eye care services and reduce the overall cost-effectiveness of teleophthalmology programs [18,30]. The VNRC has advantages closely associated with handheld systems in cost, maneuverability, relatively low weight (10‐20 kg lighter than other tabletop systems), and relatively low space demands. Our results suggest that this system can produce images of quality comparable with expensive and more sophisticated tabletop equipment, overall and across specific image metrics, possibly reaching a balance between operational requirements and quality performance, which is particularly well-suited to primary care clinics.

Our results also indicate that the quality control feature within the VNRC system, the IQCF, functions as intended and could effectively filter out the intake of undesirable, poor-quality images into practical clinical workflows. Capturing retinal images of quality is a necessary requisite for the subsequent clinical utility of those images during diagnosis; entering a high proportion of low-quality images could render a clinical workflow inefficient and impair effective care. The approach presented in this work was to embed an algorithm to score image quality in real time, prompting users to discard low-quality images (those below a score threshold) and recapture before entering an image into the clinical workflow. Thus, it was important to establish the correlation between the proportion of retinal images passing the VNRC’s IQCF threshold and downstream assessments of gradability according to ophthalmic graders. We found a moderate correlation, as nearly all the images cleared by the IQCF as “not needing recapture” were indeed found “gradable” by human assessment. The IQCF scored noticeably more images as “needing recapture” than human graders deemed “ungradable” (ie, 60%, 30/50 of images that IQCF labeled as needing recapture were actually deemed gradable during human assessment). This is probably due to the fact that the IQCF is a visibility measure, with an output that is pathology-independent; in contrast, human graders may grade images if a lesion is clearly identifiable, even in a context of low overall quality or visibility that the IQCF probably would “not pass.” This excess of images with “recapture needed” scores, however, may not represent a major practical problem, since image recapture in real time may not be a burdensome procedure. Ultimately, this ensures that the IQCF facilitates an inflow of quality images, without erring in a direction that would create backflows or inefficiencies, impairing practical clinic workflows after the fact.

Our third major result showed that participants found that handling the VNRC system was easy and intuitive and felt comfortable with it. This addresses another relevant concern for primary care practice managers considering retinal screening systems, namely, the actual or perceived need for dedicated trained staff. Furthermore, systems with which users and operators experience repeated lack of success in producing images of sufficient quality may undermine confidence. In turn, this can nudge personnel toward lower usage [18] and depress the cost-effectiveness of a screening program. Our survey results indicate that the VNRC could be an approachable system that mitigates usability barriers.

While our analyses yielded promising results, the characterization of this new DR screening system had some limitations, largely related to the generalizability of our findings. First, the images for the comparative performance analysis were collected first with the VNRC, followed by the reference camera, in order to maintain internal consistency with the larger study from which this retrospective analysis was undertaken [26]. Therefore, we cannot discount the possibility of experimental bias related to the order of measurement (it could be a learning effect or a fatigue effect) that could have influenced the results. Second, our analyzable dataset for the comparative analysis images excluded participants without complete bilateral, nonmydriatic image sets and annotations, which could have exerted some selection bias toward better-quality cases; while that effect would be expected to some extent for both groups in the comparison, it may have had a differential effect that we cannot discount. Third, we cannot interpret the potential impact of not-captured images in our comparative performance analysis. Some of those noncaptures may correspond to smaller pupil sizes, since the instructions for use for Crystalvue NFC-700 require a minimum pupil size of 4 mm. Therefore, different study conditions (eg, dimming ambient light in order to dilate pupil for an attempted recapture) may yield different comparative results from the ones in this report. Fourth, the VNRC IQCF would encourage image recaptures for low-quality scores in an actual clinic setting, until the operator has a total of 3 eye images or one of the images scores above the threshold. In this study, graders received image captures according to a prespecified sequence that may not have consistently maximized quality. Therefore, future studies are warranted to investigate performance across a more diverse set of conditions. For instance, future postmarket analyses could evaluate performance relying on the images with the top IQCF score across all eye captures, or in different environmental setups (such as lighting), to better reflect performance in real-clinic conditions. Another area for future study is the characterization of variability in larger cohorts, more widely representative of primary-care screening populations. Importantly, disease characteristics (burden and type of disease) may impact the relative performance in terms of gradeability and the association of image quality with gradeability. Finally, we did not collect information on visual acuity from the participants in the user study; while results were overall favorable, visual acuity (or lack thereof) may affect user-device interactions; therefore, it will be worthwhile to gather appropriate information in future studies to better understand it.

In summary, barriers to primary case-based DR screening overall are multifactorial [18,34,35]. Providers tend to perceive that rigorous and cost-effective implementation of teleophthalmology in primary care settings is expensive and difficult, demanding restructuring of work processes and increasing the burden on clinic staff [18,36]. Our results indicate that the VNRC system could mitigate some of these issues, particularly in underresourced environments. It has the positive operational characteristics akin to handheld systems, produces images of quality comparable with standard tabletop retinal cameras, and is able to optimize the inflow of quality images into clinical workflows. In addition, users note that they are able to handle the system and produce usable images with ease. Furthermore, the transfer and flow of the digital output are adaptable to typical primary care workflows. These results support considering this system as an integrated end-to-end retinal service suitable for primary care and warrant additional studies across a wider and diverse set of primary care clinics. Novel DR screening systems that address primary care adoption barriers may represent an advance toward more widespread access, with the potential to curtail rates of severe disease progression at the population level and ultimately contribute to better patient outcomes.