Fig. 4.1
Figure shows the receiver operating characteristic (ROC) curve for the Iowa Detection Program (IDP) as well as the sensitivity and specificity pairs of the three retinal specialists in that study for comparison. CI confidence interval (Adapted from Abramoff et al. [13])
It is hard to compare curves, and to capture the curve in a single number, the area under the ROC curve (AUC), is widely used. The AUC is a number between 0 and 1, and an automated method that is no better than a coin toss will have an AUC of 0.5, and the closer to 1 the AUC is, the better the method is able to distinguish people with diabetic eye disease from people without diabetic eye disease. A major advantage of the AUC measure is that it is not dependent on the distribution of people with and without diabetic eye disease (of some severity) in the population on which the method is tested [33]. Because the set point can be placed at a specific value, we can therefore adjust the automated method to have an expected sensitivity and specificity. Several authors have argued that the sensitivity for detection of diabetic eye disease in a screening program is not cost effective if it is higher than 80 % or even 60 % [34]. There is some rationale for such an upper bound on sensitivity as many studies have shown that 80 % exceeds the achievable sensitivity for clinicians, which is between 30–50% [7–10]. The Australian guidelines for optometrists have in the past stated a sensitivity of 60 % [35]. If a higher sensitivity is preferred for safety reasons, for a given method, this will result in the specificity then being lower, affecting the effectiveness of the automated method as explained above. Patients or institutions, including insurance companies and governments, that pay for healthcare may opt for increased specificity at the cost of lower sensitivity, knowing that there is a low risk of missing patients who need immediate treatment, while there is a cost savings by reducing the number of people who are referred unnecessarily.
4.5 Retinal Imaging Protocols for Automated Methods
The ICDR classification does not define an imaging protocol, but does refer to the Early Treatment Diabetic Retinopathy Study (ETDRS) which was based on seven stereo photographic fields. The genesis of the ICDR was due to the fact that taking and then grading images from seven stereo fields was found to be too cumbersome to be widely used in clinical practice [25]. Previous studies have shown that reading fewer fields than seven is comparable for detecting DED to reading multiple fields or a dilated retinal examination [9, 36, 37].
Because of their relatively low cost and relative ease of use, the so-called nonmydriatic cameras, which do not require the eye to be pharmacologically dilated during focusing, are in widespread use in screening programs using human readers. The quality of the photographs is lower than the typical dilated color photos used in protocols like the ETDRS [6]. However, many studies have shown that two or even one field per eye suffices to capture most cases of diabetic eye disease, and therefore, most automated methods accept one, two, or at maximum three fields per eye at this quality level [9, 38]. Because humans can easily make a difference between a retinal image and a lens or patient label image, as required by some protocols, there is in many cases no standard order or naming convention. However, most automated methods will have unpredictable results if run on non-retinal images. Potential solutions are to use image formats or protocols that capture such meta-information, including the standard for ophthalmic fundus imaging established by the Digital Imaging Communications in Medicine (DICOM) Working Group 9 or naming conventions for the right and left eye, right and left retina, and any other structure that is imaged, so that the retinal images can be identified unequivocally without additional human input. Though formally DICOM is the standard for fundus imaging, acceptance has been slow, necessitating the latter, more involved methods.
Most cameras are fundus cameras, i.e., they measure reflectance of the retina at different wavelengths simultaneously [12]. Wide-field scanning laser ophthalmoscope (SLO) imaging is an attractive modality to image the retina, though the devices have higher cost. Recently a study of 141 people with diabetes showed that wide-field SLO read by experts had high kappas κ = 0.70 compared to the same experts reading the standard 7-field photographs [39]. However, the performance of automated methods on wide-field SLO, though in development, has not yet been published. Another potentially attractive imaging modality, optical coherence tomography (OCT) has not been validated for diabetic eye disease screening, though automated algorithms to analyze these images and potentially detect DR have been published [40].
Retinal fundus imaging with a nonmydriatic camera typically takes 10 min or more. This includes preparing the camera, discussing the procedure with the patient, accessing or recording the meta-data, and imaging both eyes. Some automated methods have their output ready in less than a minute on standard hardware [13]. This quick result can be provided at the point of care and, for most people, will eliminate the need for a separate visit to learn the result or a phone call from a healthcare professional discussing the result, thus potentially increasing compliance.
4.6 Principles of Operation of Automated Methods
Most of the automated methods discussed later in this chapter work in a similar fashion. As a typical example, the Iowa Detection Program consists of separate, mostly independent components for detecting the optic disc, fovea, and retinal vessels, measuring image quality, and detecting microaneurysms, hemorrhages, exudates, cotton wool spots, and irregular lesions including large hemorrhages and neovascularization. A fusion algorithm combines the output of all of these components into a single numerical output [13]. Examined in more detail, microaneurysm detection, one of the most important components, works by examining each pixel in each image, analyzing its color and intensity as well as that of surrounding pixels and then using these pixel level measures to group neighboring pixels into candidate lesions. Candidate lesions are then similarly analyzed on their color, size, shape, location, type, and other properties. The final output produced by the fusion algorithm is a single number, the DR Index, a dimensionless number between 0 and 1. The DR Index expresses the likelihood that the patient’s images show diabetic eye disease.
Automated algorithms for microaneurysm detection were described as far back as 1984, but these analyzed angiograms, not fundus images [41]. The first microaneurysm detection algorithms were developed and evaluated by Spencer and coauthors in 1996 [42]; his method followed the standard approach of applying a sensitive candidate lesion detector followed by a classification step, and most existing microaneurysm detectors are refinements and generalizations of this approach [43, 44]. Most of the automated methods are thus considered bottom-up approaches, but an interesting exception is the University of New Mexico method developed and evaluated by Agurto and coworkers [45], who use a texture recognition approach to differentiate images with and without signs of retinopathy, a top-down approach.
4.7 Review of All Scientific Studies of Automated Methods for Mass Screening
Table 4.1 contains all studies of automated methods for diabetic eye disease screening in the scientific literature that have been evaluated on at least 300 people with diabetes [46–51]. Studies that only evaluated the performance of algorithms detecting specific lesions, as opposed to diabetic eye disease overall, were excluded, as were studies evaluating performance on only a small number of people less than 300, studies where the number of people (not images) in the tested populations could not be determined, and studies where the method required assistance by a human expert or where training data and test data were mixed. Also excluded was a study that met all of the above criteria but where some of the methods remain controversial [52, 53].
Table 4.1
Overview of all studies of automated methods for detecting diabetic eye disease in the scientific literature that have been evaluated on at least 300 people with diabetes. The sensitivity, specificity and area under the curve of the reported studies cannot be compared because the study methods were different.
Authors | Automated method | Year | n | Dataset | Protocol | Reference standard | Reference standard protocol | AUC | Sensitivity | Specificity | DME sensitivity | DME specificity |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
Abramoff [14] | Iowa-0 | 2008 | 5,692 | EyeCheck (Netherlands) [46] | 2 × 2 | Single retinal specialist | EyeCheck [46] | 0.84 | 0.84 | 0.64 | ||
Abramoff [15] | Iowa-1 | 2010 | 16,670 | EyeCheck (Netherlands) [46] | 2 × 2 | Single retinal specialist | EyeCheck [46] | 0.84 | 0.90 | 0.48 | ||
Abramoff [15] | Latim | 2010 | 16,670 | EyeCheck (Netherlands) [46] | 2 × 2 | Single retinal specialist | EyeCheck [46] | 0.82 | 0.90 | 0.44 | ||
Abramoff [13] | Iowa-3 | 2013 | 874 | Messidor-2 (France) | 2 × 1 | 3 retinal specialists adjudicated | ICDR [25] | 0.94 | 0.97 | 0.59 | 1.00 | |
Agurto [45] | New Mexico | 2010 | 822 | Texas (USA) | 2 × 3 | Single retinal specialist | Texas | 0.89 | 0.92 | 0.50 | 1.00 | |
Dupas [47] | Messidor-1 | 2010 | 716 | Messidor-1 (France) | 1 × 1 | Single retinal specialist | Messidor | 0.92 | 0.76 | 0.73 | 0.71 | |
Fleming [16] | Aberdeen-1 | 2010 | 33,535 | Grampian (UK) | 2 × 1 | NSP | NSP [30] | 1.00 | 0.41 | 0.97 | ||
Goatman [17] | Aberdeen-2 | 2011 | 8,271 | South London (UK) | 2 × 2 | NSP | NSP [30] | 0.99 | 0.26 | 0.99 | ||
Jelinek [48] | Australia | 2006 | 385 | Australia (AUS) | 2 × 1 | Clinic (single clinician) | NSP [30] | 0.93 | 0.85 | 0.90 | ||
Philip [49] | Aberdeen-0 | 2007 | 6,722 | Grampian (UK) | 2 × 1 | NSP | NSP [30] | 0.91 | 0.67 | |||
Sanchez [50] | Iowa-2
 Stay updated, free articles. Join our Telegram channel
Full access? Get Clinical Tree
Get Clinical Tree app for offline access
|