Purpose
To measure accuracy and speed for detection of vascular progression in retinopathy of prematurity (ROP) from serial images. Two strategies are compared: static side-by-side presentation and dynamic flickering of superimposed image pairs.
Design
Prospective comparative study.
Methods
Fifteen de-identified, wide-angle retinal image pairs were taken from infants who eventually developed plus disease. Image pairs representing vascular disease progression were taken ≥1 week apart, and control images without progression were taken on the same day. Dynamic flickering pairs were created by digital image registration. Ten experts independently reviewed each image pair on a secure website using both strategies, and were asked to identify progression or state that images were identical. Accuracy and speed were measured, using examination date and ophthalmoscopic findings as a reference standard.
Results
Using static images, experts were accurate in a mean (%) ± standard deviation (SD) of 11.4 of 15 (76%) ± 1.7 image pairs. Using dynamic flickering images, experts were accurate in a mean (%) ± SD of 11.3 of 15 (75%) ± 1.7 image pairs. There was no significant difference in accuracy between these strategies ( P = .420). Diagnostic speed was faster using dynamic flickering (24.7 ± 8.3 seconds) vs static side-by-side images (40.3 ± 18.3 seconds) ( P = .002). Experts reported higher confidence when interpreting dynamic flickering images ( P = .001).
Conclusions
Retinal imaging provides objective documentation of vascular appearance, with potentially improved ability to recognize ROP progression compared to standard ophthalmoscopy. Speed of identifying vascular progression was faster by review of dynamic flickering image pairs than by static side-by-side images, although there was no difference in accuracy.
Retinopathy of prematurity (ROP) is a leading cause of childhood blindness throughout the world. In the United States, the incidence of ROP is over 65% in infants with birth weight <1251 grams, resulting in an estimated 600 cases of infancy-acquired blindness annually. Meanwhile, the number of infants at risk for ROP continues to rise as neonatal survival rates improve. Standard ROP management includes dilated ophthalmoscopic examination at the neonatal intensive care unit bedside, with hand-drawn sketches to document retinal findings.
An international classification system has standardized the clinical diagnosis of ROP. One component of this classification system is plus disease, which is defined as arteriolar tortuosity and venous dilation within the posterior pole. The multicenter Cryotherapy for ROP (CRYO-ROP) and Early Treatment for Retinopathy of Prematurity (ETROP) studies have established that presence of plus disease warrants treatment with cryotherapy or laser photocoagulation. Accurate assessment of plus disease is therefore essential for ROP management. Moreover, the 2005 revised international classification of ROP defined an intermediate “pre-plus” condition. These facts underscore that ROP represents a spectrum of retinal vascular abnormalities, and that identification of vascular progression toward plus disease is critical for management.
However, vascular disease progression in ROP is often difficult to recognize with precision during serial ophthalmoscopic examinations. In particular, standard methods of documentation using retinal drawings are qualitative and imprecise regarding the appearance of retinal vessels, as well as the nature and location of peripheral disease. This essentially forces examiners to rely on memory to recognize changes in appearance, and creates problems when serial examinations are performed by different ophthalmologists. In addition, studies have shown that plus disease diagnosis may in itself be subjective, even by experts.
Digital retinal photography has potential to improve ROP management by objective documentation of retinal findings, by identification of disease progression toward plus disease through comparison with images from previous examinations, and by creating opportunities for remote telemedicine diagnosis and second opinions from experts throughout the world. The ability to recognize vascular changes accurately and quickly would provide important advantages over traditional methods using serial ophthalmoscopy. The purpose of this study is to investigate the accuracy and speed of identifying progression in severity of retinal vascular changes from serial images of infants who eventually developed plus disease. Specifically, 2 strategies for comparing serial images are compared: 1) side-by-side static image pairs; and 2) dynamic flickering image pairs generated by digital image registration, in which the images are aligned to overlay one another under the premise that this may improve recognition of subtle retinal vascular changes.
Methods
Image Selection and Processing
Images were captured during routine ROP ophthalmoscopic examinations from January 2005 to November 2010 at Columbia University and Weill Cornell Medical Center. Retinal images were obtained using a commercially available camera (RetCam; Clarity Medical Systems, Pleasanton, California, USA) by a trained neonatal nurse, vitreoretinal fellow, or attending ophthalmologist on the same days that ophthalmoscopic examinations were performed. A set of 15 de-identified wide-angle retinal image pairs from infants who eventually developed plus disease as diagnosed by an expert ROP examiner (R.V.P.C., M.F.C.) were selected for the study. Image pairs representing plus disease progression were taken at least 1 week apart. Vascular disease progression did not require a change from “pre-plus” to “plus” (or “neither pre-plus nor plus” to “pre-plus”) by clinical diagnosis to be included in the study. In other words, “vascular disease progression” was defined for study purposes as any interval change in infants who were eventually diagnosed with plus disease during routine clinical ophthalmoscopic examination. Image pairs representing “controls” with no vascular disease progression were taken on the same day.
The same 15 image pairs were presented as 2 side-by-side static images and dynamic flickering images ( Figure , and Supplemental Video , available at AJO.com ). Dynamic flickering image pairs were created using digital registration software, which aligned serial photographs to the sub-pixel level using an algorithm to identify features such as the intersections of vessels and other corresponding peripheral retinal features (MatchedFlicker V1.2; EyeIC, Narberth, Pennsylvania, USA). Dynamic images were displayed at a flicker rate of 2 Hz.
Image Interpretation by Study Participants
Eligible expert study participants were defined as practicing pediatric ophthalmologists or retina specialists who met at least 1 of 3 criteria: having been a study center principal investigator for the CRYO-ROP or ETROP studies, having been a certified investigator for either study, or having co-authored 5 or more peer-reviewed ROP manuscripts.
All images were uploaded to a secure web server. A web interface developed by the authors was used to display retinal images and collect responses, which were recorded to a secure database system (SQL Server 2005; Microsoft, Redmond, Washington, USA). Each participant who agreed to provide informed consent was given an electronic link to the study website, along with an individual secure login and password.
Participants were oriented to the study website with a 1-page instruction guide developed by the authors. ROP experts independently reviewed each image pair on the study website. The 15 image pairs were initially presented sequentially as side-by-side static images, and subsequently as dynamic flickering image pairs in the same sequence. After each image pair was displayed, participants were asked to provide 1 of 3 mutually exclusive responses with regard to ROP vascular progression toward plus disease (ie, severity of vascular dilation and tortuosity): “image 1 is more severe,” “image 2 is more severe,” or “images 1 and 2 are from the same day and show exactly the same severity.” The reference standard for vascular disease progression was based on defining the more “severe” image as the one taken on the later date by both expert ROP examiners (R.V.P.C., M.F.C.), since all eyes included in the study were eventually diagnosed with plus disease during routine clinical ophthalmoscopic examination by 1 of 2 ROP examiners (R.V.P.C., M.F.C.).
To analyze speed, the participant response times for each static or dynamic image pair were recorded using a computer time stamp that was incorporated into the study website. Time measurement began when each new website page was loaded to display an image pair, and ended when the expert participant clicked “submit” to finalize his or her response. Finally, experts were asked to report their confidence (“confident,” “somewhat confident,” or “not confident”) for identifying the more severe image in each static or dynamic pair.
Data Analysis
Accuracy of detecting vascular progression of ROP disease severity, or recognizing that there was no progression because images were identical, was measured by comparing responses to the examination date sequence. The numbers of correctly graded image pairs for progression of ROP disease severity were compared between the static side-by-side image pairs and the dynamic flickering image pairs for each expert participant. Grading times and confidence scores for detecting vascular disease progression were compared for static side-by-side image pairs vs dynamic flickering image pairs.
Statistical software was used for data analysis (Excel 2008, Microsoft, Redmond, Washington, USA; SPSS 15.0, SPSS Inc, Chicago, Illinois, USA). Comparisons between static side-by-side image pairs and dynamic flickering image pairs in evaluating ROP vascular progression were analyzed using the paired sample t test, and confidence scores were analyzed using the Pearson χ 2 test. Statistical significance was considered to be a 2-sided P value <.05.
Results
Expert Participants and Overview of Responses
A total of 10 expert participants completed the web-based study. Seven of the 10 experts (70%) were study center principal investigators for the CRYO-ROP or ETROP studies, 3 of 10 (30%) were certified investigators for either study, and 6 of 10 (60%) co-authored 5 or more peer-reviewed ROP papers. Examples of expert responses to interpreting vascular disease progression in static side-by-side and dynamic flickering image pairs are shown in the Figure and the Supplemental Video (available at AJO.com ).
Accuracy
Table 1 summarizes the accuracy of determining ROP progression, or recognizing that there was no change in the images, among the 10 expert participants. For static side-by-side image pairs, experts made an accurate response in a mean (%) ± standard deviation (SD) of 11.4 (76%) ± 1.7 out of 15 image pairs. For dynamic flickering image pairs, experts made an accurate response in a mean (%) ± SD of 11.3 (75%) ± 1.7 out of 15 image pairs. There was no statistically significant difference in the accuracy of identifying progression of ROP disease severity between these 2 strategies (P = .420, paired t test).
| Participant | Accuracy of Responses (%) | |
|---|---|---|
| Static Side-by-Side | Dynamic Flickering | |
| 1 | 10/15 (67%) | 11/15 (73%) |
| 2 | 12/15 (80%) | 13/15 (87%) |
| 3 | 13/15 (87%) | 13/15 (87%) |
| 4 | 13/15 (87%) | 13/15 (87%) |
| 5 | 12/15 (80%) | 10/15 (67%) |
| 6 | 11/15 (73%) | 9/15 (60%) |
| 7 | 10/15 (67%) | 9/15 (60%) |
| 8 | 14/15 (93%) | 13/15 (87%) |
| 9 | 10/15 (67%) | 10/15 (67%) |
| 10 | 9/15 (60%) | 12/15 (80%) |
| Mean (%) ± SD | 11.4 (76%) ± 1.7 | 11.3 (75%) ± 1.7 |
a Serial image pairs were taken from infants who eventually developed plus disease, and the reference standard was based on a diagnosis of more severity in retinal vascular changes in the image taken on the later date by both expert retinopathy of prematurity examiners (M.F.C., R.V.P.C.). There was no statistically significant difference between static and dynamic image presentation strategies ( P = .420, paired t test).
Examined another way, there were 150 corresponding pairs of 15 images that were interpreted by 10 expert participants. Among these corresponding image pair interpretations, 96 of 150 (64%) were accurately graded for progression of ROP disease severity by both the static side-by-side and dynamic flickering strategies, whereas 19 of 150 (13%) were not graded accurately by either strategy. Approximately equal numbers of image pairs were graded correctly by 1 strategy but incorrectly by the other strategy: only the static strategy was correct in 18 of 150 (12%), and only the dynamic strategy was correct in 17 of 150 (11%).
Speed and Confidence
Table 2 summarizes the speed of determining ROP progression, or recognizing that there was no change in the images, among the 10 expert participants. For static side-by-side image pairs, the mean ± SD (range) response time was 40.3 ± 18.3 (16–86) seconds. For dynamic flickering image pairs, the mean ± SD (range) response time was 24.7 ± 8.3 (15–44) seconds. Overall, diagnostic speed was significantly faster by interpretation of dynamic flickering image pairs than by interpretation of static side-by-side image pairs (P = .002, paired t test).
| Participant | Mean Response Time (Seconds) | |
|---|---|---|
| Static Side-by-Side | Dynamic Flickering | |
| 1 | 52 | 28 |
| 2 | 43 | 24 |
| 3 | 26 | 19 |
| 4 | 46 | 22 |
| 5 | 36 | 21 |
| 6 | 36 | 28 |
| 7 | 16 | 15 |
| 8 | 39 | 28 |
| 9 | 86 | 44 |
| 10 | 26 | 18 |
| Mean (SD) | 40.3 (18.3) | 24.7 (8.3) |
a Response times were measured by computer timestamps in a web-based image review system. Diagnostic speed was significantly faster by interpretation of dynamic flickering image pairs than by interpretation of static side-by-side image pairs ( P = .002, paired t test).
With regard to confidence in determining ROP progression by static side-by-side compared to dynamic flickering images, there were 150 corresponding image pairs interpreted by the 10 expert participants. By interpretation of static side-by-side images, 87 of 150 experts (58%) were confident, 52 of 150 (35%) were somewhat confident, and 11 of 150 (7%) were not confident in recognizing disease progression. By interpretation of dynamic flickering image pairs, 116 of 150 experts (77%) were confident, 31 of 150 (21%) were somewhat confident, and 3 of 150 (2%) were not confident in recognizing disease progression. The difference in these confidence level distributions was statistically significant ( P = .001, χ 2 test).
Results
Expert Participants and Overview of Responses
A total of 10 expert participants completed the web-based study. Seven of the 10 experts (70%) were study center principal investigators for the CRYO-ROP or ETROP studies, 3 of 10 (30%) were certified investigators for either study, and 6 of 10 (60%) co-authored 5 or more peer-reviewed ROP papers. Examples of expert responses to interpreting vascular disease progression in static side-by-side and dynamic flickering image pairs are shown in the Figure and the Supplemental Video (available at AJO.com ).
Accuracy
Table 1 summarizes the accuracy of determining ROP progression, or recognizing that there was no change in the images, among the 10 expert participants. For static side-by-side image pairs, experts made an accurate response in a mean (%) ± standard deviation (SD) of 11.4 (76%) ± 1.7 out of 15 image pairs. For dynamic flickering image pairs, experts made an accurate response in a mean (%) ± SD of 11.3 (75%) ± 1.7 out of 15 image pairs. There was no statistically significant difference in the accuracy of identifying progression of ROP disease severity between these 2 strategies (P = .420, paired t test).
