Retinopathy of prematurity (ROP) is a multifactorial disease of preterm infants that is characterized by the abnormal vascular development of the immature retina. The International Classification of Retinopathy of Prematurity, the established guideline for accurate diagnosis of ROP, is widely accepted as the standard for ROP assessment. The International Classification of Retinopathy of Prematurity uses a number of criteria to describe ROP, like the severity of the disease (stage 0–5), location of the disease (zone I, II, III), circumferential extent of the disease, and the presence of plus disease. Plus disease is defined as a clinically significant level of vascular dilation and tortuosity at the posterior pole of the retina. Aggressive posterior ROP describes a special kind of ROP with rapid disease progression starting from the posterior pole of the retina. According to the guidelines from the Early Treatment for Retinopathy of Prematurity trial, treatment is now considered for zone I disease with any stage ROP with plus disease, zone I disease with stage 3 ROP with or without plus disease, or zone II disease with stage 2 or 3 ROP with plus disease. Following these objective guidelines can help the examiner to make the right diagnosis and treatment decision for ROP. However, the current guidelines do not consist of quantifiable measures but describe morphologic signs. ROP diagnosis depends on examiners’ subjective interpretations of these signs at clinical examinations. Consequently, examiner-dependent variability in ROP diagnosis and treatment decisions can be assumed. This is supported by population-based studies, which demonstrated pronounced differences in examiners’ findings of the incidence of ROP and the frequency of treatment in premature infants in nations with similar healthcare systems and ROP screening programs and among centers in the same neonatal network.

Methods

We conducted an international multicenter study designed as a prospective intra- and inter-rater reliability analysis to assess inter-expert and intra-expert agreement on the diagnosis and treatment of ROP. The study protocol was approved prospectively by the Ethics Committee of the Medical University of Vienna, Austria. The research followed the tenets of the Declaration of Helsinki.

Experts

Seven experienced pediatric ophthalmology consultants specializing in ROP from 4 different centers for pediatric ophthalmology participated in this study. All centers were university hospitals located in Austria, Germany, and Croatia and were equipped with a neonatal intensive care unit (NICU). Eligible experts had to meet the following criteria: (1) clinical experience of ≥5 years in screening, diagnosis, and treatment of ROP; (2) expertise in wide-field digital imaging systems like the Retcam device; and (3) authorship or co-authorship of at least 5 pediatric ophthalmic or vitreoretinal papers published in peer-reviewed journals.

Study Population

All wide-field digital images of premature infants were obtained from the database of the routine ROP in-patient screening program at the Department of Ophthalmology of the Medical University of Vienna, Austria. The images were taken with a Retcam 3-fundus camera system with a 130-degree children’s lens (Clarity Medical Systems, Pleasanton, California, USA). All patients in the database were initially included into the study. They were subsequently excluded if they failed to meet any of the following inclusion criteria: (1) prematurity with a gestational age of <32 weeks and/or a gestational weight of ≤1500 grams, (2) no retinal diseases other than ROP, and (3) availability of 5 high-quality wide-field digital images with coverage of the posterior pole and all 4 quadrants of the retina. The quality and coverage of images was checked by the project coordinator, a pediatric ophthalmologist specialized in ROP who was not involved in the grading/staging process. In total, 52 patients each with 5 pictures, resulting in 260 high-quality wide-field digital images, were ultimately included.

Rating Procedure

The fundus images were presented to the experts for examination via a standardized web-based database. Five images of 1 selected eye of each patient covering all 4 quadrants and the posterior pole of the retina were presented. All personal patient data were masked in all the images and the experts were masked to the patients’ clinical information. Patients were identified by a randomized patient identification number only. After examining the pictures experts were asked to evaluate each patient according to the International Classification of Retinopathy of Prematurity for the categories ROP stage (0–5), presence of plus disease (2-level categorization; plus or not plus), and presence of aggressive posterior ROP (aggressive posterior ROP or no aggressive posterior ROP). Furthermore, the experts were asked to evaluate each patient on the necessity for treatment (yes or no) according to the criteria set by the Early Treatment for Retinopathy of Prematurity randomized trial. If experts answered “yes,” they were asked to suggest treatment (diode laser, anti–vascular endothelial growth factor treatment, vitreoretinal surgery). Examinations took place on 2 consecutive assessment days 8 weeks apart. Experts were asked to evaluate the images/patients on assessment day 1 and reevaluate them on assessment day 2 (test to retest). Images were presented in a random order on both days. Figure 1 provides an example of a patient presented to the experts. Supplemental Figures 1–3 show additional patient examples (Supplemental Material available at AJO.com ).

Example of an image set of a patient (identification number 36) presented in this study. The 7 experts rated this patient as follows: retinopathy of prematurity stage 0 (0), 1 (0), 2 (0), 3 (6), 4 (0), 5 (0); plus disease yes (6), no (1); aggressive posterior retinopathy of prematurity yes (6), no (1); necessity for treatment yes (7), no (0); kind of treatment diode laser (1), anti–vascular endothelial growth factor therapy (6), vitreoretinal surgery (0).

Statistical Analysis

The data were analyzed using SPSS Version 18 (SPSS Inc, Chicago, Illinois, USA), Microsoft Excel 2007 (Microsoft Corp, Seattle, Washington, USA), and the Cohen and Fleiss Kappa Program (StatsToDo, Brisbane, Australia). Demographic data values were expressed as the mean ± standard deviation and range. Multi-expert Fleiss’ kappa was used for measuring inter-expert agreement. Intra-expert agreement was measured using Cohen’s weighted kappa statistic for continuous variables and unweighted kappa statistic for dichotomous variables. Kappa values range from −1.00 (complete disagreement) to 1.00 (complete agreement) with a value of 0.00 defined as no more agreement than can be expected by chance. A value of 0.00-0.20 is accepted as slight agreement, 0.21-0.40 as fair agreement, 0.41-0.60 as moderate agreement, 0.61-0.80 as substantial agreement, and 0.81-1.00 as almost perfect agreement. In addition, percentage values for the inter-expert and intra-expert absolute agreement were provided.