Purpose
To determine the intra-observer and inter-observer reliability of a novel ocular staining score among trained ophthalmologists.
Design
Reliability analysis within a prospective, observational, multicenter cohort study.
Methods
Those enrolled in the National Institutes of Health–funded Sjögren’s International Collaborative Clinical Alliance (SICCA) who presented for follow-up at the University of California San Francisco, Aravind Eye Hospital, Johns Hopkins University, and the University of Pennsylvania were included. Study participants were graded using the ocular staining score by at least 2 masked SICCA-trained ophthalmologists. The primary outcome for this study was the intraclass correlation coefficient (ICC) for the total ocular staining score. ICCs were also calculated for tear break-up time (TBUT) and conjunctival and corneal staining.
Results
Total ocular staining score had an ICC of 0.91 for the right eye (95% confidence interval [CI] 0.85–0.96) and 0.90 for the left eye (95% CI 0.83–0.97). Corneal staining (right eye 0.86, 95% CI 0.76–0.93, left eye 0.90, 95% CI 0.81–0.95) and conjunctival staining (right eye 0.87, 95% CI 0.80–0.93, left eye 0.85, 95% CI 0.75–0.93) demonstrated excellent agreement. The ICC for TBUT was slightly lower (right eye 0.77, 95% CI 0.64–0.89; left eye 0.81, 95% CI 0.68–0.90).
Conclusions
Previous studies have shown that the ocular staining score is correlated with other diagnostic components of Sjögren syndrome. In this study, we demonstrate high reliability in grading among trained ophthalmologists, completing the validation of this test.
Sjögren syndrome is a chronic autoimmune inflammatory disorder characterized by decreased exocrine function of the salivary and lacrimal glands. It can be primary or secondary, when it is related to other autoimmune conditions such as systemic lupus erythematosus or rheumatoid arthritis. Owing to diagnostic challenges and different diagnostic criteria, the exact prevalence of Sjögren syndrome has not been well established. One study, using the American European Consensus Group criteria for diagnosis, found an annual incidence of 4 per 100 000; 70% of these cases were primary. The American College of Rheumatology currently recommends that a classification of Sjögren syndrome be made based on meeting 2 of the following criteria: (1) positive serology for anti-SSA and/or anti-SSB; (2) ocular staining score of ≥3; (3) presence of focal lymphocytic sialadenitis with focus score ≥1 focus/4 mm 2 .
The Sjögren’s International Collaborative Clinical Alliance (SICCA) is an NIH-funded group of investigators who are exploring the etiology, diagnosis, epidemiology, and treatment of Sjögren syndrome in a large prospective cohort. In order to characterize keratoconjunctivitis (KCS) associated with Sjögren syndrome, a new quantitative ocular staining score was developed and tested in this cohort. In a prior study in this cohort, abnormal ocular staining score was strongly associated with other features of Sjögren syndrome. The reproducibility of dry eye measurements has been debated. In one study of a single grader who repeated Schirmer test, tear break-up time, and cotton-thread test measurements on different days, the repeatability was poor.
In this study, we evaluate the inter-grader and intra-grader reliability of the ocular staining score among SICCA-trained ophthalmologists examining the same patient on the same day. If the ocular staining score is shown to be reliable, it may serve as a useful diagnostic tool in the evaluation of patients with both Sjögren syndrome–related and non–Sjögren syndrome–related KCS.
Methods
Study participants who presented for their Sjögren’s International Collaborative Clinical Alliance (NIH N01 DE32636) study visit at University of California San Francisco, Aravind Eye Hospital, Johns Hopkins University, or University of Pennsylvania were included. The methods of the cohort study, as well as the development of the ocular staining score and standardized training of all involved ophthalmologists, have been outlined previously. Five ophthalmologists participated in the reliability analysis. In accordance with the SICCA study, participants were asked not to apply their routine drops for the 12 hours preceding their visit and to discontinue contact lens wear for at least 7 days prior to examination. Two to 3 ophthalmologists scored each patient and were masked to each other’s ocular staining score grading. Institutional Review Board approval was obtained from all participating institutions for the Sjögren’s International Collaborative Clinical Alliance prospective cohort study, including this reliability analysis.
During the examination, study participants had tests and examination in the following order: unanesthetized Schirmer test, slit-lamp examination assessing tear break-up time (TBUT) with fluorescein, and punctate epithelial erosions (PEEs) of the cornea with fluorescein and conjunctival staining patterns with lissamine green. Lid, conjunctival, and corneal abnormalities were also noted. A subset of participants also had osmolarity using TearLab (TearLab Corporation, San Diego, California, USA). Each parameter was graded sequentially by the different investigators, who remained masked to each others’ grading.
Schirmer test I was performed prior to all other tests and before any drops were instilled in the eye. As soon as both Schirmer strips were in place the strips remained in place a maximum of 5 minutes or until completely saturated. Fluorescein (0.5% preservative free, Leiter’s Pharmacy, San Jose, California, USA) was instilled immediately after removing the Schirmer strips. The cornea was examined at the slit lamp using 10× magnification and illumination set on “high” with the cobalt blue filter between 4 and 8 minutes after instillation. TBUT was defined as the time in seconds between the patient’s last blink and the first appearance of a dry spot on the corneal surface. It was measured 3 times and the mean value was recorded. A value of 10 or greater was considered normal. Participants were then given a score of 0 if there were no PEEs, 1 if there were 1–5 PEEs, 2 if there were 6–30 PEEs, and 3 if there were more than 30. They were then given an additional point each if they had patches of confluent staining, staining in the pupillary area, or 1 or more filaments, for a maximum possible corneal staining score of 6.
After detailed assessment for lid, conjunctival, and corneal abnormalities, lissamine green (1 drop of 1% lissamine green dye; Leiter’s Pharmacy) was instilled in the eye. The conjunctival staining was then immediately assessed with 10× magnification with white illumination through a neutral-density filter. Interpalpebral nasal and temporal conjunctiva were assigned a grade of 0 for no staining, 1 for 10–32 dots, 2 for 33–100 dots, and 3 for greater than 100 dots. These values were then summed for a maximum possible score of 6 for conjunctival staining. The total ocular staining score was determined by adding the conjunctival and corneal staining with scores ranging from 0 to 12 for each eye ( Supplemental Figure , available at AJO.com ).
Inter-rater agreement was assessed using the intraclass correlation coefficient (ICC), estimated using analysis of variance methods for unbalanced data. Separate ICCs were estimated for the total ocular staining score, ocular staining score between the 2 eyes of an individual patient, and individual parts of the examination including TBUT, conjunctival lissamine staining pattern, and corneal fluorescein staining pattern. Precision of ICC estimates were summarized using percentile-based 95% bootstrap percentile confidence intervals based on resampling (n = 999 resamples) drawn at the level of the patient to account for clustering of responses. Calculations were performed in Mathematica 10.0 (Wolfram Research, Champaign, Illinois, USA).
Results
Ninety-eight eyes of 49 study participants were evaluated and scored at the University of California San Francisco, Aravind Eye Hospital, Johns Hopkins University, or University of Pennsylvania by at least 2 SICCA ophthalmologists between March 2011 and September 2012. One eye of 1 patient was excluded owing to missing data. Table 1 outlines the characteristics of the study participants included. The majority of participants were female (n = 41, 84%) with a median age of 57 (interquartile range [IQR] 51–67). Sixteen (33%) met criteria for a diagnosis of Sjögren syndrome using criteria defined by the American College of Rheumatology. Twenty-nine percent (n = 14) were positive for anti-Sjögren syndrome–related antigen A (anti-SSA) or anti–Sjögren syndrome–related antigen B (anti-SSB), while 20% (n = 10) were positive for rheumatoid factor (RF).
| Characteristic | Number | % | Median (IQR) |
|---|---|---|---|
| Age, y | 57 (51–67) | ||
| Sex, female | 41 | 84 | |
| Serology positive for anti-SSA/SSB | 14 | 29 | |
| RF positivity | 10 | 20 | |
| Focus score ≥1 | 10 | 26 | |
| Meets ACR criteria for Sjögren syndrome | 16 | 33 | |
| Schirmer mm/5 min, right eye | 9 (5–16) | ||
| Shirmer mm/5 min, left eye | 8 (5–18) | ||
| TBUT <10 s, both eyes | 15 | 30 | 5 (4–7) |
| Ocular staining score (abnormal ≥3), right eye | 5 (2–8) | ||
| Ocular staining score (abnormal ≥3), left eye | 5 (3–8) |
Stay updated, free articles. Join our Telegram channel
Full access? Get Clinical Tree
