Purpose
“Spin,” a slang term meaning the misrepresentation of study findings such that the beneficial effects of an intervention are magnified beyond what the results actually show, is a reporting practice that has been shown to influence perceptions of treatment efficacy and clinical decision making. The extent of spin and its complications were evaluated in the abstracts of systematic reviews of cataract surgery. The issue of whether particular study attributes were associated with spin was also evaluated.
Design
Cross-sectional study.
Methods
MEDLINE and Embase were searched for systematic reviews and meta-analyses relating to cataract treatment. From these search records, eligible studies were screened in duplicate. A previously developed classification system for spin was used to assess the systematic reviews that met eligibility criteria for the occurrence of the 9 most severe forms of spin. Evaluation of spin, extracted study characteristics, and appraisal of the methodological quality of each study were performed using the 16-question A MeaSurement Tool to Assess systematic Reviews (AMSTAR-2) scale in duplicate.
Results
Searches retrieved 2,059 studies, of which 110 were eligible for data extraction. At least 1 form of spin was found in 30.0% of included systematic reviews (33 of 110). Six of the 9 types of spin were identified in the study sample, the most common being type 3 in 18.2% (20 of 110) of abstracts. No significant associations were found among spin in abstracts, AMSTAR-2 appraisal, and any of the extracted study characteristics.
Results
Searches retrieved 2,059 studies, of which 110 were eligible for data extraction. At least 1 form of spin was found in 30.0% of the included systematic reviews (33 of 110). Six of the 9 types of spin were identified in the sample, the most common being type 3 in 18.2% (20 of 110) of abstracts. No significant associations were found among spin in abstracts, AMSTAR-2 appraisals, and any of the extracted study characteristics.
Conclusions
Spin was evident in approximately one-third of the abstracts of evaluated systematic reviews and meta-analyses of cataract surgery and associated complications.
Cataracts are the leading cause of blindness worldwide, affecting 20 million people. According to the National Eye Institute’s most recent estimate, in 2010, 24.4 million Americans had cataracts. Given the aging population, this estimate is expected to almost double by 2030 and to reach 50 million by 2050. Risk for cataracts increases with each decade after the age of 40 years. An estimated 70% of white Americans have cataracts by age 80, compared with 61% of Hispanic Americans and 53% of African Americans. As the prevalence of cataracts has increased with rising life expectancy, cataract surgery has become one of the most commonly performed operations worldwide.
The field of cataract surgery continues to evolve, given the sheer volume of procedures combined with advances in medicine and technology. Accordingly, research is conducted regularly to improve the efficacy and efficiency of surgical procedures for cataracts. Most surgeries are now performed in less than 15 minutes with a success rate of 90%-95%.
To guide ophthalmic surgeons in their treatment decision making and techniques, clinical practice guidelines must be updated at regular intervals to reflect the latest research findings and ensure optimal patient care. Systematic reviews are considered the gold standard for developing evidence-based practice in health care. While conducting these reviews, investigators pose specific research questions, gather evidence from all relevant studies (based on pre-specified eligibility criteria), and provide a synthesis and high-level overview of their findings.
Despite the high success rate of cataract surgery, some risk of complications, ranging from pain to permanent loss of vision, remain. In terms of knowledge transfer and exchange, because many physicians, especially those practicing in rural areas, lack access to medical journals and medical literature, they often rely solely on article abstracts to guide their decision making. It is therefore imperative that systematic reviews and particularly their abstracts are methodologically sound and objective.
A subjective method of reporting results, known as “spin,” is defined as “a specific way of reporting, intentional or not, to highlight that the beneficial effect of the experimental treatment in terms of efficacy or safety is greater than that shown by the results.” Yavchitz and associates have identified many different types of spin, of which 21 were specific to the abstract of systematic reviews. Based on spin severity, the top 9 findings are critical to the interpretation of medical literature . In the medical literature, and especially in published systematic reviews, spin can be particularly problematic because it can lead to misinformed decision making and may result in suboptimal patient care. This study evaluated the abstracts of systematic reviews and meta-analyses of cataract surgery for the 9 types of spin identified by Yavchitz and associates.
METHODS
Search Strategy
For this cross-sectional study, a systematic review librarian created the search strategy using the MEDLINE (Ovid) and Embase (Ovid) databases ( Figure 1 ). Meta-analyses and systematic reviews of cataract treatment were selected for the search by filtering search results for systematic reviews and meta-analyses. Additionally, we excluded studies that did not meet the inclusion criteria for a systematic review or meta-analysis. The librarian conducted the search on June 1, 2020, and exported the articles to Rayyan a screening program (Rayyan, Cambridge, Massachusetts, USA). Two authors (E.S. and S.D.) screened the articles based on their titles and abstracts, in a masked, duplicate fashion. The 2 authors (E.S. and S.D.) removed duplicates and resolved disagreements. A third author (R.O.) was called upon if disagreements could not be resolved.
Following the suggestions of a reviewer, we screened the 414 cataract-related articles in the Cochrane Eyes and Vision (CEV) database for articles that could be added to the review. Of the 414 cataract-related articles, 124 were within the initial screen of 180 articles. The remaining 290 were missing from the study’s database, 3 of which were previously excluded. The remaining 287 articles were additionally screened for inclusion or exclusion. Twelve of the 287 articles were determined to fit the study’s criteria for inclusion; however, the other 275 articles did not qualify under the inclusion criteria and were most commonly excluded for the following reasons: not a treatment, not cataract surgery therapy, not cataracts-related, not within date limitations, and abstract only articles.
Eligibility Criteria
Articles were included based on the following criteria: 1) a systematic review with or without a meta-analysis; 2) language restricted to English; 3) article related to the treatment of diseases caused by cataracts. Additionally, it was determined if systematic reviews or meta-analyses met the Preferred Reporting Items for Systematic Reviews and Meta-Analyses Protocol (PRISMA-P) guidelines of writing and reporting.
Training
To acquire the skills needed evaluate each systematic review and meta-analysis, authors E.S. and S.D. underwent training in systematic reviews, spin, and the use of A MeaSurement Tool to Assess systematic Reviews (AMSTAR-2) before screening the article titles and abstracts. Initially, they completed an online course about systematic reviews and meta-analyses conducted by Li and Dickersin. The authors used this course to gain a better understanding of the studies, which contributed to accurate data extraction. Next, they participated in 2 days of spin training, both online and in person. Training consisted of an analysis of the 9 most severe types of spin as described by Yavchitz and associates ( Table 1 ). The definitions provided were strictly adhered to, and data were analyzed within the text, such as forest plots, confidence intervals, and P values to determine the accuracy of the data presented in the abstract. Finally, the authors were trained to use AMSTAR-2, a tool used in evidence-based health research to evaluate the methodological quality of systematic reviews and meta-analyses. A detailed outline of the training components can be found in the study protocol, which is available on the Open Science Framework.
| 9 Most Severe Types of Spin | No. (%) Containing Spin (N = 110.0) |
|---|---|
| 1) Conclusion contains recommendations for clinical practice not supported by the findings. | 4.0 (3.6) |
| 2) Title claims or suggests a beneficial effect of the experimental intervention not supported by the findings. | 0.0 (0) |
| 3) Selective reporting of or overemphasis on efficacy outcomes or analysis favoring the beneficial effect of the experimental intervention. | 20.0 (18.2) |
| 4) Conclusion claims safety based on nonstatistically significant results with a wide confidence intervals. | 4.0 (15.4) a |
| 5) Conclusion claims the beneficial effect of the experimental treatment despite high risk of bias in primary studies. | 9.0 (8.2) |
| 6) Selective reporting of or overemphasis on harm outcomes or analysis favoring the safety of the experimental intervention. | 1.0 (0.9) |
| 7) Conclusion extrapolates the review’s findings to a different intervention (ie, claiming efficacy of one specific intervention although the review covers a class of several interventions). | 1.0 (0.9) |
| 8) Conclusion extrapolates the review’s findings from a surrogate marker or a specific outcome to the global improvement of the disease. | 0.0 (0) |
| 9) Conclusion claims the beneficial effect of the experimental treatment despite reporting bias. | 0.0 (0) |
a Safety was not mentioned in 85 articles, therefore n = 26.0.
Data Extraction
Working in a masked, duplicate fashion, authors E.S. and S.D. extracted the following items from each systematic review and meta-analysis using a pilot-tested Google form: 1) the intervention type (surgery, nonpharmacologic, pharmacologic, combination, other); 2) the funding source (industry, private, public, none, not mentioned, combination including industry, combination not including industry); 3) the date the review was received for publication; 4) whether the review complied with the PRISMA , guidelines for abstracts; 5) whether the submission guidelines for the journals in which the articles were published required adherence to PRISMA; and 6) the journal’s 5-year impact factor. After data extraction, the 2 authors were unmasked, and they met to discuss and resolve any disagreements. A third author (R.O.) was consulted to adjudicate any discrepancies that E.S. and S.D. were unable to resolve.
The same 2 authors assessed the abstract of each included systematic review in depth for the 9 most severe types of spin. The presence of spin was categorized as a binary variable if 1, spin was present, and 0 were identified. Next, the same authors evaluated the quality of each systematic review as high, moderate, low, or critically low by using the 16-question AMSTAR-2 scale. Inter-rater concordance of AMSTAR-2 scores was moderate to high across studies. Construct validity was high with both the original AMSTAR instrument (r = 0.91) and the Risk of Bias in Systematic Reviews instrument (r = 0.84), indicating the validity of the tool and its ability to make accurate inferences based on the studies in question.
Statistical Analysis
Descriptive statistics were used to distinguish overall spin frequency and its subtypes. All results are presented as both frequency counts and percentages. Stata version 16.1 software (StataCorp, College Station, Texas, USA) was used for all analyses. Due to the sample size not meeting our previously established target size for a multivariate regression, binary logistic regression was conducted to evaluate the association of spin with study characteristics, including intervention type, adherence to PRISMA guidelines, journal requirement for PRISMA guidelines, funding source, AMSTAR-2 rating, and journal impact factor. Odds ratios for the study characteristics extracted were calculated for subsequent categories of the reference groups as shown in Table 2 . All analytic decisions are documented in this study’s protocol.
| Characteristics | No. (%) of Articles (n = 110.0) | ||
|---|---|---|---|
| Total (%) | Abstract Contains Spin | Odds Ratio (95% CI) | |
| Intervention Type | |||
| Surgery | 74.0 (67.27) | 26.0 (23.6) | 1.0 [Reference] |
| Pharmacology | 33.0 (30.0) | 8.0 (7.27) | 0.59 (0.23-1.49) |
| Education | 1.0 (0.91) | 0.0 | – |
| Non-pharmacologic | 2.0 (1.82) | 0.0 | – |
| Article mentions adherence to PRISMA | |||
| No | 84.0 (76.4) | 23.0 (20.9) | 1.0 [Reference] |
| Yes | 26.0 (23.6) | 11.0 (10.0) | 1.94 (0.78-4.85) |
| Publishing journal recommends adherence to PRISMA | |||
| No | 61.0 (55.5) | 13.0 (11.8) | 1.0 [Reference] |
| Yes | 49.0 (44.5) | 21.0 (19.1) | 2.77 (1.20-6.38) |
| Funding source | |||
| Not funded | 27.0 (24.5) | 9.0 (8.2) | 1 [Reference] |
| Industry | 4.0 (3.64) | 1.0 (0.91) | 6.67 (0.06-7.35) |
| Not mentioned | 22.0 (20.0) | 5.0 (4.5) | 0.59 (0.16-2.11) |
| Private | 22.0 (20.0) | 6.0 (5.5) | 0.75 (0.22-2.57) |
| Public | 35.0 (31.82) | 13.0 (11.8) | 1.18 (0.41-3.39) |
| AMSTAR-2 rating | |||
| High | 2.0 (1.82) | 1.0 (0.91) | 1.0 [Reference] |
| Moderate | 27.0 (24.5) | 10.0 (9.1) | 0.59 0.03-10.48) |
| Low | 27.0 (24.5) | 6.0 (5.5) | 0.33 (0.18-6.19) |
| Critically low | 57.0 (51.8) | 17.0 (15.5) | 0.43 (0.03-7.20) |
| Mean ± SD journal impact factor | 3.90 (3.64) | 3.78 (2.54) | 0.99 (0.87-1.12) |
| Publication year (1993-2020) | 1.00 (0.94-1.08) | ||
RESULTS
General Characteristics
Database searches yielded 2,059 records, from which 556 duplicates were removed, leaving 1,522 unique studies that were subjected to title and abstract screening. During this initial screening, 1,878 studies were excluded that did not meet the eligibility criteria. An additional 71 articles were excluded during full-text analysis. Thus, a total of 110 systematic reviews and meta-analyses met the eligibility criteria. Figure 2 shows the screening and exclusion processes and rationale.
