Evidence-based medicine suggests that a meta-analysis with a systematic literature review provides the highest level of evidence about a new treatment or association. Clinicians and researchers use these reviews to improve clinical care and develop new research hypotheses. For these reasons, the American Journal of Ophthalmology (the Journal ) welcomes these studies. However, most clinicians and researchers understand that not all meta-analyses are created equal. In this editorial, we will highlight key considerations for authors to put the “metal” into a well-prepared and successful meta-analysis and the new guidelines for the Journal describing the process for accepting these studies.

First of all, what is a meta-analysis with a systematic review? A systematic review “aims to comprehensively locate and synthesize research that bears on a particular question, using organized, transparent, and replicable procedures at each step in the process.” Meta-analysis is “a set of statistical methods for combining quantitative results from multiple studies to produce an overall summary of empirical knowledge on a given topic.” As you can see from these definitions, one of the first (and probably most important) considerations is that the meta-analysis needs a systematic literature search that includes all relevant studies (even those that may be unpublished) and represents the population of interest. These studies are analogous to individual participants in a typical research study. Of course, typical research cannot include all possible participants, because participation is generally constrained by geography (researchers needing access to participants) and resources (a study is not infinitely large). However, the compilation of studies and their participants should be complete and represent the population of interest to allow the study results to be generalizable to a population at large (ie, to have external validity).

We will illustrate the importance of a thorough, nonbiased literature review with an absurd example. Assume that there is a treatment (called WXYZ treatment) that works well for blue eyes and poorly for others, which was developed and researched in Scandinavia for several years. A meta-analysis of these results would not be generalizable to everyone and may even be misleading, since this research was not performed on diverse populations. Similarly, a meta-analysis may contain bias even when it includes diverse studies if the author of the meta-analysis only includes studies with longer follow-up and most of these longer follow-up studies were only completed in blue-eyed participants because the original, longest-standing research was performed in Scandinavia. The longer follow-up might, at first, appear to be a neutral inclusion criterion, but it created selection bias. In other words, a meta-analysis should use thorough systematic literature searches and unbiased inclusion/exclusion criteria so that it is generalizable and should include an online appendix describing included and excluded studies and the reasons for each.

A second consideration is that a meta-analysis must be formulated around a key clinical question that will impact clinical care or researchers. For example, clinicians and researchers would not use a new meta-analysis demonstrating that increased blood sugar leads to increased diabetic retinopathy. These data have already been demonstrated over decades and do not need a meta-analysis. So, the authors’ first question to themselves is: “why is this important, and how will researchers and clinicians use the results?” If the implications are limited, the authors should consider a different topic.

The third consideration is that the meta-analysis must address a key clinical question that has not been addressed previously. One of our mentors suggested that “an hour in the library will save a month in the lab.” Similarly, some authors have spent many hours writing a meta-analysis on a topic that has already been published. So, the next question authors need to answer is why society needs a new meta-analysis and how this will challenge and/or change current knowledge or clinical care.

A fourth consideration is to avoid confounding. While a major reason for performing a meta-analysis is to increase statistical power by combining results of several studies, this achieves no useful purpose if the included studies (and, by proxy, the meta-analysis) do not adjust for important confounders. A meta-analysis may show that high blood pressure is protective of developing glaucoma but neglect to adjust for oral beta-adrenergic antagonists (which treat systemic hypertension and also lower intraocular pressure). Overall, a researcher should investigate for confounding in the studies, and exclude them or adjust for them.

A fifth consideration is that the outcome of interest must be clinically significant, not just statistically significant. To be clinically significant, the outcome of interest must have a reasonable effect size. For example, a meta-analysis may not be valuable to clinicians and researchers if it shows that an intervention (eg, drinking coffee) reduced the risk of glaucoma by 0.1% ( P < .001). Would you counsel your patient to drink more coffee based on this meta-analysis? Overall, a valuable meta-analysis would combine underpowered studies to substantiate a large treatment effect for clinicians and researchers.

Also, researchers should design meta-analyses prospectively before they identify potential studies. Retrospective research on patients often occurs based on clinical experience that did not exist until the patients were treated. But it is inexcusable to include/exclude studies for a meta-analysis after the studies were identified when their characteristics and results are known, because this can lead to intentional, or even unintentional, selection bias that includes studies that favor a specific outcome and excludes others that do not. Patients are treated by clinicians as a matter of course, but comprehensive literature searches are intentional. So although some prior familiarity with the relevant literature would be expected, inclusion/exclusion criteria for a meta-analysis must be documented in advance.

So it is critical that any meta-analysis be based on a comprehensive, systematic literature search with inclusion/exclusion criteria that are clinically relevant and determined in advance, before the literature search is performed. If you cannot perform a comprehensive literature search and synthesize all relevant results (meta-analysis), then the Journal is not interested in your manuscript. It just doesn’t have enough “metal,” thus the likelihood of bias, intentional or not, is unacceptably great.

Therefore, the Journal is implementing a revised process for considering systematic reviews/meta-analyses for publication, with 2 major requirements: the Journal will only accept meta-analyses that have been approved in advance, and all submitted meta-analyses must be based on a systematic literature search. The American Journal of Ophthalmology Guide for Authors (available at http://www.ajo.com/content/authorinfo ) has been updated to reflect this with instructions for meta-analysis (available at http://cdn.elsevier.com/promis_misc/SR_and_MA_2015-07-30.pdf ). This document, containing all the necessary sections for the systematic review/meta-analysis plan, must first be submitted by e-mail to ajo@elsevier.com . The Journal will notify the potential authors if it approves the planned meta-analysis. If it is approved, the Journal will issue a formal invitation for submission through the online submission system. The systematic review/meta-analysis planning document must then be completed and submitted along with the manuscript itself. The sole exception to this preapproval process is if the Cochrane Collaboration preapproved the protocol for the systematic review/meta-analysis.