When Is Enough, Enough? When Are More Observational Epidemiologic Studies Needed to Resolve a Research Question: Illustrations Using Biomarker–Cancer Associations Free

Translation of cancer etiology, risk, prognosis, and prediction biomarkers into prevention and control strategies relies, in part, on the ability to reproduce associations. However, repetitive investigations of established biomarker–cancer associations that do not contribute meaningful additional information to the existing evidence base, for example, fill remaining knowledge gaps, provide substantial clinical or public health support for an association, or have the potential to improve biological understanding, may be inefficient and a waste of resources (1–3).

To address these concerns, we adapted an application of existing clinical trial and research synthesis metrics, the fail-safe number (FSN; ref. 4) and conditional power (5), to determine whether or not further investigation of cancer relevant biomarkers may provide meaningful contribution to the existing evidence. In its original application, Rosenthal (6) introduced the FSN to quantify the impact of selectively unpublished research on the existing meta-analysis. The FSN indicates the number of unpublished studies with an average null effect (e.g., P ≥ 0.05) needed to be included in an updated meta-analysis to drive a statistically significant summary estimate in the existing meta-analysis (e.g., P < 0.05) to a statistically nonsignificant summary estimate (e.g., to P ≥ 0.05) in the updated meta-analysis. We adapted the FSN for observational epidemiology studies to determine whether the inference from an existing meta-analysis for a statistically significant exposure–outcome association, will likely change to a null association with the addition of further research to update the meta-analysis. In its original application, conditional power was used to guide the design of clinical trials based on effect size and sample size of an existing trial or meta-analysis. In the context of observational epidemiology and assuming a statistically nonsignificant existing meta-analysis, we adapted conditional power calculation to determine the feasibility of conducting the necessary number of future studies with sufficient power to detect a significant association of a certain size in the updated meta-analysis (5).

We applied FSN and conditional power to a collection of 98 existing meta-analyses (7) of associations between nongenomic cancer biomarkers and multiple types of cancer. More detailed illustration of their use is provided using data on a well-established biomarker–cancer relationship [i.e., Helicobacter pylori (H. pylori) and gastric cancer] and an uncertain biomarker–cancer association (i.e., androgens and prostate cancer).

FSN and conditional power were applied to findings from 98 biomarker–cancer meta-analyses (refs. 8–44; Table 1) published in 37 reports that were curated by Tsilidis and colleagues after a comprehensive PubMed search of meta-analyses of epidemiologic studies on biomarkers and cancer risk published between 1966 and 2010 (7). The purpose of that study was to evaluate whether evidence of excess statistical significance could be detected in such studies that would be indicative of publication bias.

The 98 meta-analyses included a median of seven studies (range 2–42) and described associations between a diverse range of nongenomic biomarkers and cancer risk including: insulin-like growth factor/insulin markers (21 meta-analyses); sex hormones (13 meta-analyses); dietary markers (31 meta-analyses); inflammatory markers (3 meta-analyses); infectious agents (22 meta-analyses); and environmental markers (8 meta-analyses). The most common cancer sites include breast (28 meta-analyses); prostate (24 meta-analyses); lung (10 meta-analyses); and colorectal (8 meta-analyses). Previously, using the primary study data from the studies included in each of the 98 meta-analyses, Tsilidis and colleagues (7) calculated summary estimates using fixed-effect and random-effects models and corresponding 95% confidence intervals, and I². On the basis of random-effects models, 44 (45%) of the meta-analyses reported statistically significant summary ORs, whereas on the basis of fixed-effect models 54 (55%) of the meta-analyses reported statistically significant summary ORs.

For the statistically significant meta-analyses, we used Rosenberg's version of the FSN (ref. 4; a refinement of Rosenthal's FSN, ref. 6) to quantify the number of future studies with an average null effect and average weight (i.e., inverse variance), needed to drive the existing meta-analysis summary estimate to null in the updated meta-analysis (for this work, P ≥ 0.05). To overcome the restriction of statistical significance, we used Orwin's FSN (45) to calculate the number of future studies with an average null effect (OR = 1.00) needed to reduce the updated summary effect to a range of estimates (OR = 1.05; 1.10; 1.25; 1.50; and 2.00) for the updated meta-analysis. Additional details of FSN calculation are presented in Supplementary Methods. FSN is not applicable to nonstatistically significant summary estimates.

For the nonstatistically significant meta-analyses, we calculated conditional power to determine the number of future studies needed to achieve sufficient power to detect a statistically significant summary estimate when added to the observed nonstatistically significant meta-analysis (P ≥ 0.05). We set the minimum power to 0.8 and took a pragmatic approach declaring an alternative hypothesis for the updated meta-analysis equivalent to the observed summary OR, and assumed the future studies were of average weight as those included in the observed meta-analysis. Our conditional power analyses were based on two approaches described by Roloff and colleagues (5). We implemented the first approach in the nonstatistically significant fixed-effect meta-analyses, where we assumed that no heterogeneity is present between the studies included in the existing meta-analysis (I² = 0%) and that the future studies will not introduce heterogeneity. In approach 2, focusing on the nonstatistically significant random-effects meta-analyses, we fixed the between-study heterogeneity in the future studies to be equivalent to the heterogeneity in the existing meta-analysis. Additional details of conditional power calculation are presented in Supplementary Methods

From the list of 98 meta-analyses, we selected two exemplar scenarios: (i) a well-established causal biomarker–cancer relationship supported by evidence-based classification as a group 1 carcinogen (i.e., H. pylori and gastric cancer risk; ref. 46) and (ii) a biomarker–cancer association with strong biological rationale, but several methodologic concerns leading to an uncertain biomarker–cancer association (i.e., androgens and prostate cancer). We provide these two examples both to describe the application of these adapted methods and how their use can be used in practice to inform the need for future research to be able to fill knowledge gaps and improve biological understanding. For both scenarios, we interpret the number of future studies needed determined by FSN for H. pylori and gastric cancer or by conditional power for androgens and prostate cancer within the context of the existing evidence (e.g., the number, sample size, and heterogeneity of the findings).

We calculated Rosenberg's and Orwin's FSNs and the two conditional power approaches in STATA version 13 (STATA Corp).

Among the 54 statistically significant fixed-effect [median number of studies 9 (range 2–42); median I² = 42%] and 44 statistically significant random-effects [median number of studies 9 (range 2–42); median I² = 36%] meta-analyses, median FSN (Rosenberg) was 31.5 studies (range 3.2–24,939) for the fixed-effect meta-analyses, and 31.1 studies (range 3.2–3,464) for the random-effects meta-analyses.

The influence of between-study heterogeneity on Rosenberg's FSN is illustrated by comparing the FSN between the fixed-effect and random-effects summary estimates from the same meta-analysis (Supplementary Fig. S1). The median FSN was larger for meta-analyses with extreme heterogeneity (I² > 80%; ref. 47); 1,497 and 148 for fixed-effect and random-effects meta-analyses, respectively, compared with 53 and 45 for fixed-effect and random-effects meta-analyses with low heterogeneity (I²: 1%–29%; ref. 47). The FSN was larger for the fixed-effect than for the random-effects meta-analyses, which is consistent with the assumption of no between-study heterogeneity in fixed-effect meta-analyses that results in more precise summary estimates (ref. 48; Supplementary Fig. S1). Among meta-analyses with similar between-study heterogeneity (0%, 1%–29%, 30%–59%, 60%–80%, and >80%), meta-analyses that included more studies tended to have a higher FSN (Supplementary Fig. S2) as a result of more precise summary estimates.

Rosenberg's FSN was larger when the summary estimates observed in the existing meta-analyses were higher (Supplementary Fig. S3). The influence of summary estimate size in the existing meta-analysis and in the future studies is further illustrated with Orwin's FSN, which does not take into account within- or between-study heterogeneity. Therefore, we only considered the values of Orwin's FSN for fixed-effect meta-analyses. Orwin's FSN was larger for smaller updated summary estimates (Supplementary Fig. S4). To reduce the updated summary OR to 1.05 among 38 meta-analyses with an existing summary OR > 1.05, the median of Orwin's FSN was 271 studies, whereas to reduce the updated summary OR to 2.00 among meta-analyses with an existing summary OR > 2.00 the median FSN was 33 studies. As for Rosenberg's FSN, which is based on statistical significance, Orwin's FSN, which is based on effect size, also indicates that a larger number of future studies is required for existing meta-analyses with larger as opposed to smaller summary ORs.

We used two approaches under a variety of assumptions to conduct conditional power analysis. In the first approach, we assumed no between-study heterogeneity in the existing and updated meta-analyses, and accordingly, used only the 18 fixed-effect meta-analyses with a statistically nonsignificant summary OR > 1.01. With a median power of 15% (range 0.5%–50%) for the existing meta-analyses, a median of 78 studies (range 4–994) of average weight with no between-study heterogeneity would need to be included in the updated meta-analysis to achieve 80% power to detect the summary OR as statistically significant.

In the second approach, we assumed equivalent between-study heterogeneity in the future studies as in the existing meta-analysis, and accordingly used the 21 random-effects meta-analyses with a statistically nonsignificant summary OR > 1.01. With a median power of 21% (range 6%–47%) for the existing meta-analyses, a median of 103 studies (range 5–6,656) of average weight and equivalent between-study heterogeneity as in the existing meta-analysis would need to be included in the updated meta-analysis to achieve 80% power to detect the summary OR as statistically significant.

The greater number of future studies required to achieve 80% for the random-effects compared with fixed-effect meta-analysis is consistent with their differing assumptions about between-study heterogeneity incorporated into the two approaches (Supplementary Fig. S5). By taking into account the between-study heterogeneity, our second approach incorporated additional uncertainty into the summary estimates, thereby increasing the number of future studies needed. In the both fixed-effect and random-effects meta-analyses, the number of future studies needed was smaller for larger than for smaller summary estimates (Supplementary Fig. S5).

In 1994, the International Agency for Research on Cancer (IARC) classified H. pylori as a Group 1 carcinogen (46). At the time, the evidence supporting IARC's classification included four cohort studies and nine case-control studies of H. pylori infection and gastric cancer risk. Since the initial classification, the accumulation of evidence is sufficient that the relationship is now considered well established. This is reflected in the greater than 2-fold increase in risk of gastric cancer described in the meta-analysis of 15 studies with more than 5,000 cases and controls reported by Huang and colleagues (33) Rosenberg's FSN indicates 805 future studies would be required to reduce the reported fixed-effect summary OR of 2.05 (95% CI, 1.79–2.35; I² = 76%) to null (P ≥ 0.05) and 224 future studies based on the random-effects meta-analysis (summary OR = 2.29; 95% CI, 1.71–3.05; I² = 76%). On the basis of Orwin's FSN, a total of 615 future studies averaging null effect (OR = 1.00) would be required to drive the observed fixed-effect summary OR of 2.05 to an essentially null OR of 1.05. The implementation of each FSN to the example of H. pylori and gastric cancer illustrates the futility of further investigation of the association between H. pylori and gastric cancer, while the large between-study heterogeneity (I² = 76%) suggests the need for further subgroup analysis to determine sources of heterogeneity (e.g., method of detection of H. pylori infection, adjustment for confounding, or geographic/ethnic differences in strength of the association). To this end, the geographic and ethnic differences in the distribution of gastric cancer led to further investigations that revealed a stronger association between H. pylori infection and gastric cancer in studies conducted in populations with diets high in salt-preserved foods, suggesting dietary salt may modify the pathogenic effect of H. pylori infection on gastric cancer (49, 50). The role of a high-salt diet as a potential modifier of the effect of H. pylori is supported by additional laboratory research that identified cagA gene expression in H. pylori, a marker of higher risk of gastric cancer, is upregulated by dietary salt intake (51). These findings further illustrate the importance of examining subgroups or different populations once the main effect of the etiologic cancer biomarker has been established, especially in the context of extreme heterogeneity which can help identify high-risk populations and can provide additional understanding of the underlying biology of the biomarker cancer association (e.g., effect modification).

In 1993, the Prostate Cancer Prevention Trial was launched to test the hypothesis that finasteride, a drug that blocks the conversion of testosterone into DHT, can prevent prostate cancer (52). The trial was stopped early in 2003 when an interim analysis found a 25% reduction in the period prevalence of prostate cancer in the treatment group receiving finasteride (53). This finding provided additional evidence supporting the underlying hypothesis that DHT is an etiologic factor in prostate cancer. However, several methodologic challenges encountered in population-based epidemiologic investigations including adequacy of measuring circulating hormones, difficulty integrating multiple components of the androgen pathway, difficulty in incorporating clinical and population health important outcomes, and detection bias (e.g., differential opportunity to be screened with PSA by exposure; and differential detection of prostate cancer in PSA-based prostate cancer screening due to the association between androgens and PSA concentration), have contributed to the inconsistent reports on the associations between circulating androgens and prostate cancer incidence (54). Using study-specific estimates for components in the androgen pathway and prostate cancer from a pooled analysis of harmonized primary data, (43) Tsilidis and colleagues (7) calculated fixed-effect and random-effects summary estimates (Table 2). For the six components of the androgen pathway that were not statistically significant in fixed-effect meta-analyses (with I² = 0% and a median number of studies of 8.5), conditional power indicated that 18 to 1,173 future studies of average weight as those included in the existing meta-analysis would be required to achieve 80% power to detect the summary OR in the updated meta-analysis (Table 2). For these comparisons, the large number of future studies needed to achieve sufficient power, more than twice as many studies as included in the existing meta-analyses, of the same average weight, totaling tens of thousands of cases and controls among the future studies (Table 2), may not be within reach of existing resources, and points to a situation where further research should be aimed at overcoming the methodologic challenges mentioned above (54) to fill important evidence gaps with respect to androgens and prostate cancer.

In the case of the random-effects meta-analysis with seven included studies evaluating the association between dehydroepiandrosterone sulfate (DHEAS) and prostate cancer (summary OR = 1.29; 95% CI, 0.99–1.68; I² = 17%), the five future studies required to achieve 80% power to detect the observed summary OR may be within reach of existing resources, and points to a scenario where additional research could provide a meaningful contribution to the existing meta-analysis. However, we caution against the inappropriate interpretation of applying conditional power to the example of DHEAS and prostate cancer incidence. Our approach assumed that the number of future studies are of the average weight of those already included in the existing meta-analysis and that they will not introduce additional between-study heterogeneity into the updated meta-analysis. However, this assumption may not be realistic; with respect to molecular epidemiologic investigations, measurement error in the index biomarker assay may introduce between-study heterogeneity. Furthermore, relying on the number of needed studies does not guarantee that a future study will be informative. Whether to conduct future studies on DHEAS and prostate cancer must also take into consideration the composition of the existing evidence base (e.g., existing study population characteristics and prostate cancer case mix) and failure to consider the methodologic issues previously cited as factors leading to inconsistent associations could also lead to uninformative research.

We adapted two established metrics, the FSN (4) and conditional power (5), to quantify the impact of future investigations on the inferences drawn in existing meta-analyses. Both metrics provide a heuristic approach to inform whether continued investigation is warranted versus sufficient evidence is available to establish or refute an exposure–outcome association. Our motivation to adapt the application of these metrics is to be able to quantify the impact of further investigation of the same association as the primary research question. However, the application of these metrics should not be interpreted as stopping research all together, but rather, to focus future research to address current evidence gaps and improve biologic understanding of the biomarker–cancer association by evaluating new or improved methods to measure the biomarker or using other markers correlated and more specific to the studied biomarker, evaluating clinically meaningful outcomes, and reducing heterogeneity and imprecision in the observed associations by investigating the biomarker–cancer relationship in important subpopulations. When further research does not add information to the existing literature, unnecessary and wasteful research may be undertaken (55). We envision the application of these metrics along with traditional assessments of study quality (e.g., STROBE, ref. 56; PRISMA, ref. 57) causal criteria (58), and remaining knowledge gaps (e.g., subgroup associations) by stakeholders engaged in translational epidemiologic research including principal investigators, funding agencies, grant reviewers, journal editors, and peer-reviewers to make more informed decisions about the need for additional research. While our application of FSN and conditional power focused on observational studies of etiologic biomarkers and cancer risk, these methods are equally applicable to other epidemiologic study designs including randomized trials as well as nonbiomarker exposures and other important outcomes such as mortality, and prognosis.

FSN can be calculated using several common meta-analysis software packages and calculation of conditional power is straightforward (See Supplementary Methods) but requires a number of assumptions (e.g., heterogeneity, effect size, and study weights) that influence how the corresponding metrics are interpreted, thus informing the impact of future research. We applied these metrics to 98 meta-analyses of observational epidemiologic studies evaluating the associations between nongenomic biomarkers and cancer risk to demonstrate the ability of these metrics to identify situations where future research may or may not provide a meaningful contribution to an updated meta-analysis. When adapting the application of these metrics, the patterns of the output of the FSN and conditional power analysis are consistent with the underlying computation of each metric. For example, FSN appears to increase with decreasing heterogeneity, increasing number of included studies, and increasing magnitude of summary estimates. For conditional power, the number of additional studies appears to decrease with increasing magnitude of summary estimates.

To our knowledge, no method has been introduced to directly quantify the expected impact of further observational epidemiologic research on the current evidence base. While our motivation was to explore whether the FSN and conditional power could be used to quantify the impact of future research, additional work is needed to incorporate these metrics into a formal framework for deciding whether additional epidemiologic studies addressing the same question are needed. Such a framework might include cut-off points or ranges for defining whether the number of future studies needed is too large to make additional work worthwhile. We do not envision that the framework would rely on cut-off points alone: considerations that could be incorporated into the framework beyond a cut-off point might include feasibility and cost, as well as implications for policy, and clinical and public health recommendations. Such a framework could encompass aspects of the value of information approach to deciding cost-effectiveness, which has been described for improving research prioritization and reducing waste (59).

We recognize that application of these adapted methods to existing meta-analyses is not the only strategy to minimize the problem of repetitive research. Facilitating and encouraging the publication of null results that can be included in meta-analyses such that the null results are interpreted alongside the relevant evidence is a direct way investigators and stakeholders can minimize the production of redundant uninformative research (60). An alternative approach is a coordinated effort among individual investigators to determine which exposures require additional investigations, to share and pool their data and biospecimens, to standardize an exposure's measurement, and harmonize the outcome and covariate data, all while ensuring optimal study design and minimizing selection and information bias. Using this approach, research on particular exposures is prioritized through consensus, exposure–outcome associations can be investigated in subpopulations of the pooled studies, and power is maximized. This practice-based approach has been used over the past 15 years by large consortia, including the NCI Cohort Consortium (>50 cohorts with 7 million participants; https://epi.grants.cancer.gov/Consortia/cohort.html#overview) and the Early Detection Research Network (https://edrn.nci.nih.gov) both supported by the NCI, and the Endogenous Hormones, Nutritional Biomarkers and Prostate Cancer Collaborative Group (35 studies with biomarker data on 23,000 men with prostate cancer and 35,000 controls; https://www.ceu.ox.ac.uk/research/endogenous-hormones-nutritional-biomarkers-and-prostate-cancer). We view the approach that we describe herein as complementary to the practice-based approach.

In summary, we show how FSN and conditional power can be adapted to quantify the impact of future investigations of a specified exposure and outcome on the current evidence base summarized in the corresponding meta-analysis. To illustrate the utility of these approaches, we applied them to meta-analyses of biomarkers and cancer risk. The systematic application of these metrics by researchers, funding agencies, and grant reviewers when considering future research, journal editors, and peer-reviewers when considering the novelty and impact of submitted manuscripts, could lead to more judicious use of resources and acceleration along the translational continuum from discovery to population-health impact.

No potential conflicts of interest were disclosed.

The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the NIH.

Conception and design: M.T. Marrone, K.K. Tsilidis, T.R. Rebbeck, E.A. Platz

Development of methodology: M.T. Marrone, S. Ehrhardt, T.R. Rebbeck

Acquisition of data (provided animals, acquired and managed patients, provided facilities, etc.): M.T. Marrone, K.K. Tsilidis, C.E. Joshu

Analysis and interpretation of data (e.g., statistical analysis, biostatistics, computational analysis): M.T. Marrone, K.K. Tsilidis, S. Ehrhardt, C.E. Joshu, T.R. Rebbeck, E.A. Platz

Writing, review, and/or revision of the manuscript: M.T. Marrone, K.K. Tsilidis, S. Ehrhardt, C.E. Joshu, T.R. Rebbeck, T.A. Sellers, E.A. Platz

Study supervision: E.A. Platz

We appreciate Dr. Muin Khoury's helpful comments during the conduct of this work. M.T. Marrone was supported by NCI grant T32 93140 (Platz). E.A. Platz was supported by NCI Cancer Center Support Grant P30 CA006973 (Nelson). C.E. Joshu was supported by the Prostate Cancer Foundation.

The costs of publication of this article were defrayed in part by the payment of page charges. This article must therefore be hereby marked advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.