Scientific Bias Arising from Sampling, Selective Recruitment, and Attrition: The Case for Improved Reporting Free

Almost a half a century ago, Campbell and Stanley proposed the following description of external validity, “External validity asks the question of generalizability: to what populations, settings, treatment variables, and measurement variables can this effect be generalized?” (1). This definition has been widely accepted, and careful consideration of sampling technique is considered one of the benchmarks of solid research design. Various types of sampling and the pros and cons of each are featured in Table 1. As we conceive of our research protocols, the methods by which we recruit our sample can comprise one or more of these strategies and are based largely on the research question at hand. However, recruitment of human subjects into research studies is complicated and often more complex than initially conceived. In addition, the Health Insurance Portability and Accounting Act (HIPAA) regulations and the downstream effects on Institution Review Boards have made it increasingly more difficult, time-consuming, and costly to recruit research participants and representative samples into observational studies and clinical trials (2).

Poor study accrual is one of the most common causes of failure for clinical trials, and even for the trials that are completed, subject accrual is often slower than anticipated and the number of subjects who refuse study participation is substantial (3, 4). Selective accrual has the potential of introducing bias into the study results, and can influence findings of not only behavioral and clinical research, but also epidemiologic investigations (5, 6). Likewise, for longitudinal studies (intervention or observation) attrition also can jeopardize external validity, as well as threaten other scientific constructs, such as statistical power (7).

During a session on recruitment and retention convened at the 2009 Annual American Association of Cancer Research Frontiers in Prevention Meeting, it was noted that while careful attention to these topics was usually apparent during the time of study conception, at the time of publication, such information often is lacking. To confirm this observation, an inventory of all research articles and short communications published in Cancer Epidemiology, Biomarkers and Prevention (CEBP) from September through December, 2009 (volume 18, numbers 9–12) were reviewed. We reviewed each of the 116 articles for author's inclusion of the following 6 criteria: (i) overall response rate; (ii) number of individuals excluded and explanation for individual exclusion; (iii) analysis on enrollee and nonenrollee demographic (or other) differences; (iv) report of participant dropout or attrition; (v) explanation for participant dropout; and (vi) analysis on differences between dropouts and completers on demographic characteristics or other parameters. Results of this inventory are provided in Table 2. Although some of these studies were secondary analyses and therefore were excluded or not considered applicable, of those that were deemed evaluable, 44% reported response rates, 53% listed the proportion excluded and reasons for exclusion, and only 10% reported analyses on enrollee versus nonenrollee differences; for the 17 longitudinal or intervention studies evaluated, only 18% reported dropout rates (3 of 17), and of those 3, only 2 reported reasons for dropout or an analysis that compared the characteristics of dropouts with those of completers. Thus, our perceptions were confirmed; indeed, the reports appearing within our flagship journal often lack the information needed to generalize findings and to potentially identify sources of bias. This is unfortunate, because this information is crucial for appropriate interpretation, and to weigh the contribution of the current findings against the backdrop of previous research to identify gaps that warrant further investigation.

Roughly 2 decades ago, similar concerns were expressed specifically as they pertained to published reports on clinical trials. (8–11). These discussions gave rise to the Consolidated Standards of Reporting Trials (CONSORT) which now provides guidelines, as well as a 22-item checklist and a flow diagram for use in reporting the results of clinical trials in a systematic fashion (12). While the CONSORT guidelines focus exclusively on clinical trials and include criteria that are far-reaching, (e.g., criteria for randomization, blinding, study conduct, power calculations, etc.) the framework they provide in areas of sampling and attrition is helpful [e.g., the study flow diagram, and criteria items 13 (participant selection and retention) and 21 (generalizability); ref. 12–14].

A systematic review by Plint and colleagues (2006) explored whether the CONSORT guidelines were associated with improvements in the quality of reports of randomized controlled trials (RCT) and compared report quality in journals before and after the publication of CONSORT guidelines, as well as report quality in journals that adopted CONSORT guidelines versus those that were nonadopters (15). Overall, these guidelines have been associated with an improvement in report quality, though the comparison of CONSORT adopting versus nonadopting journals show greater improvements in criteria in areas such as allocation concealment, than in areas associated with study flow, i.e., sampling, eligibility screening, and retention. That being said, among the subset of journals that have adopted CONSORT guidelines, there is a significantly higher relative risk = 8.06 (95% CI, 4.10–15.83) of attaining quality reporting on study flow–related components when comparing post- to preadoption articles. The authors concluded that indeed improvements have been realized with the implementation of the CONSORT checklist, and speculate that further improvements are likely as more journals adopt them and editorial efforts are brought to bear.

A review by Mills and colleagues (16) of reports published from 2002 to 2003 in 5 leading medical journals suggests that adherence to study flow criteria is 86% (95% CI, 82–90%) in the post-CONSORT era. Systematic reviews of current scientific reports in the various focus areas, such as urologic surgery, however, still show that relatively few reports meet the criteria established for quality reporting (17). Interestingly, a recent study that assessed trial quality from main outcomes papers, found that commercially-sponsored RCTs appeared to be of superior quality as compared to trials which were sponsored by the government (18). The reporting of attrition, or the lack thereof, was one of the major criticisms of the reports emanating from government-sponsored research. These results provide sad commentary for the bulk of studies which are likely reported in CEBP, that is, behavioral and chemopreventive RCTs that are more likely to be supported by federal funding than by commercial entities.

Although the CONSORT guidelines were the first to address the quality of research reporting and were designed specifically for clinical trials, the charge to improve reporting has been taken up by other research communities, as well. For example, in 2004, a checklist of items was drafted to improve the reporting of observational studies, i.e., Strengthening the Reporting of Observational Studies in Epidemiology (STROBE; ref. 19) and a few years later, guidelines were issued to address the reporting of genetic association studies, i.e., Strengthening the Reporting of Genetic Association studies (STREGA; ref. 20). Like CONSORT, the checklists of STROBE and STREGA address issues beyond recruitment and retention, and generalizability; however, these are common elements across all three. Thus, for the purposes of CEBP, which attracts a diverse readership of epidemiologists and behavioral scientists, the adoption of transparent guidelines to describe sampling and retention may represent “common ground” and a good place to start, especially because the means to evaluate adherence is far more objective than determining potential measurement error, adequate control for confounding, the appropriateness of design, overinterpretation of the data, etc. Although some have criticized the imposition of guidelines as a measure which will stifle creativity and promote “cookbook science,” we view it strictly as a means of reporting which will improve transparency and ultimately lead to superior research (21).

Given CEBP's mission to enhance the dissemination of unbiased scientific findings–a commitment which has spurred the creation of novel platforms, such as “Null Results in Brief” (22, 23), we propose that the guidelines espoused for CONSORT, STROBE, or STREGA (depending on the nature of the study) be adopted by our journal, to transparently convey not only a description of the population from which the sample is drawn, but chronicle numbers, as well as basic characteristics (race, age, cancer type, and stage, etc.) from the point of study contact. Although HIPAA is an acknowledged and formidable barrier to human research, and could be conceived as a barrier to such collection (2), the maintenance of databases with deidentified data on basic demographic and clinical variables should still be possible. Such information could be excerpted from tumor registries or during the point of initial contact if purposive or convenience sampling is performed and stored apart from any identifying information. Such data would help in determining whether the sample that is eventually screened and recruited differs from the pool of origin. Likewise, careful and accurate monitoring of dropout and basic analyses of differences between dropouts and completers also is important for longitudinal investigations. Such action could indeed improve the quality of reporting, allow for better transparency in assessing sources of bias, encourage the publication of studies with more representative samples, and ultimately improve our understanding of the generalizability of findings. The adoption of more rigorous criteria for publication will ultimately allow us to move science forward faster and more clearly.

No potential conflicts of interest were disclosed.