Opportunities and Challenges for the Use of Large-Scale Surveys in Public Health Research: A Comparison of the Assessment of Cancer Screening Behaviors

The National Cancer Institute (NCI) supports large-scale surveys that collect health behavior data from populations residing in distinct geographic regions and at varying time intervals. Data from these surveys are used to produce national- and state-level estimates of behaviors, helping public health researchers address issues related to behavioral patterns, trends, and geographic variation. Federal public health agencies may use items from these surveys to monitor progress toward reaching health objectives, determine the effectiveness of interventions, and identify disparities and other health indicators that suggest the need for focused attention. These surveys are often used by a subset of researchers who are familiar with them, yet the broader research community might benefit from increased awareness of the data and improved understanding of their content, methodology, and utility. Until now, no comprehensive resource existed to describe and facilitate use of these data. Thus, we at the NCI undertook a project to catalogue the characteristics, content, and strengths of these surveys as they relate to cancer screening, with the goal of maximizing their use.

We focus on cancer screening because it represents a well-defined set of health behaviors for which clinical guidelines exist. Population-based breast, cervical, and colorectal cancer screening of average-risk individuals is critical because it leads to reductions in morbidity and mortality (1–3). It is vital to know whether the U.S. population is receiving appropriate screening, and population-based surveys with self-reported information are the only mechanism available to obtain national- and state-level estimates of screening utilization. Scientific agreement exists about a limited number of key cancer screening behaviors that need to be measured. Furthermore, past research has led to an understanding of correlates, such as individual sociodemographic characteristics, personal attitudes, beliefs, and healthcare access or system factors, which are commonly associated with cancer screening behaviors (4–9), and are typically incorporated into these surveys. Together, these features make cancer screening a good example for exploring the opportunities and challenges of using surveillance systems in public health research.

Cancer screening data from large-scale surveys can be used to measure test use (including underuse and overuse), assess screening patterns and trends, explore geographic variation, identify populations at the highest risk, examine disparities in screening uptake, and discern which factors may impede or facilitate screening so as to establish priorities for intervention research and practice (10, 11). Especially in an era of fiscal retrenchment, these publicly available surveys offer researchers a range of possible uses: preliminary data, alternatives to primary data collection, and comparison samples for data collected from other studies. Survey data can also be pooled across years (12) or combined to provide more precise estimates (13) for innovative analyses and knowledge synthesis. There are, however, several aspects of the surveillance of cancer screening behaviors that may impede the use of these data by researchers. One challenge is that although multiple surveillance systems collect data on key cancer screening behaviors, the specific ways in which these behaviors are measured may vary slightly from survey to survey. Moreover, surveys typically differ in the breadth, depth, and correlates of cancer screening behaviors measured. Surveys are designed to achieve specific yet varying goals, thus they use different sampling approaches and modes of administration, and they differ in their degree of geographic granularity. Although elucidating such differences can be challenging, this variability ultimately results in unique strengths for each survey that researchers can capitalize upon to address a specific research question.

This article is designed to help researchers decide where to look, in terms of cancer screening surveys, to address different kinds of questions. Our goal is not to offer an evaluation of which survey is best in terms of content or methodology but rather to present strengths of the various surveys and the differences among them to guide researchers toward the survey that is most appropriate for their specific research questions.

To this end, we examined 4 leading, federally funded, publicly available surveys that collect data on both: (i) cancer screening constructs, defined as concepts reflecting important aspects of cancer screening that can be assessed with multiple measures (e.g., screening test use, beliefs about screening, or provider recommendations about screening) and (ii) a wide range of correlates of cancer screening, defined as variables reflecting general characteristics that past research has shown are frequently associated with cancer screening constructs (e.g., sociodemographics such as age, personal attitudes such as perceived risk of developing cancer, and healthcare access and system factors such as insurance coverage; refs. 4–9). We selected 2 surveys that provide national-level data—the National Health Interview Survey (NHIS; www.cdc.gov/nchs/nhis.htm) and the Health Information National Trends Survey (HINTS; hints.cancer.gov). We also selected 2 surveys that provide state- and county-level data widely used outside of their local areas—the Behavioral Risk Factors Surveillance System (BRFSS; www.cdc.gov/brfss) and the California Health Interview Survey (CHIS; www.chis.ucla.edu).

Our analysis focused on each survey's characteristics and content in 2005—the only year within the past decade when NHIS, HINTS, BRFSS, and CHIS concurrently collected cancer screening data. We were thus able to make direct comparisons across the different surveys at a single point in time and to examine ways in which data from the surveys have been used in the published literature as examples of their utility. We documented survey content about those cancer screening behaviors for which guidelines existed from organizations such as the U.S. Preventive Services Task Force (USPSTF; refs. 14–17) and American Cancer Society (ACS; ref. 1); USPSTF recommendations are used in preventive services coverage determinations under the Patient Protection and Affordable Care Act (18), and ACS guidelines are a widely used resource among primary care providers in the U.S. Thus, we focused on behaviors recommended at the time of data collection: breast (clinical breast examinations; mammography), cervical (Pap testing), colorectal (home and office fecal occult blood testing or FOBT; endoscopy including colonoscopy and sigmoidoscopy), and prostate (prostate-specific antigen or PSA testing; digital rectal examinations) cancer screening. To promote the use of these data resources by the research community, our analyses answered 5 primary questions related to the surveys' characteristics, content, and use, described below. We will answer each of these questions in turn, based on our analyses of the surveys and related literature, and then provide conclusions and recommendations for public health researchers' further use of these rich data resources.

To document the characteristics of each survey, we created a detailed data abstraction tool that was guided by a fixed set of indicators (e.g., sampling strategy, pretesting, and administration; extent of geocoding; data access procedures and costs) adapted from those of the National Collaborative on Childhood Obesity Research (NCCOR) Catalogue of Surveillance Systems (19). Members of the study team (J.G. Hamilton, N. Breen, C.N. Klabunde, R.P. Moser, S.C. Kobrin) used each survey's online documentation to complete the data abstraction tool (findings presented in Table 1).

Begun in 1957, the purpose of NHIS is to collect cross-sectional data on the health status, behaviors, conditions, and the use of health services in the U.S population. NHIS is conducted annually by the National Center for Health Statistics (NCHS) using face-to-face interviews of civilian, noninstitutionalized individuals living in households or group quarters in all 50 states and Washington, DC. (NHIS also collects data on children. Children are not discussed here because they are not included in population-level screening guidelines for the examined cancers.) Data for most of the cancer screening constructs are collected through periodic Cancer Control Supplements to the NHIS (most recently administered in 2000, 2005, and 2010); however, some basic measures of cancer screening were included in additional years (e.g., 2003, 2008, 2011, 2012, 2013, 2014). Survey content for NHIS is evaluated through cognitive testing and through pilot and field testing (dependent on sponsor resources). Public-use NHIS datasets and sample weights (necessary for computing national-level estimates) are available free of charge from the NCHS website (20). Restricted-use data (e.g., geocoded data) can be accessed with permission through the NCHS Research Data Center.

The purpose of HINTS is to collect cross-sectional data regarding how people access, trust, and use health and cancer information, how they use information technology to manage health and health information, and the degree to which they engage in health behaviors, particularly those relevant to cancer prevention and control. HINTS is conducted periodically (i.e., 2003, 2005, 2007/2008, 2011/2012, 2012/2013, 2013) and samples from the population of U.S. civilian, noninstitutionalized adults (ages 18 years and older). HINTS has been administered by telephone as well as through the mail. Consistent with the survey's cancer focus, cancer screening constructs have appeared in all iterations of HINTS. Survey content is evaluated and refined through cognitive testing and extensive pilot testing. HINTS datasets and sample weighting information for both the final sample weight and the replicate weights (poststratified on gender, race/ethnicity, age, and education) are publicly available free of charge from the HINTS website (21); users must consent to a “terms of use agreement.” Some geocoding variables are available; researchers must submit proposals to obtain these restricted-use data.

Begun in 1984, the purpose of BRFSS is to collect cross-sectional, state-specific data about health behaviors associated with premature morbidity and mortality from individuals in all 50 U.S. states, Washington, DC, American Samoa, Palau, Puerto Rico, the U.S. Virgin Islands, and Guam. BRFSS is conducted annually through telephone interviews with civilian, noninstitutionalized adults (ages 18 years and older). BRFSS consists of a core component of fixed questions that are asked by all states and territories, optional modules that individual states and territories can elect to include in their annual survey, and some state-added questions. Cancer screening constructs have been included in both core components and optional modules. In 2005, cancer screening–related content was included in optional modules that a subset of states and territories elected to administer. All core and optional survey content is evaluated through cognitive and field testing. Public-use BRFSS datasets and sample weighting information can be obtained free of charge from the BRFSS website (22). Some public-use geocoding variables exist, and the website's Selected Metropolitan/Micropolitan Area Risk Trends (SMART) data and documentation can be used to obtain information about metropolitan and micropolitan statistical areas with 500 or more respondents if states elect to pay for the required oversample.

The purpose of CHIS is to collect cross-sectional data regarding Californians' health status, health behaviors, insurance coverage, and access to and use of healthcare. CHIS began in 2001 and is released biannually. CHIS collects data about civilian, noninstitutionalized residents of California including one adult (ages 18 years and older), one adolescent (ages 12–17 years), and one child (ages 0–11 years) from each household sampled through telephone interviews. Cancer screening constructs have appeared in all iterations of CHIS. CHIS has historically adopted many items from NHIS; survey content is further evaluated and refined through additional cognitive and pilot testing. Although not used in 2005, behavioral coding (23) has also been periodically used to ensure the quality of CHIS survey content. After completing a one-time registration, researchers can access public-use CHIS datasets and sample weighting information free of charge from the survey website or use the online calculator AskCHIS to create their own tables using CHIS data (24). Proposals for analyses involving confidential and sensitive data (e.g., geographic identifiers) can be submitted to the CHIS Data Access Center; approvals must be obtained and fees may be charged to access such data.

One coder (J.G. Hamilton) reviewed the surveys' online documentation and 2005 questionnaires to document each item that assessed an aspect of cancer screening including the item wording, response options, and respondent eligibility. Next, 2 coders (J.G. Hamilton and S.C. Kobrin) reviewed the items and identified their discrete cancer screening constructs. Once a list of constructs and corresponding items was established, it was reviewed by the other study authors and refined until consensus was achieved. Table 2 shows the cancer screening constructs assessed by items within the 4 surveys (e.g., “Mam ever had” or was a mammogram ever had by the respondent) and indicates which survey(s) included an item to measure the construct, as well as whether the wording, response options, and universe of eligible respondents for the item were the same across the surveys. Exact item wordings, response options, and eligibility requirements are provided in Supplementary Table S1.

These surveys included constructs related to clinical breast examinations (3 separate constructs), mammography (12 constructs), Pap testing (14 constructs), home or office FOBT (12 constructs), endoscopy (8 constructs), colorectal cancer screening in general (5 constructs), PSA testing (9 constructs), and digital rectal examinations (2 constructs). NHIS assessed the largest number of constructs (47 constructs), followed by CHIS (28 constructs), HINTS (24 constructs), and BRFSS (14 constructs).

The surveys differed in the specific types of screening constructs measured. BRFSS primarily assessed basic behavioral surveillance constructs related to ever using a screening test and the timing of the most recent test. NHIS and CHIS assessed additional details including abnormal screening test results and follow-up, reasons for using or not using tests, and receipt of provider recommendations for tests. HINTS assessed screening attitudes and beliefs including guideline knowledge, test intentions, and perceptions of tests (e.g., fear, benefits). Only the basic behavioral surveillance constructs of ever using a screening test and the timing of the most recent test appeared on all 4 surveys. Ever use of mammography was the only construct with item wording and response options that were identical across surveys, although the universe of respondents differed across surveys. For timing of mammography, and ever use and timing of Pap testing, home FOBT, endoscopy, and PSA testing, item wording, response options, and universe of eligible respondents differed across the 4 surveys. In no instance was the same cancer screening construct measured with identical item wording, response options, and universe of respondents in 2 or more of the surveys.

Theoretical and empirical correlates of cancer screening constructs were identified through an examination of the 2005 NHIS, HINTS, BRFSS, and CHIS surveys' online documentation and 2005 questionnaires (see Table 3). (Although the correlates listed in Table 3 are specific to the 2005 iterations of NHIS, HINTS, BRFSS, and CHIS, many of these correlates are also found on other iterations of the surveys.) Correlates within the broad categories of sociodemographics, healthcare access and utilization, cancer history, health behaviors, and health status appeared on multiple surveys. Certain constructs within these categories, including education, income, interview language, race/ethnicity, insurance status, personal cancer history, nutrition, physical activity, tobacco use, body mass index, self-reported health, and emotional health, appeared on all 4 surveys. Other specific correlates in these categories such as sexual orientation, time in the United States, reason for lack of insurance coverage, family cancer history, health information seeking, and diagnosis of specific health conditions only appeared on 1 or 2 surveys. Similarly, correlates within the broad categories of physician communication, attitudes and beliefs, social integration and support, and objective disease risk calculators were only present on a few of the surveys.

In January 2014, we conducted for each survey a search of the Scopus and PubMed databases using the following broad search terms: [survey name] AND cancer screening AND 2005. Eligibility was limited to articles published in English since January 1, 2005. Publication lists available from the HINTS (25) and CHIS (26) websites were also reviewed. Each identified article was evaluated to determine whether it fit our objectives (e.g., used 2005 survey data relevant to a cancer site of interest). This search yielded a total of 147 publications; of these, 78 publications met the eligibility criteria, with 43 reporting on NHIS (7, 27–68), 11 on HINTS (69–79), 7 on BRFSS (80–86), and 17 on CHIS (87–103). One coder (J.G. Hamilton) reviewed these publications to develop a broad categorization scheme for the types of research questions that had been addressed with the surveys' data; publications were categorized as examining correlates of cancer screening behaviors, examining cancer screening trends over time, or enhancing understanding of cancer screening by linking different data sources (note that publications could, and often did, address more than one type of research question). Relevant information about each publication was abstracted (e.g., types of correlates examined; see Supplementary Table S2).

Across surveys, data were used most frequently to examine correlates of cancer screening behaviors (63% of identified research questions). Details about the different types of correlates examined are depicted in Fig. 1. As shown, sociodemographic correlates and healthcare access and utilization correlates were frequently used in publications arising from each survey. Publications using HINTS data reported on the greatest variety of correlates; for example, Nelson and colleagues (73) examined the extent to which sociodemographics (age, race/ethnicity, education, marital status, English fluency), healthcare access and utilization (insurance status, frequency of provider utilization), health status (body mass index, self-reported health, emotional health), health behaviors (tobacco use, information seeking), cancer history (self and family history), and attitudes and beliefs (HPV knowledge) were associated with adherence to Pap testing recommendations. Conversely, publications using BRFSS data reported on fewer correlates, primarily involving sociodemographics and healthcare access and utilization. For example, BRFSS data were used to evaluate the association between insurance coverage and colorectal cancer screening in Virginia while accounting for factors such as gender, age, and income (83). In general, BRFSS does not assess fewer correlates than the other surveys; thus, additional opportunities may exist for examining how correlates such as cancer history, health status, or health behaviors are associated with screening constructs with this data source.

Data from multiple survey iterations were also used to examine cancer screening trends over time (22% of identified research questions). For instance, NHIS data have been used to examine trends in mammography screening from 2000 to 2008 among women of varying ages (31), and CHIS data have been used to describe colorectal cancer screening rates among different ethnic groups in California over time (96).

Least frequently, data were used to enhance our understanding of cancer screening by linking different data sources (e.g., using data as control cohorts; 15% of identified research questions). In these instances, complex, multifaceted questions about cancer screening were addressed that could not be answered by a single data source. For example, the breast cancer screening practices of women whose mothers had breast cancer were examined in an investigator-designed survey and contrasted with local breast cancer screening rates measured through BRFSS, in an effort to assess the unique effects of the familial cancer experience (86). In other cases, NHIS, HINTS, BRFSS, or CHIS data were compared or integrated with data from various sources such as U.S. Census (e.g., 34, 39), international cancer screening (e.g., 88), or local insurance claims data (e.g., 49). In only 3 cases were data from the surveys under examination, namely NHIS and BRFSS, used together (51, 52, 63).

To describe where cancer screening surveillance has been focused in more recent years and where it may be heading in the future, we examined questionnaires fielded by each survey since 2005. Cancer screening constructs appeared on all of the surveys in the years following 2005, with related content appearing on NHIS in 2008, 2010, 2011, 2012, 2013, and 2014; HINTS in 2007/2008, 2011/2012, 2012/2013, and 2013; BRFSS in 2007, 2008, 2009, 2010, 2011, 2012, and 2013; and CHIS in 2007, 2009, and 2011/2012. We identified 80 new (i.e., content that was not collected by any survey in 2005) constructs (see Supplementary Table S3). Similar to 2005, few constructs were shared across multiple surveys and in no instance was the exact same item wording, response options, and eligibility requirements found.

These 80 new constructs reflect advances and emerging issues in the field of cancer screening surveillance. Multiple constructs relevant to changes in screening guidelines, uncertainty due to conflicting expert recommendations, and a shift toward shared and informed decision-making have recently appeared. For instance, NHIS, HINTS, and CHIS all included measures to assess women's awareness of and discussions with their healthcare providers regarding recent changes to the USPSTF mammography recommendations and the resulting controversy surrounding these changes (17, 104). Similarly, all 4 surveys have included measures assessing some aspect of men's awareness of conflicting risks and benefits of PSA testing, decision-making about this test, and discussions with providers regarding these issues.

Constructs relevant to the delivery and follow-up of screening tests have also begun to appear on these surveys, with measures of the cost of mammography (CHIS); adherence to recommendations for mammography follow-up tests (NHIS); receipt of Pap test results, adherence to recommendations for Pap follow-up tests, perceived utility of reminders for Pap tests (NHIS); experience of having a discussion about colorectal cancer screening tests with different types of healthcare providers, and perceived efficacy of different colorectal cancer screening tests (HINTS). Furthermore, there has been a greater emphasis on distinguishing between different types of endoscopic colorectal cancer screening tests.

Finally, a few of the surveys have begun to collect data regarding emerging cancer screening tests. NHIS has included measures of the use of breast MRI to screen for breast cancer and CT colonography to screen for colorectal cancer. Both NHIS and HINTS have incorporated measures of respondents' awareness and use of lung cancer screening tests—and given the recent positive findings of the National Lung Screening Trial (105), such measures will likely be adopted by the other surveys in future years. Measures regarding predictive cancer genetic testing have also been incorporated into recent iterations of NHIS and HINTS, including measures of respondents' awareness, use, beliefs, and discussions with providers regarding genetic tests that can identify an elevated risk for developing hereditary forms of cancer (e.g., hereditary breast and ovarian cancer). (Constructs relevant to genetic testing did appear on the 2000 and 2005 iterations of NHIS. However, since predictive genetic testing is not generally categorized as a traditional cancer screening test, we have elected to discuss such measures in the context of important future directions.)

The NHIS, HINTS, BRFSS, and CHIS surveys all provide data regarding the prevalence of important cancer screening behaviors, as each includes items designed to evaluate the use and timing of recommended cancer screening tests (1, 14–17). Yet, our review identified differences in the specific wording, response options, and respondent eligibility requirements for these items. It may be worthwhile for survey sponsors to strive for greater consistency in the construction of such items—not only because item wording and response option variations (in terms of both the range and order of options presented) may impact respondents' answers in unintended ways (106), but also because establishing a “gold standard” of measurement for public health researchers could ultimately allow for innovative data analyses that combine data. For example, common items would facilitate the integration of datasets and analyses that could not be done with data from any one of the surveys. Such integration could generate larger sample sizes (which are especially useful when trying to obtain sufficient samples for hard-to-reach populations) or allow for novel comparisons to be made across datasets (e.g., comparing subpopulations of respondents unique to each dataset, or testing for effects of methodologic differences or potential sampling biases on outcome estimates). Although several studies have integrated survey data to answer larger questions about previously unexplained statistical associations (51, 52) and to evaluate large-scale public health programs (63), many additional opportunities for innovative analyses exist. Our efforts to document the characteristics and content of these surveys will help promote this goal but must be supplemented by training opportunities and analytic guidance for working with complex, integrated datasets.

Some differences observed among the surveys reflect the unique contributions of NHIS, HINTS, BRFSS, and CHIS. These surveys vary in their focus and consequently in the depth and breadth of their measurement of cancer screening constructs such as abnormal test follow-up, provider test recommendations, and respondent attitudes and perceptions of screening. Furthermore, they differ in their measurement of theoretical and empirical correlates of cancer screening. For example, both NHIS and CHIS include extensive measures of sociodemographic characteristics, healthcare access and utilization, and health status—consistent with their roles as leading sources of population estimates for the health status and health services use of people in the U.S. and California, respectively. BRFSS, which is designed to assess behaviors associated with premature morbidity and mortality, collects detailed information regarding health behaviors and some common health conditions. HINTS predominantly focuses on people's understanding and use of cancer information; thus, HINTS collects extensive information regarding people's cancer-related attitudes, beliefs, social relationships, and discussions with healthcare providers. Each survey provides novel information that is distinct, yet complementary, to that provided by the others, while minimizing respondent burden. Each surveillance system makes an important contribution that allows researchers and other stakeholders to investigate the complexities of cancer screening at a population level.

Our review identified 3 primary categories of research questions that have been addressed with data collected by NHIS, HINTS, BRFSS, and CHIS in 2005: examining correlates of cancer screening behaviors, exploring cancer screening trends over time, and enhancing our understanding of cancer screening by linking data sources. Although a number of important questions have already been addressed with these data, researchers could continue to use these resources to answer novel questions (e.g., investigations of the prevalence, drivers, and consequences of guideline-concordant and guideline-discordant screening, characterization of screening experiences in subgroups such as the growing cancer survivor population or older adults with various comorbidities). Furthermore, researchers can continue to build on these approaches, both by undertaking innovative analyses and by capitalizing on the variety of new cancer screening constructs that have appeared in recent years. For instance, these data could be used to examine correlates of and trends in the uptake of novel screening tests and changes in screening behaviors in response to shifting policies (e.g., healthcare reform, tailoring of screening recommendations to specific subgroups based on factors such as age), and linked with other data sources for modeling to determine the cost effectiveness or resource demands of different screening-related scenarios. Although the nation's uptake of novel and emerging tests to detect breast, colorectal, and lung cancer as well as genetic predisposition to cancer is likely low at the present time, large-scale surveys are providing valuable baseline data that will allow researchers to better understand how these tests can be integrated into preventive health services in the future. It is noteworthy that in the few instances where constructs were shared across surveys, there was variation in item wording, response options, or respondent eligibility requirements. Although some item differences arise as a consequence of the surveys' different modes of administration, we at the NCI will evaluate future items with an eye toward consistency. Achieving greater consistency in the basic items of greatest interest to multiple stakeholders would pave the way for future innovative and integrative analyses while still maintaining the unique strengths of the individual surveys. In the meantime, we hope that this article will provide researchers with more in-depth knowledge of these data resources, enabling them to capitalize on these surveys to a greater extent by addressing new questions in novel ways and thus further our understanding of cancer screening in the U.S.

No potential conflicts of interest were disclosed.

The content of this publication does not necessarily reflect the views or policies of the Department of Health and Human Services, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. Government.

The authors thank Drs. Erik Augustson, William Klein, April Oh, and Heather Patrick for their valuable feedback during the development of this project.

J.G. Hamilton was supported, in part, by the National Cancer Institute Cancer Prevention Fellowship Program.

The costs of publication of this article were defrayed in part by the payment of page charges. This article must therefore be hereby marked advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.