Yield of Claims Data and Surveys for Determining Colon Cancer Screening among Health Plan Members

Colorectal cancer is the second leading cause of cancer death in the United States, with 148,810 new cases and 49,960 deaths expected in 2008. Screening for colorectal cancer has been shown to reduce the incidence and mortality from colorectal cancer (1). Use of colorectal cancer screening has increased since 2002, but current self-reported screening rates remain relatively low: 60.6% of age-eligible adults reported being up-to-date with screening in 2006 (2, 3).

Accurate measurement of colorectal cancer screening is important for documenting trends in colorectal cancer screening and for informing and evaluating improvement efforts. All current sources for data on colorectal cancer screening are imperfect. Self-reported data have the advantage of coming directly from the patient but may be limited if the patient does not understand the question, cannot differentiate between types of tests, or is unable to recall the specific timing of an event several years ago. Medical records have the advantage of being recorded by health care providers but are limited by deficiencies in documentation and because care may take place in multiple settings. They can also be expensive to obtain if the provider does not have an electronic record that is easy to query. Claims data have the advantage of being easier to obtain but only reflect tests performed and billed through that payer during the window of time used for the database, which may be shorter than the recommended screening interval, particularly for colonoscopy. Thus, combining claims data with self-report of screening tests may result in a more accurate estimate of screening prevalence by including tests performed through other payers or conducted outside of the claims data “window.”

Several studies have attempted to compare the different sources of data for colorectal cancer screening to assess their agreement (4-9). Although there is no currently agreed-upon “gold standard,” investigators have also measured the sensitivity and specificity of different methods compared with one another and have reached mixed results. Few studies have examined the relationship between self-report and claims data among members of large insurers that are not part of a staff-model system (9).

We examined colorectal cancer screening test completion for average-risk members of a large national health insurer from claims data and then we examined the extent to which members reported colorectal cancer screening that was not captured by claims data. We also examined the factors associated with self-reported colorectal cancer screening not found in the claims data, which may help to explain discrepancies between claims data and self-report.

Data for this research come from the Communicating Health Options through Information and Cancer Education (CHOICE) Study. CHOICE is a cluster-randomized trial in primary care practices in Georgia and Florida that is testing the effectiveness of a patient-directed decision aid and practice-level academic detailing for increasing adherence to colorectal cancer screening guidelines. The study is a collaboration between two university research groups (at Emory University and University of North Carolina) and the Southeast Region of Aetna Health Management, LLC.

To identify potential practices for participation, the study team obtained a list from Aetna of primary care physicians (family practice, general practice, or internal medicine) in the Atlanta, Tampa, and Orlando areas who were participating in the fully insured HMO or Emory CHOICE products. The list included the number of HMO or Emory CHOICE members ages between 52 and 75 years who had selected the physician as their primary care doctor. Physicians in the same medical practice (with the same address) were linked together. Medical practices recruited to the study had a minimum of 50 potentially eligible patients ages 52 to 75 years. Of 310 practices contacted, 15 practices from Georgia and 17 practices from Florida agreed to participate in the study.

Approval for the study was obtained from the Emory University Institutional Review Board and included a partial Health Insurance Portability and Accountability Act waiver to identify potential study participants (persons not current on colorectal cancer screening) through the claims database.

To identify potentially eligible participants, we obtained claims data from Aetna for members ages between 52 and 80 years whose primary care physicians had agreed to participate in the CHOICE study. We expanded the age range to 80 to include all members who might be eligible for screening. The claims extracts were received in three waves across 2005 and 2006 as practices were recruited into the study. A final claims extract was obtained in January 2007 to identify newly eligible members in participating practices. Each extract included data for the current year and the 3 previous years, which is the time frame contained in the member claims database.

We obtained claims information on previous colorectal cancer screening, including fecal occult blood test (FOBT), sigmoidoscopy, colonoscopy, or barium enema (10). We collected information on all screening tests because some individuals had more than one type of test. Data elements extracted included the most recent screening date (if any) for each of the four types of colorectal cancer screening and the total number of claims records for each of the exclusionary conditions.

Exclusions included individuals at increased risk for colorectal cancer (because the intervention was designed for average-risk patients) or with medical conditions that would limit their ability to participate in the study or who might not be considered reasonable candidates for screening. Above average-risk persons were defined as adults with a personal history of colorectal cancer or polyps, a known history of colorectal cancer or polyps in a first-degree relative, or a known history of inflammatory bowel disease (11). Other conditions warranting exclusion included dementia, chronic obstructive pulmonary disease, heart failure, coronary artery disease, current treatment for cancer or history of metastatic cancer, cirrhosis, upper or lower gastrointestinal bleeding, unintentional weight loss of >10% within 6 months, blindness, or uncorrectable hearing impairment. To be excluded, the individual had to have at least two claims with either a diagnosis or procedure code indicating that they had one of these additional conditions. A list of the specific codes used is available from the authors.

From the original claims data extracts, we obtained data on 5,759 Aetna HMO members (see Fig. 1). We excluded 1,739 people due to having evidence for one or more exclusion criteria (e.g., claims data indicated that 1,119 had been previously treated for cancer, 495 had a history of coronary artery disease, 171 had a history of inflammatory bowel disease, 158 had claims for gastrointestinal bleeding, and 157 had hearing impairments). Thus, our study population for determining colorectal cancer screening status from claims data was 4,020 individuals.

The 2,954 members who were thought to be at average risk and who did not have claims for current colorectal cancer screening were sent a letter and a brief eligibility survey. The eligibility survey was used to obtain information about receipt of colorectal cancer screening or increased risk (e.g., relatives' history of colorectal cancer) that may not be in the claims data. The survey asked about individuals' completion of colorectal cancer screening (whether they had ever had colorectal cancer screening tests and when the tests were most recently completed: within the previous year, 1-5 years, 5-10 years, or >10 years) as well as additional demographic information not in the claims data (education, income, race/ethnic group, smoking status, and routine physical activity), medical and family risk factors, health behavior and medical care utilization, and self-reported health. The primary method of survey administration was mail, but, after two mailings, nonresponders were invited to complete the survey online or by telephone interview. Respondents received a US$10 gift card to thank them for completing the survey.

After exclusion of 396 additional members who were found to be ineligible, the final eligible survey sample was composed of 2558 members. Of those surveyed, 57% were women, 73% were ages 52 to 59 years, 21% were ages 60 to 64 years, and 7% were ages ≥65 years. We received 1,595 survey responses (72% written surveys, 22% by phone, and 6% online) for a final response rate of 62%. The response rate differed somewhat by gender (67% for women versus 57% for men) but not by age: it was 62% for those ages 52 to 59 years, 62% for those ages 60 to 64 years, and 66% for those ages ≥65 years. Response rates were higher for members living in Georgia than for those in Florida (69% versus 58%).

The primary outcome of interest was the proportion of members who were considered up- to-date with colorectal cancer screening. Being up-to-date was defined as having evidence of FOBT within 1 year, sigmoidoscopy or barium enema within 5 years, or colonoscopy within 10 years. We examined the yield of the claims data alone and the combined yield of claims data and the eligibility survey.

Using descriptive analyses, we first examined colorectal cancer screening from claims data for the average-risk members. For those whose claims records did not show evidence of screening, we then examined self-reported colorectal cancer screening from their surveys.

Based on these two sources of data, we then created a combined screening estimate for the study population, taking into account nonresponders to the eligibility survey. That is, we calculated the denominator by taking the total of all age-eligible members (n = 5,759) minus those with medical exclusions per claims (n = 1,739), those who were determined to be ineligible after we mailed the eligibility survey (n = 396), and those who were at above average risk according to self-report (n = 326), leaving 3,298 for further analysis. We then estimated the minimum (lower estimate) colorectal cancer screening rate by adding the total screened per claims (n = 1,066) and the total screened per self-report (n = 498) divided by the adjusted denominator (n = 3,298), assuming no screening among the survey nonresponders. The maximum (upper estimate) screening rate was obtained by assuming that the nonresponders (n = 963) were screened at the same rate as the average-risk responders.

We also examined factors associated with screening. For the initial claims sample, we examined basic demographic characteristics that were associated with screening. For analyses of factors associated with self-reported colorectal cancer screening among those with no claims report of screening who completed the eligibility survey, we used χ² tests to assess significant bivariate relationships between self-reported screening status and a larger set of demographic and health characteristics (from the survey). We then conducted logistic regression analysis to determine the independent relationship between self-reported colorectal cancer screening status and demographic and health characteristics among average-risk members.

Claims data indicated that 27% of average-risk members (1,066 of 4,020) were current for colorectal cancer screening. Colonoscopy was the most common test: 789 members (74% of those up-to-date with screening) had evidence for colonoscopy in the available claims data. Claims for FOBT within 1 year were noted for 375 (35%) members. Among those tested, 648 had only colonoscopy, 246 had only FOBT, and 123 had evidence of both recent FOBT and colonoscopy.

Of the 1,269 average-risk members with no claims evidence for current colorectal cancer screening, 498 (39%) reporting being up-to-date with colorectal cancer screening (see Fig. 1). Colonoscopy was again the most frequently reported test: 365 (73%) respondents responded that they had received a colonoscopy within 10 years, with the majority (296) reported as being within 5 years. FOBT within 1 year was reported by 119 (24%); other tests were much less frequently reported.

Compared with the claims data sample, the survey responders were similar in terms of gender (58% of claims sample and 60% of survey responders), state of residence (slightly higher percentage from Florida: 56% of claims sample and 54% of survey responders), and age range (higher percentage for those ages 52-59 years: 72% of claims sample and 70% of survey responders; see Tables 1 and 2). Furthermore, our survey sample was 75% White and 19% African American, with a wide range of education levels. Most of the survey respondents reported either good (42%) or very good (34%) health status, and only a minority were current smokers (13%; see Table 2).

Combining claims data (which found 27% screening) and self-reported data for colorectal cancer screening, and taking nonresponders into account, the estimates of total colorectal cancer screening rates for average-risk members in the HMO ranged from 47% to 59%.

Claims Data. For the 4,020 beneficiaries in the claims analysis, we observed demographic differences in being up-to-date with screening. Women were somewhat more likely than men to have current colorectal cancer screening (28% of women versus 24% of men; P = 0.006). Those living in Georgia were more likely than those living in Florida to be current on colorectal cancer screening (29% Georgia versus 25% Florida; P = 0.003). There were no statistically significant differences in screening rates for different age groups.

Survey Data. In bivariate analysis of those completing the eligibility survey (all of whom had no claims record of screening), members who reported current colorectal cancer screening were more likely to be older, nonsmokers, and have higher education levels (Table 2).

From the survey data, we found that the relationships between self-reported colorectal cancer screening and age, education level, and smoking status remained significant in the multivariate analysis (Table 3). The final logistic regression model showed a positive association between self-reported colorectal cancer screening and each older age group, with higher likelihood of reporting colorectal cancer screening rates in each successive age group [ages 60-64 years: odds ratio (OR), 1.59; 95% confidence interval, 1.20-2.11; ages 65-69 years: OR, 2.34; 95% confidence interval, 1.45-3.79]. We found that the highest level of education (postgraduate and professional degree) was positively associated with self-reported colorectal cancer screening (OR, 1.66; 95% confidence interval, 1.14-2.41) and that being a current smoker was negatively associated with self-reported colorectal cancer screening (OR, 0.56; 95% confidence interval, 0.38-0.83).

We found that, among health plan members, the combination of 3 years of claims data and survey information had significantly higher yield than claims data alone for identifying colorectal cancer screening. We estimated the additional yield from the survey to be ∼20 to 30 percentage points, after accounting for survey response rate. Beneficiaries who had evidence of screening by claims were slightly more likely to be female and reside in Georgia. Among those with no claims evidence of screening, older beneficiaries, nonsmokers, and those with more education were more likely to report screening. We did not find differences by race in this insured, mainly biracial sample.

Most of the reported colorectal cancer tests were colonoscopy, with FOBT second most common. Among the survey respondents, the majority of colonoscopies were reported as having been performed within 5 years, suggesting that the truncated time window of available claims data was not the main cause of the discrepancy between claims and survey data.

Our findings add to a growing literature examining different data sources for measuring colorectal cancer screening status. Baier et al. at Kaiser Permanente in Colorado compared self-reported screening elicited through a computer-assisted telephone interview with data from their administrative claims for 329 patients ages between 40 and 75 years who had been enrolled in the health plan for at least 48 months (4). They found that self-report was highly sensitive (96% for FOBT and 96% for endoscopy) and reasonably specific (86% for FOBT and 92% for endoscopy) when compared with claims data. Lipkus et al. compared self-report against actual receipt of FOBT cards or claims data, indicating test completion among 658 members of the carpentry trade participating in a trial (5). They found that reporting having performed “a FOBT test to screen for colorectal cancer only” was moderately sensitive (69%) for actually completing the test within the past year; specificity was 85% and concordance was 74%.

Fiscella et al. examined the relationship between self-report and claims data for several aspects of preventive care, including colorectal cancer screening, for participants in the Medicare Current Beneficiary Survey. They found fair agreement for White patients (κ = 0.37), but agreement for minority patients was lower (κ = 0.19), with most disagreement resulting from positive self-reports not verified by claims records (6).

Schenck et al. recently compared data from a survey of 561 Medicare beneficiaries in North and South Carolina, chart reviews of the medical records from their main physicians, and Medicare claims over the previous 5 years to identify endoscopic testing for colorectal cancer. They found excellent agreement between claims and office medical records (>90%). There was good agreement between self-report and the office medical record (79%) as well as between self-report and claims information (74%). Rates obtained by self-report were slightly higher than those obtained by claims data (50% versus 45%). Using claims data as the gold standard, self-report had 73% sensitivity and 68% specificity for an endoscopic exam. The κ was 0.40, indicating fair agreement (7). More recently, Schenck et al. published a similar analysis of stool testing and found that medical records and Medicare claims had 82% agreement, self-report and medical records had 70% agreement, and self-report and claims had 67% agreement. They concluded that no one data source was sufficient to accurately measure screening performance (8).

Schneider et al. examined the relative yield of claims data, medical record review, and beneficiary surveys for evidence of colon cancer screening among >189,000 members of five large health plans and found that the different data sources had considerable variation in yield (9). Agreement between a hybrid assessment method and the survey was modest (κ = 0.34). The survey response rate was 48%, and screening rates by the hybrid method were higher for those who responded to the survey versus nonrespondents (63% versus 46%), suggesting that survey response bias may affect results (9).

This study has several limitations. First, we did not have self-report data on those with claims indicating they had been screened, so we could not assess agreement between claims and self-report. Second, we could not examine in detail factors associated with having a claim or not having a claim related to colorectal cancer screening. Third, because we were unable to measure self-report in survey nonresponders, we can only estimate the additional yield of self-report based on assumptions about the actual screening rates among nonrespondents. Our survey data were limited, and many other factors potentially associated with screening or lack of screening were not examined. Our survey questions asked about the timing of screening relative to guidelines (e.g., 1, 5, and 10 years), so it was not possible to determine how many tests were not reported in claims data due to the 3-year “window” for these records. Also, it was not possible in this study to distinguish between routine screening and diagnostic tests for colorectal cancer, although even tests performed for diagnostic purposes can be considered as fulfilling the requirements for screening. Our study population comes from members of one health plan, in one region of the United States, and were drawn from providers who had agreed to participate in the larger CHOICE study. The screening levels, additional yield of the survey, and factors associated with screening may differ in other populations.

Finally, the additional yield of self-reported screening must be viewed in the context that there is no single “gold standard” for screening. As is clear from our study and the others described above, all data sources have the potential for error and it is possible that some of the additional reports of screening in our survey are incorrect. Nevertheless, our findings suggest that quality improvement programs, intervention trials, or entities such as Healthcare Effectiveness Data and Information Set (HEDIS) that seek to evaluate health plan performance cannot rely solely on claims data if they wish to identify patients in need of screening. Future efforts should examine techniques for obtaining and maintaining better, more comprehensive data to accurately categorize screening status. Such sources could include shared databases among insurers, personal health records, or shared electronic medical records. In addition, outreach efforts to improve screening should account for imperfect information and give individuals and providers the opportunity to update incorrect information easily. Doing so will allow quality improvement efforts to focus their efforts on those who are not up-to-date with screening.

No potential conflicts of interest were disclosed.

The costs of publication of this article were defrayed in part by the payment of page charges. This article must therefore be hereby marked advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.