Validation of Self-reported Colorectal Cancer (CRC) Screening in a Study of Ethnically Diverse First-Degree Relatives of CRC Cases

The most commonly used outcome in cancer screening intervention studies is self-report. However, prior research has found that self-report of cancer screening may be inaccurate and prone to biases (1, 2). Therefore, validation of self-reports through medical chart audit has long been viewed as the “gold standard” for assessing the veracity of self-reports. Most validation studies have focused on breast and cervical cancer screening, and therefore, relatively little information is available about the accuracy of self-report for colorectal cancer (CRC) screening (1). Results of the handful of validation studies focused on CRC screening have yielded inconsistent results. Several studies have found that self-reports of endoscopy may be fairly accurate, with high concordance between self-report and medical records (2-5). The accuracy of self-report may be higher for colonoscopy than sigmoidoscopy (4, 6), and concordance between self-report and medical records has been found to be substantially lower for fecal occult blood testing (FOBT; refs. 2-4). Although overreporting is suspected across CRC screening tests, one study found that subjects underreported versus overreported FOBT receipt (7).

The majority of studies validating self-report of cancer screening have been conducted in health maintenance organizations, within a single health care system, or with patients enrolled in large cohort studies (2-5). In these settings, researchers can often directly access medical charts, including billing information, procedure documentation, and physician notes. However, many cancer screening intervention studies are conducted outside of these settings. Cancer screening interventions are often delivered remotely to individuals recruited through cancer registries, surname lists, and random digit dialing. In these settings, subjects are likely to receive cancer screening at numerous diverse settings dispersed over broad geographic areas. In addition, many cancer screening intervention studies are conducted in community settings. This approach is necessary to reach populations that may not already be connected with the health care system, including low-income, uninsured, and ethnic minority populations. Nonclinical settings for cancer screening intervention research have included churches, community-based organizations, social clubs, barber shops, and work sites. The majority of these studies do not arrange for subjects to receive screening from any particular site. Consequently, subjects may be screened in various settings within the community, including individual physician offices, community clinics, health fairs, and county health department facilities. Researchers in these settings face many challenges to validating self-reports of screening, including being dependent on health care providers to provide the necessary information and having to navigate many different health care settings to receive the requested information.

We describe here the results of validation activities that were conducted in the context of a study that evaluated the effect of a print and telephone counseling intervention on CRC screening rates in a sample of first-degree relatives recruited through a population-based cancer registry. Only one prior study in the literature has validated self-reported CRC screening among individuals with a family history of CRC. That study was conducted in Canada (4) and found that colonoscopy was reported very accurately, but that, in comparison, self-reports of sigmoidoscopy and FOBT were less reliable. In addition, no prior studies have assessed the validity of self-reported CRC screening in the context of a screening intervention trial.

The parent study was a two-group randomized intervention trial to increase CRC screening within an ethnically diverse sample of first-degree relatives of CRC cases. First-degree relatives were recruited by initially contacting a random sample of CRC cases identified through the state-wide California Cancer Registry. African-American, Asian, and Latino cases were oversampled to achieve ethnic diversity in the study population. Cases were asked to provide information about their first-degree relatives. A total of 1,645 cancer cases identified 5,073 first-degree relatives. Baseline telephone interviews were conducted with eligible first-degree relatives referred to the study. Eligibility included the following: 40 to 80 years of age; living in the United States or Canada; English or Spanish speaking; sister, brother, or child of case; no personal history of CRC; and no history of very high-risk syndromes such as inflammatory bowel disease, hereditary nonpolyposis CRC, and familial adenomatous polyposis. Of the 3,667 eligible relatives, 2,595 (71%) completed the baseline survey. Of these, 49% (n = 1,253) had obtained CRC screening recently, defined as a FOBT in the past year, and/or sigmoidoscopy in the past 5 years, and/or colonoscopy in the past 10 years. Another 2% (n = 62) were excluded from this analysis because ethnicity was missing or they were not African-American, Asian, Latino, or White. The remaining 1,280 first-degree relatives (siblings or children) who did not report recent CRC screening constituted the analytic sample for the trial. A two-group nested design was used. All subjects within the same family were assigned to the same study arm to decrease potential contamination. The intervention group received a tailored print intervention within 2 weeks of the baseline interview. Both groups were contacted 6 months after baseline to assess screening receipt, and intervention subjects not adherent to screening at 6 months received a telephone counseling intervention appended to the end of the telephone interview. Twelve-month follow-up interviews were conducted with both groups. Of the 1,280 nonadherent subjects randomized to the intervention, 80% (n = 1,030) completed the 6-month interview and 74% (n = 948) completed the 12-month interview. The control group received no intervention until after the trial was completed. Study data were collected from September 1999 to April 2004. The study protocol was approved by the University of California at Los Angeles Institutional Review Board. Additional details about the study results are forthcoming.

At the end of the 12-month follow-up telephone interview, all subjects were asked to allow the study team to contact their health care provider to verify receipt of screening during the study period. Subjects were informed that we would like to contact their provider regardless of whether they received a recent screening test and that we would like to obtain contact information (i.e., name, address, and telephone number) for the provider (physician or clinic) they had seen for CRC screening or their usual provider (if they were not screened during the study period). All subjects who verbally agreed to the validation activities (567 of 948 subjects at follow-up) were sent a validation form to sign and return to our office. The top portion of the form was pre-populated with participant information such as name and address and included a place for the participant's signature indicating agreement to participate in the validation protocol. The bottom portion of the form was for the medical provider to indicate participant receipt of CRC screening. Health care providers were sent the signed validation form, a cover letter briefly explaining the study and the need for validation, a study abstract, and $10 to offset the cost of completing and returning the form (e.g., staff time). Health care providers who did not return the validation form were contacted by telephone. Multiple attempts were made to contact each provider by telephone. Providers were allowed to submit the information via mail and fax or provide information over the telephone.

Self-reported CRC Screening. At the 6- and 12-month follow-up interviews, subjects were asked if they had received a FOBT, sigmoidoscopy, or colonoscopy since their last telephone interview (i.e., baseline or 6-month interview). A brief definition of each test was provided before assessing screening receipt. Subjects reporting any of the tests at the 6- or 12-month follow-up were asked if it was for screening or diagnostic purposes. Only subjects who had had a test for screening purposes (routine checkup, no symptoms, and no prior abnormal findings) were considered “screened” for both the intervention and validation studies.

Physician Report of CRC Screening. The bottom half of the validation form included space for the health care provider to indicate the two most recent dates of CRC screening (FOBT, sigmoidoscopy, and colonoscopy). Providers were also asked to indicate the reason for the test: the screening of an asymptomatic patient versus an indicated procedure following symptoms or abnormal finding on test. They were also asked to indicate whether the test result was positive or negative.

We compared subgroups of subjects created by inclusion versus exclusion from the various steps involved in the validation process (depicted in Fig. 1) using χ² tests to assess bivariate relationships and logistic regression for multivariate comparisons. All variables listed in Table 1 were included in the multivariate analyses, except for income (due to substantial missing data) and country of birth (applicable only to Latinos and Asians). Because all subjects within the same family were randomized to the same study arm (the average family size was 1.4), we used the GENMOD procedure, with a logit link, in Statistical Analysis System (Windows version 9.1) for our multivariate analyses. This procedure fits models to correlated responses using generalized estimating equations equivalent to logistic regression with SEs adjusted for correlated data. Concordance was calculated by assessing the percent total agreement (% of instances when self-reports matched provider reports) between self-report and medical records for receipt of FOBT, sigmoidoscopy, and colonoscopy during the 12-month study period. We also calculated sensitivity (% true positives), specificity (% true negatives), positive predictive value (PPV; % of self-reports of screening verified by provider report), and negative predictive value (NPV; % of self-reports of “no screening” verified by provider report). The exact procedure in Stata 9.0 was used to calculate the 95% confidence intervals (95% CI) for the various measures of agreement (8). Fisher's exact test was used to compare the differences in measures of agreement between the intervention and control group on FOBT, colonoscopy, and any screening.

Figure 1 depicts the response rates for the validation activities. Of the 948 individuals who completed the 12-month survey, 567 (60%) verbally agreed to validation and were sent a validation form for signature and return to our office. Of these, 171 (30%) subjects returned the signed form. We were able to validate screening status for 123 (72%) of this group based on information provided by 121 different physicians. Due to the dropout that occurred at the various steps in the process, we were able to complete validation for only 13% of the total sample or 22% of those who verbally agreed to participate in validation.

Characteristics of the total follow-up sample (n = 948) appear in the first column of Table 1. Of the 948 subjects in our 12-month follow-up sample, 771 (81.5%) resided in California, and the remaining 177 were distributed among 37 other states across the country (data not shown). Among the subjects residing in California, 231 resided in Los Angeles County. Thus, 30% of the California residents and 24% of the total sample resided in Los Angeles.

To examine the representativeness of our validation sample, we compared the characteristics of the subgroups created by the various steps in the validation process both bivariately (Table 1) and multivariately (Table 2). First, we compared those who verbally agreed to participate in the validation protocol to those who did not (Table 1, columns B versus C). The 45 subjects who indicated that they had no provider were placed in the “did not verbally agree” category because bivariate analyses indicated that these two groups did not differ on demographic characteristics. Subjects who verbally agreed to validation were more likely, than subjects who did not agree, to have health insurance and to self-report CRC screening during the study period. Multivariate logistic regression analyses (Table 2) found that insurance status [odds ratio (OR), 2.75] and self-reported screening (OR, 5.14) remained independent predictors of verbally agreeing to validation. Within the subgroup of 567 subjects who agreed to the validation, we compared the 171 subjects who returned the signed validation form to the 396 subjects who did not (Table 1, columns D versus E). Bivariately, subjects who returned the validation form and provided medical release were more likely to be White or Asian compared with Black or Latino, have higher levels of education, have been born in the United States (only calculated for Latinos and Asians), and self-report CRC screening within the study period. Ethnicity (OR, 0.54 for African-American versus White), education (OR, 1.96), and self-reported screening (OR, 3.07) were retained in the multivariate model (Table 2). There were no statistically significant differences between the subgroup for whom physicians returned completed validation forms (n = 123) and those subjects for whom physicians did not respond (n = 48; Table 1, columns F versus G). Finally, we made a comparison between the 123 subjects for whom we were able to complete the validation process and the remainder of subjects (n = 825) in our 12-month follow-up sample (Table 1, columns G versus H). Bivariate analyses indicated that members of the final validation sample (n = 123) were more likely than the remainder of subjects (n = 825) to be White and Asian, to have higher levels of education and health insurance, to self-report CRC screening during the study period, and to have been born in the United States (among Latinos and Asians only). Education (OR, 1.88) and screening status (OR, 3.46) were the only variables retained in the multivariate analysis.

Stratified bivariate logistic regression analyses conducted separately among Latinos and Asians indicated that country of birth was statistically significantly associated with participation in validation activities only among Asians (data not shown): compared with foreign-born Asians, those born in the United States were more likely to agree verbally to be part of the validation study (OR, 2.14; 95% CI, 1.17-3.91; P < 0.05), to return the signed medical release form (OR, 4.23; 95% CI, 1.79-9.98; P < 0.05), and to be in the final validation sample (OR, 4.74; 95% CI, 2.00-11.27; P < 0. 05). Country of birth was not a statistically significant predictor of participation among Latinos.

Table 3 displays CRC screening rates obtained in the study for the validation sample (n = 123) by provider and self-reports as well as the self-reported screening rates for the 825 subjects for whom validation was not conducted. For every type of screening test, except sigmoidoscopy, the self-reported rate of screening for the validation sample was substantially higher than the provider rate of screening. For example, 31% of subjects self-reported FOBT screening, whereas the providers reported 12% screening. Furthermore, self-reported rates of screening for the validation sample were higher than for the nonvalidated sample for all tests, except sigmoidoscopy.

Table 4 displays the percent agreement between self-report and provider report for the three types of CRC screening tests separately and for any endoscopy (sigmoidoscopy or colonoscopy). Overall concordance between provider and self-report ranged from “fair” (74%) for any screening to “excellent” (96%) for sigmoidoscopy based on criteria suggested by Tisnado et al. (9). Observed NPVs (proportion of “not screened” self-reports confirmed by physicians as not screened) for all screening tests were excellent. In contrast, the PPVs (proportion of screened self-reports confirmed by physician as screened) for all screening tests were poor. The lowest PPV was observed for FOBT. Provider report corroborated self-report of a FOBT in only 29% of cases. PPVs for sigmoidoscopy and colonoscopy were substantially higher (56% and 55%, respectively) but still poor. Because our work and previous research (3) has found that patients are unsure of the type of endoscopic procedure they received, we examined the rate of agreement for receipt of any endoscopy within the study period (i.e., colonoscopy or sigmoidoscopy). The PPV for any endoscopy was slightly higher (62%) than that for either procedure separately, suggesting that a few subjects may have inaccurately reported which type of endoscopy they had undergone. The PPV for “any screening” (FOBT and/or sigmoidoscopy and/or colonoscopy) was also poor. We also assessed whether provider corroboration of self-reported screening was influenced by including dates of screening reported by physicians that were outside of the study period. Previous research has found that patients often inaccurately remember the timing of medical tests (10). We asked providers to indicate the two most recent dates on which subjects received any of the three cancer screening tests but did not specify the study period. The PPV increased only slightly (<3% on average; data not shown) when comparing self-reports with provider reports of screening that occurred outside the study period. This suggests that inaccurate recall of the timing of tests did not contribute statistically to the poor PPVs observed.

Because this validation study was conducted in the context of a screening intervention trial, we computed the various measures of agreement, by intervention versus control, for FOBT, colonoscopy, and any screening, although small cell sizes warrant caution in interpretation of findings. We did not compute similar information for sigmoidoscopy due to the extremely low prevalence of this procedure in our sample (e.g., one provider report and four self-reports). For FOBT, overall concordance was very similar by intervention versus control (73% and 77%) as was the PPV. Observed sensitivity and specificity seemed to differ by intervention group status; however, substantial overlap in the confidence intervals indicated lack of statistical significance. For colonoscopy, most measures of agreement were lower in the intervention versus the control group, although the differences were not statistically significant, except for the overall concordance rate (76% intervention versus 91% control group). Similar findings were observed for any screening test, with the only statistically significant difference being for specificity (50% intervention versus 79% control).

Our most substantial challenge was obtaining participant written consent to inclusion in the validation substudy. Consequently, we were able to complete the validation with only 13% of our follow-up sample. The parent trial was conducted entirely via mail or telephone contact. Therefore, to obtain written informed consent, as required by our Human Subjects Protection Committee, we obtained verbal consent at the end of our last follow-up telephone interview (12 months), mailed a validation form to those agreeing, and had to rely on subjects mailing back the signed form. The largest dropout occurred at this last step. Although 60% of subjects verbally agreed to the validation, only 30% of those who agreed or 18% of the total sample actually returned the completed signed form. This low response proportion is somewhat surprising given that the vast majority of the sample had already participated in three study interviews and about half had received educational materials and a telephone counseling session as part of the intervention. No telephone reminders were included for this portion of the trial due to resource limitations, which may explain the low response. In addition, all subjects were aware that they were contacted for this study because they were at increased risk for CRC due to family history and that screening was important for early detection. Therefore, subjects who were not screened during the study period (70% according to self-report) may have been hesitant to allow study staff to check with their providers to verify screening, resulting in lower proportions of verbal agreement to participate in validation as well as a lower proportions of return of the medical release forms. This hypothesis is consistent with our findings that subjects who self-reported screening were statistically significantly more likely to participate in validation activities. Subjects may also have been hesitant to allow us to contact their providers due to concerns that it might prompt providers to encourage subjects to be screened, something subjects may or may not be interested in pursuing. In contrast to the low response rate among subjects, the response rate among physicians was fairly high at 72%. This is probably due to the aggressive telephone follow-up that was implemented for this phase of the validation study. We were able to cover the cost of the physician follow-up due to the small numbers involved. This was a labor-intensive process because almost every subject identified a unique physician. Had the subject response rate been higher, it is unlikely that we could have implemented this intensive physician reminder protocol.

Given the low participant response, it was important to examine how the validation sample may have differed from the overall sample on demographic factors and receipt of screening. Overall, at each step in the process, subjects who participated in the validation protocol were statistically significantly more socioeconomically advantaged and less likely to be African-American or Latino than those who declined participation. Among Asian subjects, participants in validation activities were more likely to be born in the United States compared with nonsubjects. Furthermore, subjects who self-reported having received a CRC screening test during the study period were more likely to participate. Given these differences, we cannot generalize the results of our validation to the total sample.

Among the 123 subjects for whom we were able to obtain provider verification of screening status, we observed a wide variation in level of agreement between provider and self-report for the three CRC screening procedures assessed. Agreement also varied by specific screening modality. In general, NPVs were uniformly high for all tests and combination of tests, indicating that provider reports corroborated self-reports when subjects reported no screening. However, PPVs (self-report of screening receipt confirmed by provider report) were low across the board but varied by type of procedure: the PPV was 62% for any endoscopic test and 29% for FOBT. The PPV observed for FOBT in this study was almost identical to that observed by Madlensky et al. (4) who also recruited a sample of first-degree relatives in Canada. This finding of inaccurate self-report of FOBT is reflected in other measures of agreement that we calculated in this study (sensitivity, specificity, and concordance), supports the results of other studies in the literature (2-4), and has been attributed to less thorough documentation of FOBT in medical records and the fact that the test is completed at home and returned to the provider's office or laboratory, increasing the possibility of misfiling or misplacing results. It is also possible that subjects could not accurately recall the physician to whom they returned the FOBT samples. For subjects who had an endoscopy, multiple physicians may have been involved in the process (e.g., primary care physician made the referral, whereas gastroenterologist conducted the procedure); however, we sought validation from only one physician. The overreporting of all three tests that was observed in our study is generally supported by the literature (1, 2, 4-6, 11).

It is important to keep in mind that the findings of this study may be specific to the characteristics of the population we recruited, the time interval over which we assessed screening, and the fact that the data were collected from subjects participating in a screening intervention trial. Most previous studies have been conducted with average-risk participants. It is possible that first-degree relatives, aware of their increased need for screening, may overreport screening more frequently than average-risk populations, although no data are available to directly answer this question. However, the levels of agreement we observed in our study for sigmoidoscopy, colonoscopy, and endoscopy are higher, in almost every instance, than those in a recent study by Schenck et al. (6) in an average-risk sample. Although multiple factors probably contributed to these differences, one contributing factor may be that our study attempted to verify receipt of screening over a much smaller time interval (1 year versus 5 years).

This is one of the first studies to examine the accuracy of self-reports in the context of an intervention trial. This is of particular importance in community intervention trials because the majority of such trials rely on self-reports to assess the efficacy or effectiveness of the interventions being tested. There is little available information about how screening promotion interventions may influence the accuracy of self-report. Although we attempted to examine whether the accuracy of self-report differed by intervention versus control group assignment, we hesitate to draw conclusions given our overall small sample size coupled with very small and unbalanced cell sizes. Our results suggest that, if a bias according to group assignment exists, it is likely to be more prominent for self-report of endoscopy versus FOBT. Overall, intervention and control groups were not statistically significantly different in all but two instances. There was a statistically nonsignificant trend toward higher accuracy in the control group compared with the intervention group, suggesting that intervention trials may overestimate effect sizes if they rely on self-reports to assess screening outcomes. Further research is needed to fully examine this troubling possibility.

The norm in cancer screening validation research is to regard the provider report as the gold standard against which the accuracy of self-report can be evaluated. However, the majority of studies that have evaluated the accuracy of self-report have been conducted in settings very different from the present one. Most studies have taken place within the context of health maintenance organizations (3, 11, 12), within a singular health care system (7), or with subjects in a limited geographic area attending a limited number of health centers (5, 13). In these settings, researchers can often directly access medical charts, including billing information, procedure documentation, and physician notes. In contrast, our subjects were drawn from 38 different states and there was practically no overlap among the identified physicians. We did not have access to subjects' medical records and had to rely on subjects to provide the correct information about their health care provider and to return the signed medical release. We then had to rely on the health care provider to gather information from any pertinent sources that might document screening (i.e., procedural notes, billing documentation, and clinical notes). The accuracy of provider report in this situation is not clear. Furthermore, validation in nonclinical settings in which subjects receive care from numerous providers has low feasibility. The cost involved in conducting a validation study in such a situation may not be sustainable within the budget limitations of a typical community intervention trial. For the particular protocol we used, further research may be warranted to determine whether investing in more intensive follow-up with subjects who did not return the signed consent forms could result in a more representative study sample.

The major limitation of our study was that we were able to conduct validation activities on only a very small proportion (13%) of our intervention trial sample, thus calling into question the representativeness of our findings. Future studies will need to invest more resources into achieving a higher subject response rate. Repeated mail and/or telephone reminders accompanied by monetary or other incentives for returning medical release forms may be required. The fact that we used aggressive follow-up accompanied by a $10 incentive for physicians may have been responsible for the higher provider response rate (72%) achieved. Providers may also have been motivated to provide the information because the validation form included their patient's signature. If resource constraints are an issue, it may be more efficient to focus intense validation efforts on a subsample of the intervention trial sample rather than trying to obtain validation data on all subjects. For example, validation studies could select all subjects self-reporting screening and a random subsample of other subjects. Validation studies may also be more feasible if associated with community trials that take place in geographic areas where there are fewer providers and significant proportions of subjects share the same providers or health care facilities. This would increase the familiarity of the study team with the protocol for obtaining medical record information from various health care settings and may increase the responsiveness on the part of both subjects and provider offices. In addition, although validation of self-report of screening through medical record reviews may in theory be a goal for every screening study, it may be more practical to gather information about the accuracy of self-report from studies that primarily focus on this issue and thus have the resources available to implement a rigorous protocol (14). Funding agencies will need to be more willing to provide additional resources, over and above those available for implementing the intervention trial, to include validation protocols in community intervention studies.

The results of our study seriously call into question the feasibility of attempting to conduct validation of self-reports in community settings in the United States in which subjects are dispersed over a very broad geographic area and receive care from numerous nonoverlapping providers and there are no centralized records. In this context, direct access to patient medical records is often not available and instead researchers must rely on the voluntary participation of physicians who may not be familiar with the investigators or their home institutions. Further research is needed to develop feasible and cost-effective validation protocols in such situations or at the very least to establish consensus guidelines about the appropriate ways to assess screening.