The Effects of Survey Mode and Asking About Future Intentions on Self-Reports of Colorectal Cancer Screening

Colorectal cancer (CRC) screening rates are often ascertained via self-reports but can be subject to overreporting bias. Asking about intention to get screened before asking about past screening may minimize overreporting of cancer screening. In a statewide survey conducted from July through October of 2005, we embedded an experiment that tested the effect of question ordering (asking about future intention to get screened before or after asking about past screening; “future first” and “future second,” respectively), crossed with survey mode (mail versus telephone), on CRC screening rates. Weighted analysis focused on 752 respondents who were ages 50 years or older. We found (a) that asking about future intentions to get screened before asking about past screening (future first) statistically significantly lowers reports of past CRC screening [70.9% future second versus 58.0% future first; odds ratio (OR), 1.83; 95% confidence interval (95% CI), 1.08-3.13]; (b) that there was no main effect of survey mode; and (c) that the effect of the ordering of the future intentions item varies by survey mode. In the mailed survey, the odds of reporting past CRC screening were almost thrice greater in the future second condition compared with the future first condition (72.4% versus 49.0%, respectively; OR, 2.74; 95% CI, 1.22-6.17). In the telephone condition, the odds of reporting were only 28% higher in the future second (69.5%) condition than in the future first condition (63.9%; OR, 1.28; 95% CI, 0.64-2.57). The results suggest that asking about future intentions to get screened before the actual behavior elicits lower, and arguably more truthful reports of CRC screening but mainly in mailed surveys. (Cancer Epidemiol Biomarkers Prev 2008;17(4):785–90)

Population-based colorectal cancer (CRC) screening levels are often ascertained via self-report, as this approach represents a cost-effective alternative to more effort-intensive methods such as medical record review (1). However, self-reports of cancer screening—including certain types of CRC screening tests such as fecal occult blood test and sigmoidoscopy—have been found to be prone to overreporting bias (2-4), probably due to the desire of some to cast themselves in a socially desirable light coupled with the fact that cancer screening is a socially desirable health promotion behavior. Recent reviews of the literature on the accuracy of self-reported health care utilization in general (5) and of cancer-screening behavior in particular (3) have implicated questionnaire design features such as item wording and question ordering, as well as mode of data collection (e.g., mail or telephone), as factors affecting report veracity. The focus of our study was to test the unique and interactive effects of these two factors on self-reported CRC screening behavior.

There is ample evidence that survey respondents offer socially desirable responses to questions asking about socially proscribed behaviors (e.g., illicit drug use) more often in a telephone survey context than when asked to complete similar questions in a self-administered, mailed survey (6-9). However, very little information exists on how telephone and mail-survey respondents differ in their reporting of socially desirable behaviors such as cancer screening. The information that exists suggests a certain level of telephone and mail-survey mode equivalence in cancer screening reporting accuracy (10, 11), but the sheer paucity of evidence renders judgment on the issue far from final. Moreover, it is not clear whether the forces driving overreports of socially desirable behaviors are different from those prompting the underreporting of socially undesirable behaviors (12). As such, further investigation of the effect of telephone versus mail data collection modes seems warranted.

Many questionnaire design features and their effects on self-reports of cancer screening could also be investigated. We chose to focus on how screening items are structured as suggested by Bhandari and Wagner (5) and extend the recent work in this area by Johnson and colleagues (12). These latter authors found evidence suggesting that asking about future intentions to get screened for cancer before the actual question about past screening—Pap smears, mammograms, and clinical-gynecologic examinations in this case—increased the accuracy of self-reports of past screening when compared with medical records. The authors posited that by asking respondents if they plan on engaging in a future socially desirable activity before the actual past behavior, they will be under less social pressure to overreport their past practice of that behavior. Because several of their findings did not reach statistical significance in multivariate models, the authors characterized their results as suggestive and call for further methodologic work in this area.

In addition, Vernon and colleagues (3) have observed few methodologic studies of questionnaire design features in the cancer-screening literature. It is in this context that we set out to test the effect of asking about future intention to get screened for CRC (either before or after the past CRC screening behavior question) crossed with survey mode (mail versus telephone). We are aware of no other study that has tested the effect of these two factors simultaneously.

The population for this survey included noninstitutionalized Minnesota residents who were 18 y and older. The sampling frame used a list-appended random digit dial sample purchased from Genesys Marketing Systems Group, whereby postal addresses were appended to the random digit dial telephone numbers if they were found in listed directories. The overall sample was stratified by region (Twin Cities —Minneapolis/St. Paul— metropolitan, non-Twin Cities metropolitan, and rural) and residents of Olmsted County, and those ages ≥50 y were intentionally oversampled through the use of geographic and age information on telephone numbers available from Genesys Marketing Systems Group. All data were weighted so that the sample reflected the sex, age, geographic, and racial/ethnic distribution of adults in the state.

From July 15 through October 25 2005, the University of Minnesota Center for Survey Research in Public Health conducted a mixed-mode mail or telephone survey of 4,210 adults as part of their Omnibus Survey series. The Center for Survey Research in Public Health Omnibus Survey is fielded quarterly and allows research institutions, government agencies, and nonprofit organizations to each ask a few questions as part of a larger survey. The questionnaire contained ∼40 items measuring such health-related issues as general health, health insurance coverage, use of online health information, testing for colon cancer, and oral health. It also measured demographics and attitudes toward tattoos, drug use, and surveys. The current investigation focuses only on the items relating to colon cancer and screening and the data come from an experiment embedded in the larger survey that was conducted for other purposes.

Telephone numbers with an appended address were randomly assigned to receive either a mailed survey or a telephone survey. Participants without appended addresses (i.e., households with unlisted telephone numbers) were assigned to a telephone survey sample. A total of 1,492 and 2,718 participants were assigned to the mail and telephone modes, respectively; ∼70% of those assigned to the telephone mode had an appended address. Once participants were randomly assigned to data collection mode, they were then assigned to one of two question wording conditions within mode where we altered the position of the future intention to get screened for CRC (either before or after the CRC screening item; hereafter called “future first” and “future second,” respectively). The actual wording for the items in each condition is provided in the Appendix.

A multiple-contact data-collection protocol was deployed for both mail and telephone conditions. For those in the mail-survey mode, the initial mailing consisted of a cover letter, survey, and a business reply envelope. A reminder postcard was mailed ∼ 10 d after the initial mailing. A third mailing that included a cover letter, another questionnaire, and another business reply envelope was sent to survey nonrespondents ∼2 wk after the mailing of the postcard reminder. A total of 9 participants in the mail mode were found to be ineligible due to being age <18 y. The response proportion was calculated as the number of completions divided by the number of eligible participants using the response rate calculation formula set forth by the American Association for Public Opinion Research.³

A total of 741 mailed surveys were received, for a response proportion of 50% (741 of 1,483). For participants randomly or necessarily assigned to the telephone sampling frame, the survey instrument used Computer-Assisted Telephone Interview. Calls to attempt an interview were made by trained Center for Survey Research in Public Health interviewers at all times of the day (morning, afternoon, and evening) and days of the week (weekdays and weekends). Telephone numbers were attempted up to 20 times per case and messages left on answering machines to increase participation.

A total of 893 participants were found to be ineligible, mainly due to having nonworking telephone lines, and 895 telephone interviews were completed, for an American Association for Public Opinion Research response rate calculation formula response proportion of 49% (895 of 1,825). The institutional review boards at both the University of Minnesota and Mayo Clinic approved the consent and study procedures.

We focus here on the 759 respondents ages 50 y or more because extant guidelines recommend routine CRC screening in this population subgroup (13); seven participants did not respond to the CRC screening item. A total of 752 participants were available for analysis with 172, 215, 167, and 198 in the mail/future first, telephone/future first, mail/future second, and telephone/future second conditions, respectively.

The analyses were framed by three working hypotheses. First, we hypothesized that lower, and arguably more truthful, reporting of past CRC screening would be observed in the condition where the screening behavior item follows the future intention item (future first). Second, we hypothesized that CRC screening rates would be lower in the mailed version of the questionnaire than in the telephone version. Third, we hypothesized that CRC screening rates would be influenced by an interaction between question ordering and mode of data collection. Specifically, we hypothesized that the effect of asking the future intentions item before past screening would be greatest in the telephone condition where question order is believed to be most salient. There is evidence in the survey research methods literature demonstrating that order effects are more prominent in surveys administered via interview (e.g., in-person and telephone) than in self-administered surveys, largely due to the serial administration of items in the former (14-16).

To compare the respondents in the mail and telephone conditions, we analyzed their sociodemographic characteristics by mode. This was done to assess selection into mode after random assignment—there is some evidence of variation in mode preferences and that different populations may respond to different modes (17-19)—and to identify possible confounders that might have needed to be controlled for in the primary analyses. To assess differences across modes, we used weighted linear regression with mode as the independent variable to compare age (the dependent variable); the Rao-Scott χ² test (weighted) to compare sex, race, educational attainment, employment status, marital status, and health status.

Logistic regression analysis was used to determine whether or not CRC screening rates varied by question ordering and/or mode of data collection (both adjusting and not adjusting for covariates). In the adjusted analyses, we included, not only the demographic variables that were found to vary across mail and telephone conditions but also a variable that identified whether a telephone number was found in a listed directory (hereafter called telephone mode listed status) because only those with appended addresses could be assigned to the mail condition. Doing so was important because white-pages frames exclude those with unlisted telephone numbers. Guterbock and colleagues (20) found that those with unlisted numbers were more likely to be African American and younger in age; a finding confirmed by Lepkowski and colleagues (21). However, Lepkowski et al. (21) also found that listed status was not related to any of their substantive economic measures, and Smith and colleagues (22) found that listed status was not related to estimates of HIV-related risk behavior. Nonetheless, we include this variable in the analysis, although use of a listed frame may not alter the substantive findings of our study.

All P values are two sided and a P value of ≤0.05 was regarded as statistically significant. All reported percentages, means, and analyses are weighted and were done using SAS v. 9.1 software (SAS Institute, Inc.) using SAS survey procedures.

Table 1 provides the sociodemographic characteristics of respondents in each data collection mode. Sex (P = 0.04), education (P = 0.03), and health status (P = 0.01) all varied significantly by data collection mode where mail-survey respondents were less likely to be female, and more likely to be of lower educational status and less healthy than those responding to the telephone survey. As a result, these three variables were entered as covariates—along with telephone mode listed status—into the adjusted analyses referenced below. No statistically significant differences were observed for age, race, employment status, and marital status—these groups were equivalent across data-collection modes.

Table 2 shows the effects of question ordering and mode of data collection on past CRC screening behavior. Overall, in the logistic regression model with the main effects of mode and order, we did not find evidence of a significant main effect of data collection mode on CRC screening, but we did observe support for the hypothesis that CRC screening rates are statistically significantly lower in the future first condition (58.0%) than in the future second condition [70.9%; odds ratio (OR), 1.83; P = 0.03]. To address the hypothesis that CRC screening rates would be influenced by an interaction between question ordering and mode of data collection, we also looked at a logistic regression model that included the interaction between mode and order to investigate the effect of order within mode. Although the interaction term was not statistically significant (P = 0.16), indicating that the effect of order does not differ across modes, we found that the effect of question order was statistically significant in the mail mode but not the telephone mode (Table 2). Specifically, there was a 23 percentage point difference in the screening rates between the future second (72.4%) and future first (49.0%) conditions in the mail mode (OR, 2.74; 95% confidence interval, 1.22-6.17), whereas the difference between the two question ordering conditions fell to five percentage points in the telephone-survey condition (69.5% and 63.9% in the future second and future first conditions, respectively; OR, 1.28; 95% confidence interval, 0.64-2.57). Adjusting the models for sex, education, health status, and telephone mode listed status did not alter the pattern of results described above.

Aside from a few validation studies comparing self-reports to medical records, there is surprisingly little methodologic work in the area of self-reported CRC screening accuracy despite the evidence suggesting that respondents overreport cancer-screening behavior in surveys (2-4) and despite the widespread use of self-reports in major surveys such as the Behavioral Risk Factor Surveillance System, the National Health Interview Study, and the Health Information National Trends Survey. We found support for our hypothesis that asking about future intentions to get screened before asking about past screening statistically significantly lowers reports of past CRC screening. Although we did not validate self-reports against medical records, our findings are consistent with those observed by Johnson and colleagues (12) who found that asking about future intentions to get screened for cancer before the question about actual past screening behavior decreased self-report and that those reports were more accurate compared with medical records. As such, our results offer at least implicit support for the notion that, by asking respondents if they plan on engaging in a future socially desirable activity before the actual past behavior, they will be under less social pressure to overreport their past practice of that behavior.

Contrary to expectations, we found no support for our second hypothesis that CRC screening rates would be lower in the mailed version of the questionnaire than in the telephone version, in either unadjusted analyses or those that adjusted for the possible effects of respondent selection into mode. Our findings are consistent with the scant prior research focusing on self-reports of cancer screening (10, 11) but are inconsistent with what might be expected from the literature on mode effects and socially undesirable behavior such as illicit drug use (6-9). It may be that the forces driving overreports of socially desirable behaviors are different from those prompting the underreporting of socially undesirable behaviors as suggested by Johnson et al. (12).

We had also hypothesized that asking about future intentions before past CRC screening would have its greatest effect in the telephone version where items are asked in a linear and sequential manner. However, we found that our item ordering affected only screening self-reports in the mailed version of the survey where the estimated odds of reporting past CRC screening were close to three times greater when asked about future intentions after the screening item. It may be that respondents do proceed through self-administered forms in a linear manner and/or that question ordering is not as salient in telephone surveys vis-à-vis mail as conjectured by past researchers (14-16).

Our findings in this area might also be attributable to an increased definitional clarity brought about by the manner in which respondents were exposed to the information in the mail-mode condition. Examining the wording of questions in the appendix, it is clear that respondents in the mail mode were afforded the luxury of seeing what might be included in the phrasing “…tested for colon cancer…” asked as part of the question series by looking just beneath or above it to see a listing of different testing methods (e.g., fecal occult test, sigmoidoscopy, etc.). Having access to this information—information that was not at all available to telephone respondents until they had already been asked and then answered this question—could have clarified what was considered as being “tested.” This supposition would be consistent with the results of cognitive interview studies that CRC test descriptions help clarify what is included under the general rubric of CRC screening (23), as well as research by Baier and colleagues (24) demonstrating that self-reported CRC screening can be quite accurate if the items are carefully phrased and accompanied by test descriptions.

Therefore, it may be that asking about future intentions of a behavior will have an effect on reports of past behavior only in situations of definitional clarity about the target behavior (i.e., the respondent knows exactly what he or she is being asked about) rather than due to a specific survey mode. If this is indeed true, the current results suggest that inclusion of a future intention item in surveys that have specific descriptions of the screening tests, such as the measures developed by Vernon and colleagues (23) or the screening items used in the Health Information National Trends Survey,⁴

increase the accuracy of self-reported colon cancer screening behavior. However, this may be the case only in situations where the tests in question are unfamiliar (newly introduced) to the respondent, or when there are multiple tests being considered.

In considering the above findings, it is important to note some potentially important limitations. First, the reader must be mindful that, whereas past investigators (4, 11, 12) focused on consistency between self-reports and medical records as the primary measure of accuracy, we looked only at the former in the current study. The early research literature in the area of social desirability has typically pointed to the finding that people tend to overreport socially desirable behaviors such as exercise and underreport socially undesirable behaviors such as substance use (25-27). In this context, higher reports of socially desirable behaviors are usually assumed to reflect less honest self-disclosure (28). Nonetheless, dependence on self-reports as a primary measure of accuracy represents a limitation of the current investigation.

Future research should attempt to replicate our results and incorporate comparisons of self-reports to some external “gold standard,” as was done by Johnson et al. (12). Second, our study used items that did not include complete descriptions of the screening tests, contrary to the recommendations of some (23, 24). The fact that our experiment was embedded in a larger study conducted for reasons unrelated to the present experiment limited our ability to do so. As mentioned above, the absence of such descriptions in our question asking may have confounded our results somewhat and, thus, limited the inferential value of our findings.

In conclusion, health researchers and policy makers rely on self-reports of screening behavior. This study has shown that the quality of self-reported CRC screening is affected by the structure and order of the screening items and by the interaction of question ordering and mode of data collection, to a certain extent. Specifically, the findings suggest that asking about future intentions to get screened before asking about past CRC screening elicits lower, and arguably more truthful reports of CRC screening but mainly in mailed surveys. The results also underscore the importance of responding to the calls of many in the field to undertake methodologic studies of factors that affect the accuracy of self-reported cancer-screening behavior (3). We encourage others to continue this line of inquiry by incorporating more defensible measures of report accuracy, using agreed-upon measures of CRC screening behavior, and extending the topic to other types of cancer screening and health-related behaviors.

Q5 Are you planning on being tested for colon cancer in the next 12 months?

1. Yes

2. No

3. I just had one

Q6 Have you ever been tested for colon cancer?

1. Yes

2. No Go to page 3 question 8

Q7 Doctors use several different methods to test for colon cancer. Which of the following tests have you had?

• a. A stool blood test also known as a fecal occult blood test

• b. A Sigmoidoscopy

• c. A Colonoscopy

• d. A Barium Enema

• e. A CT Colonoscopy or Virtual Colonoscopy

• f. Any other test (Specify):_________________________

Q5 Have you ever been tested for colon cancer?

1. Yes

2. No Go to question 7

Q6 Doctors use several different methods to test for colon cancer. Which of the following tests have you had?

• a. A stool blood test also known as a fecal occult blood test

• b. A Sigmoidoscopy

• c. A Colonoscopy

• d. A Barium Enema

• e. A CT Colonoscopy or Virtual Colonoscopy

• f. Any other test (Specify):_________________________

Q7 Are you planning on being tested for colon cancer in the next 12 months?

1. Yes

2. No

3. I just had one