Factorial Validity and Invariance of a Survey Measuring Psychosocial Correlates of Colorectal Cancer Screening in Ontario, Canada—A Replication Study

Fecal occult blood testing (FOBT) and flexible sigmoidoscopy reduce colorectal cancer (CRC) morbidity and mortality in proportion to use (1-3). Despite this evidence and screening guidelines established in the United States since 1997 and in Canada since 2001, CRC screening (CRCS) prevalence is lower than desirable (4-6). Based on current U.S. estimates, 46.5% of males and 43.1% of females reported CRC testing with one of the three recommended tests (FOBT, flexible sigmoidoscopy, or colonoscopy) within recommended time intervals (7). CRCS prevalence is also modest in Ontario, Canada where recent estimates indicate that FOBT screening uptake was ∼9% (excluding individuals with a history of CRC, colonic polyps, or inflammatory bowel disease; ref. 8). Although this may be a status quo underestimation, it is considerably lower than current U.S. population estimates, indicating that 16.1% of males and 15.3% of females reported a home-based FOBT in the past year (7). Efforts are now under way to rapidly increase Canadian CRCS rates through population-based screening recruitment in multiple provinces. Survey data about the attitudes and norms of the Canadian population regarding screening are useful for designing the required recruitment interventions and examining whether associations between CRCS and psychosocial variables replicate in U.S. and Canadian populations. However, correlate patterns can only be compared if the factor structure of the survey items used remains invariant across groups.

Thus, empirical evidence regarding the factor structure and invariance of the survey items used is critical for establishing psychometric validity. If the factor structure is not equivalent across the groups assessed, findings regarding group differences will be biased due to measurement error. In addition, behavioral interventions based on survey data with poor construct validity may be irrelevant and ineffectual when used with target populations. To date, only three studies have examined the reliability and validity of scales designed to measure psychosocial factors associated with CRCS (9-11). Two of these studies had limited generalizability due to homogenous sample characteristics (9, 10). The third had a more diverse sample and replicated the factor structure established by Vernon and colleagues, showing invariance across African-American and Caucasian groups. Because the sample only consisted of U.S. residents, it remains unclear whether these scales, assessing psychosocial correlates of CRCS, can be validly used with non-U.S. populations who receive care in different health care contexts. According to best practices, the validity of instruments should be investigated for every new population (12).

In this study, we add to the validity literature by reexamining the factorial invariance of the 16 survey items developed by Vernon and colleagues, by assessing Canadian attitudes and norms toward CRCS. Whereas previous studies examined these survey items in a U.S. population of male automotive workers and in a primary care clinic population of African-Americans and Caucasians, the current study examines the factor structure in a randomly selected population of CRCS-eligible Ontario, Canada residents (N = 1,013; 50 years and above). We focused on replicating previous survey validation work (9, 11) in two ways: (a) by determining if the previously identified five-factor model for the 16 CRCS items fits data from a different national population and (b) by examining whether the five-factor model was equivalent across sex in this different population.

The survey investigated is based on five psychosocial constructs: salience-coherence, perceived susceptibility, response efficacy, cancer worries, and social influence. These constructs were, in turn, drawn from the Preventive Health Model, a self-regulation model shown to predict intention and behavior for CRCS (13, 14).

A standardized scripted protocol was administered to Ontario residents, aged 50 y and above, via random digit dialing based on sampling methodology of the Canada Survey Sample, a selection engine that generates random samples of residential telephone numbers (15). The Canada Survey Sample maintains a comprehensive list of all populated exchanges across Canada (and Ontario) and is updated regularly at 3-mo intervals. Detail on the random digit dialing and Canada Survey Sample methodology is presented elsewhere (15, 16). Altogether, 1,558 individuals, 50 y and above, were phone contacted to obtain a sample of 1,013 for a response rate of 69%, following a contact rate of 31%. The survey originated from Cancer Care Ontario, a planning and research organization that advises the provincial government on cancer care and informs health care providers. Ethical approval of the study was obtained from the Sunnybrook Health Sciences Centre and the University of Toronto.

Details about the 16 items measuring five constructs (salience and coherence, perceived susceptibility, response efficacy, cancer worries, and social influence) are described by Tiro and colleagues (11). Item wording, response options, and frequency distributions are available as Supplementary Data.

χ² tests assessed potential male-female differences for marital status and education. Cronbach's α was computed for the five scales. With respect to the following procedures, we replicated the analytic processes used by Tiro et al. (11). Single-group confirmatory factor analyses (CFA) assessed data fit to the proposed five-factor model for males and females separately. Several absolute and incremental fit indices were used to assess suitability of the proposed model. The χ² test statistic indicates absolute model fit by comparing the observed covariance matrix with a covariance matrix from a perfectly fit model. The χ² test is sensitive to sample size and requires normally distributed data for validity.

The 16 individual survey items were nonnormally distributed; therefore, in addition to the traditional χ² statistic, we included the Satorra-Bentler χ² test statistic because it includes a scaling correction for nonnormal sampling distributions (17). The ratio of the χ² test statistic to the degrees of freedom is suggested as an alternative to the χ² test, with values less than or equal to two indicating acceptable fit (18). The root mean square error of approximation (RMSEA), another measure of absolute model fit, measured the closeness of fit between an a priori model and the observed data. RMSEA values of <0.05 indicate good fit (19, 20) and 90% confidence intervals for the RMSEA were also calculated. The comparative fit index (CFI), an incremental fit measure, was also used to compare a target model with a more restricted baseline model, adjusted for sample size (21). CFI values of >0.90 indicate acceptable fit according to Kline (22), whereas Hu and Bentler suggest that values of ≥0.95 indicate good fit (19).

After finding acceptable fit of the five-factor model, two-group CFA was done to determine if the model was similar for men and women. For this objective, two models were created. The first model was unconstrained whereby all factor loadings, error terms, and variances were allowed to vary between male and female samples. The second model was constrained whereby all factor loadings were equal between the two groups. We calculated the χ² difference between the unconstrained and constrained models to evaluate equality between the covariance structures. The difference test statistic follows a χ² distribution and a nonsignificant test statistic indicates invariance across models (i.e., the survey items measure similar latent constructs across groups).

More men than women reported being married or living as married and having completed a 4-year university education (Table 1). Cronbach's α coefficients were largely similar between men and women for all of the constructs except for social influence (Table 2).

CFI fit indices indicate adequate fit for both men and women (0.931 and 0.930, respectively). RMSEA values and the associated 90% confidence intervals support good fit (Table 3). Examination of the LaGrange multiplier tests suggests including two error covariance terms ([cc_drthk: I want to do what my doctor or health professional thinks I should do about colorectal cancer screening and cc_famthk: Members of my immediate family think I should have colorectal cancer screening]; [cc_drdo: My doctor or health professional thinks I should have CRC screening and cc_famdo: I want to do what members of my immediate family think I should do about colorectal cancer screening]; refs. 23, 24). After including these two error covariances, the model fit improves in both sex subgroups with all CFI values >0.95 and RMSEA values <0.05. Adding shared error variance between survey items measuring the same construct (in this case social influence) is a common finding among scales measuring attitudes and beliefs (21, 24).

The multigroup CFA comparing fit between the constrained and unconstrained models found a significant test statistic (mean-adjusted χ²difference = 30.742; P = 0.001). Modification indices recommend releasing constraints for the social influence item (cc_drdo: My doctor or health professional thinks I should have CRC screening) and the salience-coherence item (cc_import: Having colorectal cancer screening is an important thing for me to do), although making these changes did not result in a nonsignificant test statistic (see Table 3). However, the difference in RMSEA values is ≤0.001 when comparing the constrained and unconstrained models. Further, comparison of the factor loadings across sex found a similar pattern of direction and magnitude. Examination of the item frequencies across males and females found significant differences in the response distributions with females more likely using the “strongly disagree” and “strongly agree” options of the Likert scale.

Our findings from single-group CFA support the application of a five-factor model for the 16 items measuring psychosocial correlates of CRCS to a randomly sampled population of Canadian men and women. Our data provide evidence for partial factorial invariance across men and women with a lack of equivalence shown for the social influence items. Collectively, these findings suggest that salience and coherence, cancer worries, perceived susceptibility, and response efficacy are interpreted similarly by Canadian men and women. Thus, our data suggest that these items can be used to evaluate and compare psychosocial correlates across U.S. and Canadian samples.

Although the factor loadings for the social influence items were statistically different between men and women, the difference may not be clinically meaningful. According to Gregorich (25), our results may have two interpretations, notably (a) the social influence items are being interpreted differently by sex (i.e., not equivalent factor loadings) and (b) the factor loading estimates are biased due to extreme response style. The latter is suggested if one group favors using the extremes of the Likert scale (e.g., strongly agree/strongly disagree versus agree/don't know/disagree). The significant differences found in the empirical distribution of the CRCS items by sex, the small difference in RMSEA values across models, and the similar pattern but slightly different magnitude of factor loadings by sex support the latter hypothesis. Future research could use qualitative methods such as cognitive interviewing to uncover the reasons for differential response styles among men and women and determine whether the social influence items have different meanings by sex. Such a study is now being undertaken by several authors of this article. There is also a need for quantitative research to directly test the hypothesis that women show more extreme responses than men and, if so, to what degree in which subpopulations. Given the long history of breast and cervical cancer screening, it is possible that females exhibit a more extreme response style than males as, over time, they have processed more communication messages about cancer screening and prevention and thus may be more inclined to state strong agreement or disagreement. This process of generalization, from breast and cervical cancer screening to CRCS, may be especially true in populations where current levels of CRCS awareness are relatively modest.

In terms of reliability, four scales (salience and coherence, cancer worries, perceived susceptibility, and response efficacy) had internal consistency estimates above or nearly equal to the typical 0.70 acceptability criteria (26). As Cronbach's α is reactive to the number of items used per subscale, reliability could be increased if homogenous items were added, particularly because two subscales contain only two items (cancer worries and response efficacy; ref. 24). On the other hand, a countervailing value favors survey brevity as fewer items are typically associated with higher response rates in sampled populations. Regarding the social influence scale, which showed a Cronbach's α of 0.548, one possible reason for the lower value is that the scale represents a composite reflection of two subconstructs: motivations to comply with key referents and key referents observed as supportive or unsupportive of CRC screening. If efforts were directed at elevating the Cronbach's α of this scale, one strategy might be adding homogenous items to each subconstruct. From an intervention perspective, this would only be valuable if researchers could identify additional referents who influence screening behavior. Regardless of adding items, Cronbach's α for a social influence scale will mathematically be lower than other scales of comparable length because α is based on item-to-scale correlations and any sampled population will include normatively and nonnormatively driven individuals (e.g., people who respond versus do not respond to external pressure regarding performance of health behaviors and key referents who are versus who are not supportive of CRCS).

In conclusion, there are reasons, other than national boundaries, to hypothesize that a random Canadian CRC screening eligible sample might differ in attitudes toward CRC screening from U.S. samples. There are differences in the health care system, likely differences in individual knowledge and awareness, and in communication messages, emphasizing the importance of cancer screening and early detection. Thus, it cannot be taken for granted that items assessing attitudes toward CRCS, with evidential support of factorial validity and invariance across race and sex in a U.S. population, would show a similarly invariant factor structure in a Canadian population. Our findings are thus worth noting as they lay the foundation for using this survey with multinational populations to determine if factors influencing CRCS are similar for individuals living in different health care contexts.

No potential conflicts of interest were disclosed.

The costs of publication of this article were defrayed in part by the payment of page charges. This article must therefore be hereby marked advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.