Assessment of Lifetime Cumulative Sun Exposure Using a Self-Administered Questionnaire: Reliability of Two Approaches

Exposure to UV radiation from the sun (solar UVR) is the principal environmental risk factor for skin cancers (1). There is also a growing body of evidence that solar UVR exposure may reduce the risk of some diseases, including non–Hodgkin lymphoma (2-4), prostate cancer (5-7), breast cancer (7, 8), colon or colorectal cancer (7, 9), and multiple sclerosis (10).

Estimates of cumulative lifetime UVR are commonly derived from questionnaire-based recall of hours spent outdoors on weekdays or weekend days during different age intervals; i.e., a time-based approach. For example, participants have been asked “on weekdays, how many hours did you generally spend outdoors in sunlight (between 9:00 a.m. and 3:00 p.m.) in the summer when you were (X) to (Y) years old?” (11). Time-based questionnaires have asked participants to estimate a typical number of hours outdoors in all activities in a given age interval without directing them to use any particular estimation method. We hypothesized that participants may gain additional memory cues and estimate time outdoors more reliably if asked about time spent outdoors in specific activities in particular periods of life; i.e., an activity-based approach.

Most studies examining the reliability of responses to questions about time spent outdoors have done so using information from interviewer-administered questionnaires (12-15); no study has assessed the reliability using self-administered questionnaires. It would be informative to evaluate the reliability from self-administered questionnaires because they are the primary source of information in large cohort studies.

One interview-based reliability study has reported that women's responses were less reliable than men's and attributed this difference to women's combined roles in work and family (13). Other such studies, although reporting no difference between males and females, have not reported gender-specific reliability estimates (12, 14, 15). In addition to the fact that women multitask in their frequently combined work and family roles (13), women report different outdoor activity patterns to men (13, 16-18).

In this investigation, we evaluated the test-retest reliability of responses to time-based and activity-based questions on lifetime time spent outdoors in self-administered mailed questionnaires in the same subjects at two different times; and analyzed reliability in men and women separately.

The investigation (Fig. 1) was carried out among volunteers from the U.S. Radiologic Technologists (USRT) Study, a collaborative study between the U.S. National Cancer Institute, University of Minnesota, and the American Registry of Radiologic Technologists. The USRT cohort consists of 146,022 radiologic technologists who are predominantly female (73%), Caucasian (95%), and born before 1950 (58%; mean, 1944; refs. 19, 20). Identification of the cohort was initiated in 1982, and three mail surveys were conducted during 1984 to 1989, 1993 to 1998, and 2003 to 2005.

Details of the selection of volunteers for the assessment of solar UVR exposure have been published previously (21, 22). Briefly, the third USRT mail survey containing time-based sun exposure questions was mailed to 1,800 USRT cohort members residing in the greater Minneapolis metropolitan area (latitude, 44.96°N; longitude, 93.27°W) and surrounding areas of Wisconsin (northern latitudes) or in Raleigh, North Carolina (latitude, 35.82°N; longitude, 78.66°W) and Atlanta, Georgia (latitude, 33.76°N; longitude, 84.42°W; southern latitudes) in the summer of 2004. Participants from two geographic regions were studied because there were potential differences in outdoor behavior patterns by latitude (23), as latitude influences outdoor temperatures, which to some extent, determines outdoor activities at different times of the year. A stratified sample of 300 participants (including 150 individuals in each geographic region) was randomly selected from respondents who returned the questionnaire within 4 weeks. Within each region, the sample was further stratified by age group (40-59, 60+ years) and gender. Our goal was to enroll 120 of the 300 targeted participants, including approximately 15 in each age-gender-region category. A total of 127 participants were initially enrolled and 124 (98%) completed subsequent questionnaires and other data collection. The number of participants were similarly distributed by gender (female, n = 64; male, n = 60), age (40-59 years, n = 60; ages 60+, n = 64), and geographic region (northern latitudes, n = 63; southern latitudes, n = 61).

The Institutional Review Boards at the National Cancer Institute and University of Minnesota approved the protocol. We focused on the assessment of time spent outdoors from 9:00 a.m. to 3:00 p.m. in the summer because a very high proportion of annual ambient UVR irradiance is incident during this period in the summer (24) and is considered most hazardous for UV exposure in the continental United States (25). In addition, individual outdoor activity patterns may vary substantially by season (23). Therefore, in this methods study, we inquired about time spent outdoors in the summer during the defined age intervals (<13, 13-19, 20-39, 40-64, and 65+ years). These age intervals are similar to those used in other studies (12, 14, 26).

We used a test-retest design to answer our research questions; participants were asked to complete each questionnaire type twice at similar intervals (mean, 223 days for the time-based questionnaire; mean, 195 days for the activity-based questionnaire) but at different times (Fig. 1). All questionnaires were sent by mail, and formatted for self-administration. The baseline (test) time-based questionnaire was administered during the summer of 2004, and the baseline activity-based questionnaire in the spring of 2005.

Time-Based Questionnaire. In the time-based questionnaire, participants were asked to report the number of hours they usually spent outdoors, according to the period during the week (weekday or weekend) and age intervals. Participants were given the following response categories for estimating the average number of hours spent outdoors each day between 9:00 a.m. and 3:00 p.m. in the summer: 0, <1, 1 to 2, 3 to 4, and 5 to 6 h/d.

Activity-Based Questionnaire. The origin of the activity-based questionnaire was an activity diary. Participants were asked to complete an activity diary for 7 consecutive days between September 1 and October 5 in 2004. In the diary, participants were asked to list their primary activity type (indoor or outdoor) and specific outdoor activities in every 30-min interval between 9:00 a.m. and 5:00 p.m. (Fig. 1). Using categories for grouping similar types of activities that were derived from two large national surveys (23, 27) and expert judgment, the original 4,697 entries were eventually reduced to seven major categories consisting of driving, gardening, outdoor home maintenance, walking and light exercise, water activities, outdoor recreation, and outdoor leisure. Outdoor recreation included playing golf, soccer, biking, hiking, running, jogging, or moderate to heavy exercise. Outdoor leisure included sitting or relaxing outside, eating or cooking outside, smoking, supervising children, or watching sports games. These seven categories captured >94% of total time spent outdoors among the study participants (17).

For summer outdoor exposures recalled for the age intervals 20 to 39, 40 to 64, and 65+, we used these seven categories, along with time spent outdoors in the participant's job, in our activity-based questionnaire. For recall of summer outdoor exposures during the age intervals 0 to 12 and 13 to 19, the only categories included in the activity-based questionnaire were (a) organized sports, (b) other than organized sports (e.g., sitting or relaxing outside, running or playing outside, selling lemonade), (c) water activities, and (d) time spent outdoors in a job (ages 13-19 only). Participants were asked, for instance, “how many hours in total from Monday through Friday did you generally spend outdoors doing activity (A) between 9:00 a.m. and 3:00 p.m. during the summer when you were (X) to (Y) years old?” The following response categories were provided: 0, <1, 1 to 2, 3 to 4, 5 to 6, 7 to 8, 9 to 10, >10 h in total from Monday through Friday or on the combined weekend days (Saturday and Sunday). The activity-based questionnaire consisted of a total of 62 questions: 31 questions for each of summertime weekday and weekend day exposures including 3, 4, 8, 8, and 8 questions for weekday and for weekend day exposures during age intervals 0 to 12, 13 to 19, 20 to 39, 40 to 64, and 65+, respectively.

The maximum total number of hours participants reported spending outdoors during the weekdays and on weekend days between 9:00 a.m. and 3:00 p.m. in the activity-based questionnaire should not have exceeded 30 h or 12 h, respectively. Participants, however, tended to overestimate the number of hours spent on each activity, and thus the total hours (across activities) often exceeded 30 or 12 h. The proportion of participants who overestimated the number of hours ranged from 0% (ages 0-12, weekdays, males and females) to 57.6% (ages 20-39, weekend days, males). Participants were more likely to overestimate their exposure during weekend days than during weekdays, and more males than females overestimated their time spent outdoors. Although these overestimated responses raised concerns about the validity of the number of hours reported in the activity-based questionnaire, to avoid distorting the reliability of estimated hours spent outdoors, we did not transform these overreported values; participants with overreported values were retained in the analysis.

Cumulative lifetime sun exposure during 9:00 a.m. to 3:00 p.m. in the summer was calculated by summing weekday and weekend day exposures across different age intervals (time-based questionnaire), or across activity categories and age intervals (activity-based questionnaire) using the midpoint values of the time categories. To account for the effect of current age (e.g., more total hours spent outdoors for older than for younger participants), the cumulative measure was then averaged to give the rate of sun exposure (i.e., average number of hours spent outdoors in the summer per year).

Analysis of Reliability. For ordinal variables (e.g., original response categories that were used in both assessment approaches), the test-retest reliability was evaluated by the proportion of participants with perfect agreement and weighted κ, which compares the observed agreement with that expected by chance while giving more weight to more disparate responses (28). If the participants' responses were limited to two categories in both test and retest data, the weighted κ reduces to the simple unweighted κ. Guidelines have been proposed for qualitative interpretations of the κ statistic: 0.81 to 1.0 (almost perfect), 0.61 to 0.80 (substantial), 0.41 to 0.60 (moderate), 0.21 to 0.40 (fair), 0.00 to 0.20 (slight), and <0.00 (poor; ref. 29).

For continuous variables (e.g., average number of hours spent outdoors in the summer per year), we first used a log-transformation to stabilize the variance, and then used a one-way ANOVA to obtain the intraclass correlation coefficient (ICC; ref. 28). The ICC can be interpreted as the correlation between the test and retest responses for the same individual (28). ICCs were evaluated separately according to the exposure time window: childhood and adolescence ages (0-19 years), adult ages (20+ years), and lifetime (0+ years). Guidelines have been proposed for the interpretation of ICC: 0.75 to 1.00 excellent, 0.40 to 0.74 fair to good, and 0 to 0.39 poor (30).

Comparing Reliability of Activity-Based Questionnaire with Time-Based Questionnaire. We compared weighted κ values, as well as ICCs, from the two questionnaire approaches to evaluate the reliability of the two approaches. For weighted κ comparisons, however, because different numbers of questions and time categories were used for the two questionnaire approaches, we had to reduce the numbers of response categories in the activity-based approach to obtain “average” weighted κ values, as weighted κ values are affected by the number of response categories. The reduction of categories was done as follows: for weekdays and for weekend days within each age interval, we first calculated the proportion of participants, separately for males, females, and the total population, within each response category for the time-based questions at baseline (test data); in some cases, the response categories were combined to ensure sufficient sample size in each response category to calculate weighted κ values. We then regrouped the response categories for the baseline data for activity-based questions (using the sum of hours from all activities within the time period during the week, separately for weekdays and weekend days), so that the regrouped activity-based response categories would have a similar distribution to the time-based response categories. The new hour categories for the activity-based questions were then applied to the activity-based retest data, and used to calculate the average weighted κ values for the activity-based questionnaire. For instance, for weekday exposure during ages (X) to (Y), the categories for the time-based questions had a distribution of a%, b%, c%, d%, and e% for 0, <1, 1 to 2, 3 to 4, and 5 to 6 h/d, respectively. For data from the activity-based questionnaire, we summed up the total hours spent outdoors across activities carried out during weekdays for ages (X) to (Y). The total hours in the test data of the activity-based approach were combined into five new categories to provide a distribution of a%, b%, c%, d%, and e%, respectively [e.g., < 5, 5-9.9, 10-14.9, 15-24.9, and 25+ h as the five new categories for all weekdays combined for ages (X) to (Y)]. The same five new categories were used to provide frequency distributions for the activity-based retest data, and an average weighted κ value was calculated based on the new combined categories.

To test the null hypothesis H₀: D = 0, where D is the unknown variable representing the difference between the weighted κ estimates from time-based and activity-based questionnaires, we calculated the value of T using the formula:

where D̂ is the estimate of D based on the data from individuals who answered both the test and the retest questionnaires in both questionnaire approaches, and V̂_JK is the variance of the difference estimated by the jackknife method (31).

To calculate the P value for rejecting the null hypothesis, we then compared T to a t distribution with d degrees of freedom, where d is the number of individuals minus 1. The same statistical method was applied to compare the ICCs between the activity-based and the time-based estimates.

Among the 124 participants who returned the retest questionnaires, the mean age was 60.1 years (range, 48-80) in males and 59.7 years (range, 45-74) in females. The mean time interval between the test and retest was 223 days (range, 181-290) for the time-based questionnaire and 195 days (range, 125-261) for the activity-based questionnaire.

We first compared the estimated number of hours spent outdoors in the summer per year (averaged between the test and retest questionnaires) from the two questionnaire approaches (Table 1). During childhood and adolescence (ages 0-19 years), the reported number of hours spent outdoors was similar between the two questionnaires for all participants combined. During adult ages (20+ years), the estimated total hours from the activity-based approach were higher than those from the time-based approach in both genders.

For all participants, assessment of test-retest reliability by weighted κ for the time-based questionnaire generally showed fair to moderate reliability; weekday exposure between ages 20 and 39, for which reliability was slight, was an exception (Table 2). For women, κ generally indicated slight to moderate reliability with the lowest reliability on weekdays during ages 20 to 39 and 40 to 64. For men, reliability was slight to moderate with the lowest reliability observed on weekdays during ages 20 to 39.

Test-retest agreement for the activity-based questionnaire assessed by weighted κ showed fair to moderate reliability for all participants combined (Table 2). For women, weighted κ values gave slight to moderate reliability, with slight reliability found for exposures in weekdays during childhood and at ages 65+. For men, weighted κ statistics indicated fair to moderate reliability for weekdays and weekend days during all ages.

We compared the weighted κ values for time spent outdoors from activity-based questionnaire with those from the time-based questionnaire (Table 2). Only 3 of the 30 comparisons gave P < 0.05, and of those that did, 2 comparisons were in favor of greater reliability for the activity-based questionnaire (all participants and weekday exposures during ages 20 to 39 and females and weekday exposures during 40 to 64 years of age) and 1 comparison for the time-based questionnaire (all participants and weekend day exposures at 13-19 years of age). There is, thus, no strong evidence in these comparisons for greater reliability of either questionnaire with respect to the categorical estimates of outdoor hours in individual age groups.

Table 3 shows the test-retest reliability for average hours spent outdoors from time-based and activity-based questionnaires by gender and period of life. The ICCs for the two questionnaire types were similar in childhood and adolescence in all subjects and in males and females. They were, however, consistently higher for the activity-based questionnaire in adult years and over the lifetime in all subjects and both genders, with ICCs ranging from 0.64 to 0.72 for the activity-based questionnaire and 0.25 to 0.65 for the time-based questionnaire. These differences were statistically significant for all subject and females but not for males, in whom the ICCs for the time-based questionnaire in adulthood and over the lifetime were substantially higher [0.55; 95% confidence interval (CI), 0.34-0.71; and 0.65; 95% CI, 0.46-0.79, respectively] than in females (0.25; 95% CI 0.00-0.47, and 0.33; 95% CI, 0.09-0.54).

To our knowledge, this is the first study to compare the reliability of time-based and activity-based sun exposure questionnaires, whether self-administered (as in this case) or interviewer-administered. The retest questionnaires were sent 4 to 9 months after the baseline (test) questionnaire. The activity-based questionnaire produced higher estimated time spent outdoors, especially in adult years and during lifetime. We also wanted to learn whether cumulative estimates of time spent outdoors could be improved, especially for women, using an activity-based questionnaire instead of the more usual time-based questionnaire. Although there was little apparent difference between the two questionnaire types in reliability of age group-specific categorical estimates of time spent outdoors on weekdays and weekend days, as assessed by κ statistics (Table 2), the activity-based approach showed significantly higher ICCs for continuous estimates of average hours outdoors in adult years and over the lifetime than the time-based questionnaire (Table 3). This difference was more evident in females than in males and was not evident for childhood and adolescence exposure.

One concern is the large difference between the estimated numbers of hours spent outdoors based on the two questionnaire approaches. A key potential strength for the activity-based questionnaire is the opportunity to trigger the respondent's memory to recall the full spectrum of outdoor activities. This is particularly true for adult years, during which participants perform various types of outdoor activities that may have not been captured using the traditional time-based questionnaire. For instance, one activity category that is likely not captured in the time-based approach is “driving”, which has been shown to account for 39% of total time spent outdoors in our study population during the assessment of short-term sun exposure (17). Despite the potential advantage discussed above, the activity-based approach also allows a greater chance of inaccuracies because of the number of questions asked. It is likely that the activity-based questionnaire approach gave much higher estimates in time spent outdoors than the time-based questionnaire approach due to the combination of these factors.

Other studies of reliability using time-based questionnaires have observed similar or higher reliability than the activity-based questionnaires in this study. These previous studies were all interviewer-administered (12-15), and thus, could presumably have used cues to help elicit responses. In an Australian population of 62 cases with basal cell carcinoma and 102 controls (84 females and 80 males, ages 40–64 years at the first interview), English et al. reported an ICC of 0.77 for lifetime hours spent outdoors using face-to-face interviews 5 years apart. In another study in Australia of 52 cases with multiple sclerosis and 52 controls (69 females and 35 males; mean age, 43.6 for cases and 44.7 for controls) who were interviewed 11 weeks apart, the authors reported weighted κ values ranging from 0.43 to 0.74, according to four age intervals and two seasons and disease status, using face-to-face interviews (15). In a multicenter European study of 116 cases of basal cell and squamous cell carcinomas and 120 controls (106 females and 130 males; mean age, 61.2 for cases and 62.4 for controls), Rosso et al. obtained ICC values of 0.68, 0.79, and 0.56 for weighted outdoor hours in a lifetime for work, vacation, and sports, respectively, using face-to-face interviews 18 to 26 months apart (14).

For the current investigation, the test-retest reliability for the time-based questionnaire was lower in females than in males (Table 3). Some, but not all, studies have also reported gender differences in time spent outdoors and types of outdoor activities (16-18). Compared with males, estimating lifetime cumulative sun exposure in females during adult years has been reported to be particularly difficult, possibly because of females' frequently combined work and family roles (13). Kricker et al., evaluating the reliability of a shortened telephone questionnaire against its longer face-to-face version 9 to 20 months apart in an Australian population of 15 ocular melanoma cases and 45 controls (34 males and 26 females), also reported lower reliability of cumulative sun exposure among females (ICC, 0.54) than in males (ICC, 0.73; ref. 13). The same study also reported that females had a lower reliability of working day sun exposure than males (ICC, 0.48 for females and 0.71 for males), whereas females had higher reliability of non–working day exposure than males (ICC, 0.67 for females and 0.53 for males; ref. 13). As a second component of their study, Kricker et al. compared the correlations between occupational exposure and total exposure (occupational + recreational) in males and females. The correlation was lower in females than in males (R = 0.54 and 0.78, respectively; ref. 13). In contrast, other studies reported no differences in reliability by gender but did not give gender-specific estimates of reliability (12, 14, 15). Lastly, in the USRT cohort, using self-administered, mailed questionnaires, test-retest responses to the time-based questionnaire approach was evaluated in a previous study of 481 radiologic technologists from the United States (ages 44 to 80 years old, including 238 males and 243 females from throughout the United States) who volunteered to complete the same questionnaire about lifetime UVR solar exposures at two times, approximately 9 months apart. In this larger nationwide population, the reliability of responses about lifetime sun exposure was fair to good for both women (ICC, 0.62; 95% CI, 0.53-0.70) and men (ICC, 0.53; 95% CI, 0.42-0.62).⁷

Our study has several strengths, including high participation rates, a wide range of ages, a substantial number of participants in both genders, the inclusion of participants from two very different latitudes, and systematic assessment of sun exposure at specified intervals. For the activity-based approach, we provided a comprehensive list of activities in our questionnaire to potentially improve participants' recall.

The study also has some limitations. Most importantly, when answering the activity-based questionnaire, participants overestimated the total hours spent outdoors when the number of hours spent in a given activity were summed, raising questions about the validity of the activity-based questionnaire. The degree of overestimation (compared with the maximum number of hours of possible response) was lower during weekdays (particularly for females), and higher on weekend days. Part of the overestimation may have been due to study participants recalling their time outdoors from 9 a.m. to 5 p.m., which corresponds more closely to the working day, rather than the 9 a.m. to 3 p.m. time frame asked in the questionnaires. The overreporting on the activity-based questionnaire may have contributed to the higher number of total outdoor hours recorded using the activity-based questionnaire (Table 1).

We cannot completely rule out chance as an alternative explanation for the higher ICC in the activity-based questionnaire because of the multiple comparisons made. Nonetheless, given the body of literature indicating gender differences in outdoor activity patterns, we consider that the activity-based questionnaire approach provides a promising new direction to improve reliability in the questionnaire assessment of sun exposure. This position is supported by the fact that the activity-based questionnaire showed the greatest improvement in women's recall of their outdoor hours, which is consonant with the basis for which it was designed.

In conclusion, a newly developed activity-based questionnaire, which asks detailed contribution from each activity category, produced an increase in overall reliability of sun exposure in adult years and over the lifetime, in comparison with a time-based questionnaire, which was greater in women than men. There is, however, uncertainty about the validity of the activity-based questionnaire in light of many participants estimating daily outdoor hours that were greater than the specified time window during the day. Research is needed to improve the accuracy of the activity-based questionnaire while retaining its high reliability, for instance, by alerting participants when the total hours across activities exceed the maximum.

No potential conflicts of interest were disclosed.

The costs of publication of this article were defrayed in part by the payment of page charges. This article must therefore be hereby marked advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.

The authors gratefully acknowledge the participation of the USRT cohort members; Diane Kampa and Rick Hoffbeck (University of Minnesota), and Jeremy Miller (IMS) for their assistance.