Exposure to UV radiation from the sun (solar UVR) is the principal environmental risk factor for skin cancers (1). There is also a growing body of evidence that solar UVR exposure may reduce the risk of some diseases, including non–Hodgkin lymphoma (2-4), prostate cancer (5-7), breast cancer (7, 8), colon or colorectal cancer (7, 9), and multiple sclerosis (10).

Estimates of cumulative lifetime UVR are commonly derived from questionnaire-based recall of hours spent outdoors on weekdays or weekend days during different age intervals; i.e., a time-based approach. For example, participants have been asked “on weekdays, how many hours did you generally spend outdoors in sunlight (between 9:00 a.m. and 3:00 p.m.) in the summer when you were (X) to (Y) years old?” (11). Time-based questionnaires have asked participants to estimate a typical number of hours outdoors in all activities in a given age interval without directing them to use any particular estimation method. We hypothesized that participants may gain additional memory cues and estimate time outdoors more reliably if asked about time spent outdoors in specific activities in particular periods of life; i.e., an activity-based approach.

Most studies examining the reliability of responses to questions about time spent outdoors have done so using information from interviewer-administered questionnaires (12-15); no study has assessed the reliability using self-administered questionnaires. It would be informative to evaluate the reliability from self-administered questionnaires because they are the primary source of information in large cohort studies.

One interview-based reliability study has reported that women's responses were less reliable than men's and attributed this difference to women's combined roles in work and family (13). Other such studies, although reporting no difference between males and females, have not reported gender-specific reliability estimates (12, 14, 15). In addition to the fact that women multitask in their frequently combined work and family roles (13), women report different outdoor activity patterns to men (13, 16-18).

In this investigation, we evaluated the test-retest reliability of responses to time-based and activity-based questions on lifetime time spent outdoors in self-administered mailed questionnaires in the same subjects at two different times; and analyzed reliability in men and women separately.

### Overview

The investigation (Fig. 1) was carried out among volunteers from the U.S. Radiologic Technologists (USRT) Study, a collaborative study between the U.S. National Cancer Institute, University of Minnesota, and the American Registry of Radiologic Technologists. The USRT cohort consists of 146,022 radiologic technologists who are predominantly female (73%), Caucasian (95%), and born before 1950 (58%; mean, 1944; refs. 19, 20). Identification of the cohort was initiated in 1982, and three mail surveys were conducted during 1984 to 1989, 1993 to 1998, and 2003 to 2005.

Figure 1.

Test-retest design of the solar UVR questionnaire reliability study.

Figure 1.

Test-retest design of the solar UVR questionnaire reliability study.

Close modal

### Study Population

Details of the selection of volunteers for the assessment of solar UVR exposure have been published previously (21, 22). Briefly, the third USRT mail survey containing time-based sun exposure questions was mailed to 1,800 USRT cohort members residing in the greater Minneapolis metropolitan area (latitude, 44.96°N; longitude, 93.27°W) and surrounding areas of Wisconsin (northern latitudes) or in Raleigh, North Carolina (latitude, 35.82°N; longitude, 78.66°W) and Atlanta, Georgia (latitude, 33.76°N; longitude, 84.42°W; southern latitudes) in the summer of 2004. Participants from two geographic regions were studied because there were potential differences in outdoor behavior patterns by latitude (23), as latitude influences outdoor temperatures, which to some extent, determines outdoor activities at different times of the year. A stratified sample of 300 participants (including 150 individuals in each geographic region) was randomly selected from respondents who returned the questionnaire within 4 weeks. Within each region, the sample was further stratified by age group (40-59, 60+ years) and gender. Our goal was to enroll 120 of the 300 targeted participants, including approximately 15 in each age-gender-region category. A total of 127 participants were initially enrolled and 124 (98%) completed subsequent questionnaires and other data collection. The number of participants were similarly distributed by gender (female, n = 64; male, n = 60), age (40-59 years, n = 60; ages 60+, n = 64), and geographic region (northern latitudes, n = 63; southern latitudes, n = 61).

### Data Collection

The Institutional Review Boards at the National Cancer Institute and University of Minnesota approved the protocol. We focused on the assessment of time spent outdoors from 9:00 a.m. to 3:00 p.m. in the summer because a very high proportion of annual ambient UVR irradiance is incident during this period in the summer (24) and is considered most hazardous for UV exposure in the continental United States (25). In addition, individual outdoor activity patterns may vary substantially by season (23). Therefore, in this methods study, we inquired about time spent outdoors in the summer during the defined age intervals (<13, 13-19, 20-39, 40-64, and 65+ years). These age intervals are similar to those used in other studies (12, 14, 26).

We used a test-retest design to answer our research questions; participants were asked to complete each questionnaire type twice at similar intervals (mean, 223 days for the time-based questionnaire; mean, 195 days for the activity-based questionnaire) but at different times (Fig. 1). All questionnaires were sent by mail, and formatted for self-administration. The baseline (test) time-based questionnaire was administered during the summer of 2004, and the baseline activity-based questionnaire in the spring of 2005.

Time-Based Questionnaire. In the time-based questionnaire, participants were asked to report the number of hours they usually spent outdoors, according to the period during the week (weekday or weekend) and age intervals. Participants were given the following response categories for estimating the average number of hours spent outdoors each day between 9:00 a.m. and 3:00 p.m. in the summer: 0, <1, 1 to 2, 3 to 4, and 5 to 6 h/d.

Activity-Based Questionnaire. The origin of the activity-based questionnaire was an activity diary. Participants were asked to complete an activity diary for 7 consecutive days between September 1 and October 5 in 2004. In the diary, participants were asked to list their primary activity type (indoor or outdoor) and specific outdoor activities in every 30-min interval between 9:00 a.m. and 5:00 p.m. (Fig. 1). Using categories for grouping similar types of activities that were derived from two large national surveys (23, 27) and expert judgment, the original 4,697 entries were eventually reduced to seven major categories consisting of driving, gardening, outdoor home maintenance, walking and light exercise, water activities, outdoor recreation, and outdoor leisure. Outdoor recreation included playing golf, soccer, biking, hiking, running, jogging, or moderate to heavy exercise. Outdoor leisure included sitting or relaxing outside, eating or cooking outside, smoking, supervising children, or watching sports games. These seven categories captured >94% of total time spent outdoors among the study participants (17).

For summer outdoor exposures recalled for the age intervals 20 to 39, 40 to 64, and 65+, we used these seven categories, along with time spent outdoors in the participant's job, in our activity-based questionnaire. For recall of summer outdoor exposures during the age intervals 0 to 12 and 13 to 19, the only categories included in the activity-based questionnaire were (a) organized sports, (b) other than organized sports (e.g., sitting or relaxing outside, running or playing outside, selling lemonade), (c) water activities, and (d) time spent outdoors in a job (ages 13-19 only). Participants were asked, for instance, “how many hours in total from Monday through Friday did you generally spend outdoors doing activity (A) between 9:00 a.m. and 3:00 p.m. during the summer when you were (X) to (Y) years old?” The following response categories were provided: 0, <1, 1 to 2, 3 to 4, 5 to 6, 7 to 8, 9 to 10, >10 h in total from Monday through Friday or on the combined weekend days (Saturday and Sunday). The activity-based questionnaire consisted of a total of 62 questions: 31 questions for each of summertime weekday and weekend day exposures including 3, 4, 8, 8, and 8 questions for weekday and for weekend day exposures during age intervals 0 to 12, 13 to 19, 20 to 39, 40 to 64, and 65+, respectively.

The maximum total number of hours participants reported spending outdoors during the weekdays and on weekend days between 9:00 a.m. and 3:00 p.m. in the activity-based questionnaire should not have exceeded 30 h or 12 h, respectively. Participants, however, tended to overestimate the number of hours spent on each activity, and thus the total hours (across activities) often exceeded 30 or 12 h. The proportion of participants who overestimated the number of hours ranged from 0% (ages 0-12, weekdays, males and females) to 57.6% (ages 20-39, weekend days, males). Participants were more likely to overestimate their exposure during weekend days than during weekdays, and more males than females overestimated their time spent outdoors. Although these overestimated responses raised concerns about the validity of the number of hours reported in the activity-based questionnaire, to avoid distorting the reliability of estimated hours spent outdoors, we did not transform these overreported values; participants with overreported values were retained in the analysis.

### Statistical Analysis

Cumulative lifetime sun exposure during 9:00 a.m. to 3:00 p.m. in the summer was calculated by summing weekday and weekend day exposures across different age intervals (time-based questionnaire), or across activity categories and age intervals (activity-based questionnaire) using the midpoint values of the time categories. To account for the effect of current age (e.g., more total hours spent outdoors for older than for younger participants), the cumulative measure was then averaged to give the rate of sun exposure (i.e., average number of hours spent outdoors in the summer per year).

Analysis of Reliability. For ordinal variables (e.g., original response categories that were used in both assessment approaches), the test-retest reliability was evaluated by the proportion of participants with perfect agreement and weighted κ, which compares the observed agreement with that expected by chance while giving more weight to more disparate responses (28). If the participants' responses were limited to two categories in both test and retest data, the weighted κ reduces to the simple unweighted κ. Guidelines have been proposed for qualitative interpretations of the κ statistic: 0.81 to 1.0 (almost perfect), 0.61 to 0.80 (substantial), 0.41 to 0.60 (moderate), 0.21 to 0.40 (fair), 0.00 to 0.20 (slight), and <0.00 (poor; ref. 29).

For continuous variables (e.g., average number of hours spent outdoors in the summer per year), we first used a log-transformation to stabilize the variance, and then used a one-way ANOVA to obtain the intraclass correlation coefficient (ICC; ref. 28). The ICC can be interpreted as the correlation between the test and retest responses for the same individual (28). ICCs were evaluated separately according to the exposure time window: childhood and adolescence ages (0-19 years), adult ages (20+ years), and lifetime (0+ years). Guidelines have been proposed for the interpretation of ICC: 0.75 to 1.00 excellent, 0.40 to 0.74 fair to good, and 0 to 0.39 poor (30).

Comparing Reliability of Activity-Based Questionnaire with Time-Based Questionnaire. We compared weighted κ values, as well as ICCs, from the two questionnaire approaches to evaluate the reliability of the two approaches. For weighted κ comparisons, however, because different numbers of questions and time categories were used for the two questionnaire approaches, we had to reduce the numbers of response categories in the activity-based approach to obtain “average” weighted κ values, as weighted κ values are affected by the number of response categories. The reduction of categories was done as follows: for weekdays and for weekend days within each age interval, we first calculated the proportion of participants, separately for males, females, and the total population, within each response category for the time-based questions at baseline (test data); in some cases, the response categories were combined to ensure sufficient sample size in each response category to calculate weighted κ values. We then regrouped the response categories for the baseline data for activity-based questions (using the sum of hours from all activities within the time period during the week, separately for weekdays and weekend days), so that the regrouped activity-based response categories would have a similar distribution to the time-based response categories. The new hour categories for the activity-based questions were then applied to the activity-based retest data, and used to calculate the average weighted κ values for the activity-based questionnaire. For instance, for weekday exposure during ages (X) to (Y), the categories for the time-based questions had a distribution of a%, b%, c%, d%, and e% for 0, <1, 1 to 2, 3 to 4, and 5 to 6 h/d, respectively. For data from the activity-based questionnaire, we summed up the total hours spent outdoors across activities carried out during weekdays for ages (X) to (Y). The total hours in the test data of the activity-based approach were combined into five new categories to provide a distribution of a%, b%, c%, d%, and e%, respectively [e.g., < 5, 5-9.9, 10-14.9, 15-24.9, and 25+ h as the five new categories for all weekdays combined for ages (X) to (Y)]. The same five new categories were used to provide frequency distributions for the activity-based retest data, and an average weighted κ value was calculated based on the new combined categories.

To test the null hypothesis H0: D = 0, where D is the unknown variable representing the difference between the weighted κ estimates from time-based and activity-based questionnaires, we calculated the value of T using the formula:

$T=\frac{{\hat{D}}{-}D}{\sqrt{{\hat{V}}_{\mathrm{JK}}({\hat{D}}{-}D)}}$

where is the estimate of D based on the data from individuals who answered both the test and the retest questionnaires in both questionnaire approaches, and JK is the variance of the difference estimated by the jackknife method (31).

To calculate the P value for rejecting the null hypothesis, we then compared T to a t distribution with d degrees of freedom, where d is the number of individuals minus 1. The same statistical method was applied to compare the ICCs between the activity-based and the time-based estimates.

### Population Characteristics

Among the 124 participants who returned the retest questionnaires, the mean age was 60.1 years (range, 48-80) in males and 59.7 years (range, 45-74) in females. The mean time interval between the test and retest was 223 days (range, 181-290) for the time-based questionnaire and 195 days (range, 125-261) for the activity-based questionnaire.

### Estimated Hours Spent Outdoors

We first compared the estimated number of hours spent outdoors in the summer per year (averaged between the test and retest questionnaires) from the two questionnaire approaches (Table 1). During childhood and adolescence (ages 0-19 years), the reported number of hours spent outdoors was similar between the two questionnaires for all participants combined. During adult ages (20+ years), the estimated total hours from the activity-based approach were higher than those from the time-based approach in both genders.

Table 1.

Comparison of average number of hours spent outdoors in the summer during 9:00 a.m. and 3:00 p.m. per year among 124 U.S. radiologic technologists from two geographic regions, according to type of questionnaire approach

Time-based questionnaire, mean (SD)*Activity-based questionnaire, mean (SD)*
Total population
Childhood and adolescence (ages 0-19) 332.16 (112.53) 313.80 (148.35)
Adult years (ages 20+) 181.027 (87.70) 382.90 (245.38)
Lifetime (all ages) 229.61 (84.06) 363.63 (196.77)
Female
Childhood and adolescence (ages 0-19) 289.29 (100.17) 272.02 (123.41)
Adult years (ages 20+) 150.99 (64.73) 347.42 (184.46)
Lifetime (all ages) 197.36 (66.12) 323.03 (144.19)
Male
Childhood and adolescence (ages 0-19) 383.12 (105.67) 363.46 (160.90)
Adult years (ages 20+) 215.50 (97.97) 423.64 (297.19)
Lifetime (all ages) 269.60 (87.28) 413.98 (239.08)
Time-based questionnaire, mean (SD)*Activity-based questionnaire, mean (SD)*
Total population
Childhood and adolescence (ages 0-19) 332.16 (112.53) 313.80 (148.35)
Adult years (ages 20+) 181.027 (87.70) 382.90 (245.38)
Lifetime (all ages) 229.61 (84.06) 363.63 (196.77)
Female
Childhood and adolescence (ages 0-19) 289.29 (100.17) 272.02 (123.41)
Adult years (ages 20+) 150.99 (64.73) 347.42 (184.46)
Lifetime (all ages) 197.36 (66.12) 323.03 (144.19)
Male
Childhood and adolescence (ages 0-19) 383.12 (105.67) 363.46 (160.90)
Adult years (ages 20+) 215.50 (97.97) 423.64 (297.19)
Lifetime (all ages) 269.60 (87.28) 413.98 (239.08)
*

Mean and SD calculated based on the average of test and retest questionnaires.

### Test-Retest Reliability of Time Spent Outdoors: Ordinal Response Categories for Time-Based and Activity-Based Questionnaires

For all participants, assessment of test-retest reliability by weighted κ for the time-based questionnaire generally showed fair to moderate reliability; weekday exposure between ages 20 and 39, for which reliability was slight, was an exception (Table 2). For women, κ generally indicated slight to moderate reliability with the lowest reliability on weekdays during ages 20 to 39 and 40 to 64. For men, reliability was slight to moderate with the lowest reliability observed on weekdays during ages 20 to 39.

Table 2.

Agreement statistics for estimated hours spent outdoors in the summer during 9:00 a.m. and 3:00 p.m. among 124 U.S. radiologic technologists from two geographic regions, according to the type of questionnaire

NTime-based questionnaire
Activity-based questionnaire
P*
Agreement (%)κ (95% CI)Agreement (%)κ (95% CI)
Total population
Weekdays
Ages 0-12 121 52.9 0.36 (0.23-0.50) 49.2 0.28 (0.15-0.40) 0.34
Ages 13-19 122 51.2 0.43 (0.31-0.55) 47.2 0.36 (0.25-0.49) 0.48
Ages 20-39 121 37.7 0.15 (0.03-0.28) 47.2 0.40 (0.28-0.51) 0.008
Ages 40-64 122 45.1 0.26 (0.13-0.38) 51.6 0.42 (0.30-0.54) 0.06
Ages 65+ 33 39.4 0.38 (0.16-0.60) 52.9 0.48 (0.25-0.70) 0.65
Weekend days
Ages 0-12 118 59.2 0.38 (0.24-0.53) 54.9 0.33 (0.21-0.45) 0.58
Ages 13-19 119 59.2 0.50 (0.39-0.61) 48.8 0.31 (0.20-0.43) 0.02
Ages 20-39 120 52.5 0.42 (0.30-0.55) 52.0 0.30 (0.18-0.42) 0.12
Ages 40-64 118 47.9 0.38 (0.27-0.49) 43.9 0.40 (0.28-0.52) 0.88
Ages 65+ 33 36.4 0.33 (0.11-0.55) 47.1 0.37 (0.12-0.63) 0.81

Female
Weekdays
Ages 0-12 64 48.4 0.28 (0.10-0.46) 42.2 0.17 (0.00-0.35) 0.17
Ages 13-19 64 48.4 0.35 (0.18-0.53) 40.6 0.21 (0.04-0.37) 0.27
Ages 20-39 64 39.1 0.17 (0.00-0.35) 54.7 0.37 (0.20-0.54) 0.18
Ages 40-64 64 43.8 0.10 (0.00-0.28) 54.7 0.37 (0.20-0.55) 0.04
Ages 65+ 15 66.7 0.40 (0.06-0.74) 60.0 0.10 (0.00-0.61) NA
Weekend days
Ages 0-12 64 57.8 0.37 (0.17-0.56) 53.1 0.38 (0.22-0.54) 0.93
Ages 13-19 63 52.4 0.41 (0.24-0.58) 47.6 0.23 (0.03-0.40) 0.14
Ages 20-39 64 45.3 0.37 (0.21-0.53) 54.7 0.48 (0.31-0.64) 0.32
Ages 40-64 63 42.9 0.30 (0.15-0.45) 57.1 0.45 (0.29-0.61) 0.13
Ages 65+ 15 46.7 0.26 (0.00-0.60) 80.0 0.68 (0.33-1.00) 0.49

Male
Weekdays
Ages 0-12 57 59.6 0.39 (0.18-0.60) 64.9 0.47 (0.28-0.67) 0.59
Ages 13-19 58 51.7 0.38 (0.21-0.57) 53.4 0.36 (0.16-0.57) 0.86
Ages 20-39 57 35.1 0.14 (0.00-0.31) 47.4 0.33 (0.17-0.49) 0.12
Ages 40-64 58 46.6 0.34 (0.18-0.51) 43.1 0.39 (0.22-0.55) 0.75
Ages 65+ 18 33.3 0.39 (0.12-0.66) 27.8 0.23 (0.00-0.59) 0.61
Weekend days
Ages 0-12 54 63.0 0.34 (0.12-0.55) 59.3 0.27 (0.07-0.47) 0.64
Ages 13-19 56 69.6 0.56 (0.39-0.73) 60.7 0.34 (0.13-0.55) 0.08
Ages 20-39 56 62.5 0.49 (0.30-0.67) 53.6 0.37 (0.17-0.57) 0.35
Ages 40-64 55 54.5 0.45 (0.26-0.63) 54.5 0.46 (0.28-0.64) 0.90
Ages 65+ 18 44.4 0.38 (0.07-0.70) 38.9 0.22 (0.00-0.58) 0.58
NTime-based questionnaire
Activity-based questionnaire
P*
Agreement (%)κ (95% CI)Agreement (%)κ (95% CI)
Total population
Weekdays
Ages 0-12 121 52.9 0.36 (0.23-0.50) 49.2 0.28 (0.15-0.40) 0.34
Ages 13-19 122 51.2 0.43 (0.31-0.55) 47.2 0.36 (0.25-0.49) 0.48
Ages 20-39 121 37.7 0.15 (0.03-0.28) 47.2 0.40 (0.28-0.51) 0.008
Ages 40-64 122 45.1 0.26 (0.13-0.38) 51.6 0.42 (0.30-0.54) 0.06
Ages 65+ 33 39.4 0.38 (0.16-0.60) 52.9 0.48 (0.25-0.70) 0.65
Weekend days
Ages 0-12 118 59.2 0.38 (0.24-0.53) 54.9 0.33 (0.21-0.45) 0.58
Ages 13-19 119 59.2 0.50 (0.39-0.61) 48.8 0.31 (0.20-0.43) 0.02
Ages 20-39 120 52.5 0.42 (0.30-0.55) 52.0 0.30 (0.18-0.42) 0.12
Ages 40-64 118 47.9 0.38 (0.27-0.49) 43.9 0.40 (0.28-0.52) 0.88
Ages 65+ 33 36.4 0.33 (0.11-0.55) 47.1 0.37 (0.12-0.63) 0.81

Female
Weekdays
Ages 0-12 64 48.4 0.28 (0.10-0.46) 42.2 0.17 (0.00-0.35) 0.17
Ages 13-19 64 48.4 0.35 (0.18-0.53) 40.6 0.21 (0.04-0.37) 0.27
Ages 20-39 64 39.1 0.17 (0.00-0.35) 54.7 0.37 (0.20-0.54) 0.18
Ages 40-64 64 43.8 0.10 (0.00-0.28) 54.7 0.37 (0.20-0.55) 0.04
Ages 65+ 15 66.7 0.40 (0.06-0.74) 60.0 0.10 (0.00-0.61) NA
Weekend days
Ages 0-12 64 57.8 0.37 (0.17-0.56) 53.1 0.38 (0.22-0.54) 0.93
Ages 13-19 63 52.4 0.41 (0.24-0.58) 47.6 0.23 (0.03-0.40) 0.14
Ages 20-39 64 45.3 0.37 (0.21-0.53) 54.7 0.48 (0.31-0.64) 0.32
Ages 40-64 63 42.9 0.30 (0.15-0.45) 57.1 0.45 (0.29-0.61) 0.13
Ages 65+ 15 46.7 0.26 (0.00-0.60) 80.0 0.68 (0.33-1.00) 0.49

Male
Weekdays
Ages 0-12 57 59.6 0.39 (0.18-0.60) 64.9 0.47 (0.28-0.67) 0.59
Ages 13-19 58 51.7 0.38 (0.21-0.57) 53.4 0.36 (0.16-0.57) 0.86
Ages 20-39 57 35.1 0.14 (0.00-0.31) 47.4 0.33 (0.17-0.49) 0.12
Ages 40-64 58 46.6 0.34 (0.18-0.51) 43.1 0.39 (0.22-0.55) 0.75
Ages 65+ 18 33.3 0.39 (0.12-0.66) 27.8 0.23 (0.00-0.59) 0.61
Weekend days
Ages 0-12 54 63.0 0.34 (0.12-0.55) 59.3 0.27 (0.07-0.47) 0.64
Ages 13-19 56 69.6 0.56 (0.39-0.73) 60.7 0.34 (0.13-0.55) 0.08
Ages 20-39 56 62.5 0.49 (0.30-0.67) 53.6 0.37 (0.17-0.57) 0.35
Ages 40-64 55 54.5 0.45 (0.26-0.63) 54.5 0.46 (0.28-0.64) 0.90
Ages 65+ 18 44.4 0.38 (0.07-0.70) 38.9 0.22 (0.00-0.58) 0.58
*

P value from two-sided t test comparing the difference of two weighted κ values from each questionnaire approach, with the variance of the difference estimates by the jackknife method. P value could not be calculated for the comparison of weekday sun exposure during age 65+ among females because the participants' responses were limited to two categories, whereas estimation of weighted κ values requires at least three response categories.

Analysis restricted to participants who answered both the test and retest questionnaires in both types of questionnaire approaches (time-based and activity-based).

Test-retest agreement for the activity-based questionnaire assessed by weighted κ showed fair to moderate reliability for all participants combined (Table 2). For women, weighted κ values gave slight to moderate reliability, with slight reliability found for exposures in weekdays during childhood and at ages 65+. For men, weighted κ statistics indicated fair to moderate reliability for weekdays and weekend days during all ages.

We compared the weighted κ values for time spent outdoors from activity-based questionnaire with those from the time-based questionnaire (Table 2). Only 3 of the 30 comparisons gave P < 0.05, and of those that did, 2 comparisons were in favor of greater reliability for the activity-based questionnaire (all participants and weekday exposures during ages 20 to 39 and females and weekday exposures during 40 to 64 years of age) and 1 comparison for the time-based questionnaire (all participants and weekend day exposures at 13-19 years of age). There is, thus, no strong evidence in these comparisons for greater reliability of either questionnaire with respect to the categorical estimates of outdoor hours in individual age groups.

### Test-Retest Reliability of Time Spent Outdoors: Continuous Average Hours Spent Outdoors from Time-Based and Activity-Based Questionnaires

Table 3 shows the test-retest reliability for average hours spent outdoors from time-based and activity-based questionnaires by gender and period of life. The ICCs for the two questionnaire types were similar in childhood and adolescence in all subjects and in males and females. They were, however, consistently higher for the activity-based questionnaire in adult years and over the lifetime in all subjects and both genders, with ICCs ranging from 0.64 to 0.72 for the activity-based questionnaire and 0.25 to 0.65 for the time-based questionnaire. These differences were statistically significant for all subject and females but not for males, in whom the ICCs for the time-based questionnaire in adulthood and over the lifetime were substantially higher [0.55; 95% confidence interval (CI), 0.34-0.71; and 0.65; 95% CI, 0.46-0.79, respectively] than in females (0.25; 95% CI 0.00-0.47, and 0.33; 95% CI, 0.09-0.54).

Table 3.

ICC for estimated average hours spent outdoors per year in the summer during 9:00 a.m. and 3:00 p.m. among 124 U.S. radiologic technologists from two geographic regions, according to the type of questionnaire

N*Time-based questionnaire
Activity-based questionnaire
P
ICC (95% CI)ICC (95% CI)
Total population
Childhood and adolescence (ages 0-19) 116 0.56 (0.43-0.68) 0.55 (0.41-0.67) 0.89
Adult years (ages 20+) 116 0.43 (0.27-0.57) 0.69 (0.58-0.77) 0.003
Lifetime (all ages) 112 0.52 (0.37-0.65) 0.69 (0.58-0.78) 0.05
Female
Childhood and adolescence (ages 0-19) 63 0.46 (0.24-0.63) 0.50 (0.29-0.66) 0.73
Adult years (ages 20+) 62 0.25 (0.00-0.47) 0.66 (0.49-0.78) 0.002
Lifetime (all ages) 62 0.33 (0.09-0.54) 0.64 (0.47-0.77) 0.03
Male
Childhood and adolescence (ages 0-19) 53 0.62 (0.42-0.76) 0.56 (0.35-0.72) 0.67
Adult years (ages 20+) 54 0.55 (0.34-0.71) 0.71 (0.55-0.82) 0.12
Lifetime (all ages) 50 0.65 (0.46-0.79) 0.72 (0.56-0.83) 0.52
N*Time-based questionnaire
Activity-based questionnaire
P
ICC (95% CI)ICC (95% CI)
Total population
Childhood and adolescence (ages 0-19) 116 0.56 (0.43-0.68) 0.55 (0.41-0.67) 0.89
Adult years (ages 20+) 116 0.43 (0.27-0.57) 0.69 (0.58-0.77) 0.003
Lifetime (all ages) 112 0.52 (0.37-0.65) 0.69 (0.58-0.78) 0.05
Female
Childhood and adolescence (ages 0-19) 63 0.46 (0.24-0.63) 0.50 (0.29-0.66) 0.73
Adult years (ages 20+) 62 0.25 (0.00-0.47) 0.66 (0.49-0.78) 0.002
Lifetime (all ages) 62 0.33 (0.09-0.54) 0.64 (0.47-0.77) 0.03
Male
Childhood and adolescence (ages 0-19) 53 0.62 (0.42-0.76) 0.56 (0.35-0.72) 0.67
Adult years (ages 20+) 54 0.55 (0.34-0.71) 0.71 (0.55-0.82) 0.12
Lifetime (all ages) 50 0.65 (0.46-0.79) 0.72 (0.56-0.83) 0.52

NOTE: Average number of hours spent outdoors was log-transformed to stabilize the variance.

*

Analysis restricted to participants who answered both the test and retest questionnaires in both types of questionnaire approaches (time-based and activity-based).

P value from two-sided t test comparing jackknife estimates of ICC from different questionnaire approaches.

To our knowledge, this is the first study to compare the reliability of time-based and activity-based sun exposure questionnaires, whether self-administered (as in this case) or interviewer-administered. The retest questionnaires were sent 4 to 9 months after the baseline (test) questionnaire. The activity-based questionnaire produced higher estimated time spent outdoors, especially in adult years and during lifetime. We also wanted to learn whether cumulative estimates of time spent outdoors could be improved, especially for women, using an activity-based questionnaire instead of the more usual time-based questionnaire. Although there was little apparent difference between the two questionnaire types in reliability of age group-specific categorical estimates of time spent outdoors on weekdays and weekend days, as assessed by κ statistics (Table 2), the activity-based approach showed significantly higher ICCs for continuous estimates of average hours outdoors in adult years and over the lifetime than the time-based questionnaire (Table 3). This difference was more evident in females than in males and was not evident for childhood and adolescence exposure.

One concern is the large difference between the estimated numbers of hours spent outdoors based on the two questionnaire approaches. A key potential strength for the activity-based questionnaire is the opportunity to trigger the respondent's memory to recall the full spectrum of outdoor activities. This is particularly true for adult years, during which participants perform various types of outdoor activities that may have not been captured using the traditional time-based questionnaire. For instance, one activity category that is likely not captured in the time-based approach is “driving”, which has been shown to account for 39% of total time spent outdoors in our study population during the assessment of short-term sun exposure (17). Despite the potential advantage discussed above, the activity-based approach also allows a greater chance of inaccuracies because of the number of questions asked. It is likely that the activity-based questionnaire approach gave much higher estimates in time spent outdoors than the time-based questionnaire approach due to the combination of these factors.

Other studies of reliability using time-based questionnaires have observed similar or higher reliability than the activity-based questionnaires in this study. These previous studies were all interviewer-administered (12-15), and thus, could presumably have used cues to help elicit responses. In an Australian population of 62 cases with basal cell carcinoma and 102 controls (84 females and 80 males, ages 40–64 years at the first interview), English et al. reported an ICC of 0.77 for lifetime hours spent outdoors using face-to-face interviews 5 years apart. In another study in Australia of 52 cases with multiple sclerosis and 52 controls (69 females and 35 males; mean age, 43.6 for cases and 44.7 for controls) who were interviewed 11 weeks apart, the authors reported weighted κ values ranging from 0.43 to 0.74, according to four age intervals and two seasons and disease status, using face-to-face interviews (15). In a multicenter European study of 116 cases of basal cell and squamous cell carcinomas and 120 controls (106 females and 130 males; mean age, 61.2 for cases and 62.4 for controls), Rosso et al. obtained ICC values of 0.68, 0.79, and 0.56 for weighted outdoor hours in a lifetime for work, vacation, and sports, respectively, using face-to-face interviews 18 to 26 months apart (14).

For the current investigation, the test-retest reliability for the time-based questionnaire was lower in females than in males (Table 3). Some, but not all, studies have also reported gender differences in time spent outdoors and types of outdoor activities (16-18). Compared with males, estimating lifetime cumulative sun exposure in females during adult years has been reported to be particularly difficult, possibly because of females' frequently combined work and family roles (13). Kricker et al., evaluating the reliability of a shortened telephone questionnaire against its longer face-to-face version 9 to 20 months apart in an Australian population of 15 ocular melanoma cases and 45 controls (34 males and 26 females), also reported lower reliability of cumulative sun exposure among females (ICC, 0.54) than in males (ICC, 0.73; ref. 13). The same study also reported that females had a lower reliability of working day sun exposure than males (ICC, 0.48 for females and 0.71 for males), whereas females had higher reliability of non–working day exposure than males (ICC, 0.67 for females and 0.53 for males; ref. 13). As a second component of their study, Kricker et al. compared the correlations between occupational exposure and total exposure (occupational + recreational) in males and females. The correlation was lower in females than in males (R = 0.54 and 0.78, respectively; ref. 13). In contrast, other studies reported no differences in reliability by gender but did not give gender-specific estimates of reliability (12, 14, 15). Lastly, in the USRT cohort, using self-administered, mailed questionnaires, test-retest responses to the time-based questionnaire approach was evaluated in a previous study of 481 radiologic technologists from the United States (ages 44 to 80 years old, including 238 males and 243 females from throughout the United States) who volunteered to complete the same questionnaire about lifetime UVR solar exposures at two times, approximately 9 months apart. In this larger nationwide population, the reliability of responses about lifetime sun exposure was fair to good for both women (ICC, 0.62; 95% CI, 0.53-0.70) and men (ICC, 0.53; 95% CI, 0.42-0.62).7

7

Chu-Ling Yu, personal communication.

These mixed findings suggest that the potential gender difference in test-retest reliability is still an evolving area. Additional studies are needed to examine the potential gender differences in the reliability of cumulative sun exposure, by properly accounting for various types of activities.

Our study has several strengths, including high participation rates, a wide range of ages, a substantial number of participants in both genders, the inclusion of participants from two very different latitudes, and systematic assessment of sun exposure at specified intervals. For the activity-based approach, we provided a comprehensive list of activities in our questionnaire to potentially improve participants' recall.

The study also has some limitations. Most importantly, when answering the activity-based questionnaire, participants overestimated the total hours spent outdoors when the number of hours spent in a given activity were summed, raising questions about the validity of the activity-based questionnaire. The degree of overestimation (compared with the maximum number of hours of possible response) was lower during weekdays (particularly for females), and higher on weekend days. Part of the overestimation may have been due to study participants recalling their time outdoors from 9 a.m. to 5 p.m., which corresponds more closely to the working day, rather than the 9 a.m. to 3 p.m. time frame asked in the questionnaires. The overreporting on the activity-based questionnaire may have contributed to the higher number of total outdoor hours recorded using the activity-based questionnaire (Table 1).

We cannot completely rule out chance as an alternative explanation for the higher ICC in the activity-based questionnaire because of the multiple comparisons made. Nonetheless, given the body of literature indicating gender differences in outdoor activity patterns, we consider that the activity-based questionnaire approach provides a promising new direction to improve reliability in the questionnaire assessment of sun exposure. This position is supported by the fact that the activity-based questionnaire showed the greatest improvement in women's recall of their outdoor hours, which is consonant with the basis for which it was designed.

In conclusion, a newly developed activity-based questionnaire, which asks detailed contribution from each activity category, produced an increase in overall reliability of sun exposure in adult years and over the lifetime, in comparison with a time-based questionnaire, which was greater in women than men. There is, however, uncertainty about the validity of the activity-based questionnaire in light of many participants estimating daily outdoor hours that were greater than the specified time window during the day. Research is needed to improve the accuracy of the activity-based questionnaire while retaining its high reliability, for instance, by alerting participants when the total hours across activities exceed the maximum.

No potential conflicts of interest were disclosed.

Grant support: Intramural Research Program of the NIH and the National Cancer Institute.

The costs of publication of this article were defrayed in part by the payment of page charges. This article must therefore be hereby marked advertisement in accordance with 18 U.S.C. Section 1734 solely to indicate this fact.

The authors gratefully acknowledge the participation of the USRT cohort members; Diane Kampa and Rick Hoffbeck (University of Minnesota), and Jeremy Miller (IMS) for their assistance.

1
Armstrong BK, Kricker A. The epidemiology of UV induced skin cancer.
J Photochem Photobiol B
2001
;
63
:
8
–18.
2
Armstrong BK, Kricker A. Sun exposure and non-Hodgkin lymphoma.
Cancer Epidemiol Biomarkers Prev
2007
;
16
:
396
–400.
3
Kricker A, Armstrong BK, Hughes AM, et al. Personal sun exposure and risk of non Hodgkin lymphoma: a pooled analysis from the Interlymph Consortium.
Int J Cancer
2008
;
122
:
144
–54.
4
Hartge P, Lim U, Freedman DM, et al. Ultraviolet radiation, dietary vitamin D, and risk of non-Hodgkin lymphoma (United States).
Cancer Causes Control
2006
;
17
:
1045
–52.
5
John EM, Koo J, Schwartz GG. Sun exposure and prostate cancer risk: evidence for a protective effect of early-life exposure.
Cancer Epidemiol Biomarkers Prev
2007
;
16
:
1283
–6.
6
Luscombe CJ, Fryer AA, French ME, et al. Exposure to ultraviolet radiation: association with susceptibility and age at presentation with prostate cancer.
Lancet
2001
;
358
:
641
–2.
7
Freedman DM, Dosemeci M, McGlynn K. Sunlight and mortality from breast, ovarian, colon, prostate, and non-melanoma skin cancer: a composite death certificate based case-control study.
Occup Environ Med
2002
;
59
:
257
–62.
8
John EM, Schwartz GG, Dreon DM, Koo J. Vitamin D and breast cancer risk: the NHANES I Epidemiologic Follow-up Study, 1971-1975 to 1992. National Health and Nutrition Examination Survey.
Cancer Epidemiol Biomarkers Prev
1999
;
8
:
399
–406.
9
Freedman DM, Looker AC, Chang SC, Graubard BI. Prospective study of serum vitamin D and cancer mortality in the United States.
J Natl Cancer Inst
2007
;
99
:
1594
–602.
10
van der Mei IA, Ponsonby AL, Dwyer T, et al. Past exposure to sun, skin phenotype, and risk of multiple sclerosis: case-control study.
BMJ
2003
;
327
:
316
.
11
Kricker A, Armstrong BK, English DR, Heenan PJ. A dose-response curve for sun exposure and basal cell carcinoma.
Int J Cancer
1995
;
60
:
482
–8.
12
English DR, Armstrong BK, Kricker A. Reproducibility of reported measurements of sun exposure in a case-control study.
Cancer Epidemiol Biomarkers Prev
1998
;
7
:
857
–63.
13
Kricker A, Vajdic CM, Armstrong BK. Reliability and validity of a telephone questionnaire for estimating lifetime personal sun exposure in epidemiologic studies.
Cancer Epidemiol Biomarkers Prev
2005
;
14
:
2427
–32.
14
Rosso S, Minarro R, Schraub S, et al. Reproducibility of skin characteristic measurements and reported sun exposure history.
Int J Epidemiol
2002
;
31
:
439
–46.
15
van der Mei IA, Blizzard L, Ponsonby AL, Dwyer T. Validity and reliability of adult recall of past sun exposure in a case-control study of multiple sclerosis.
Cancer Epidemiol Biomarkers Prev
2006
;
15
:
1538
–44.
16
Hall HI, May DS, Lew RA, Koh HK, Nadel M. Sun protection behaviors of the U.S. white population.
Prev Med
1997
;
26
:
401
–7.
17
Kwok RK, Linet MS, Chodick G, et al. Simplified categorization of outdoor activities for male and female U.S. indoor workers—a feasibility study to improve assessment of ultraviolet radiation exposures in epidemiologic study questionnaires. Photochem Photobiol 2008 Jul 17. [Epub ahead of print].
18
McCarthy EM, Ethridge KP, Wagner RF, Jr. Beach holiday sunburn: the sunscreen paradox and gender differences.
Cutis
1999
;
64
:
37
–42.
19
Boice JD, Jr., Mandel JS, Doody MM, Yoder RC, McGowan R. A health survey of radiologic technologists.
Cancer
1992
;
69
:
586
–98.
20
Sigurdson AJ, Doody MM, Rao RS, et al. Cancer incidence in the US radiologic technologists health study, 1983-1998.
Cancer
2003
;
97
:
3080
–9.
21
Chodick G, Freedman MD, Kwok RK, et al. Agreement between contemporaneously recorded and subsequently recalled time spent outdoors: implications for environmental exposure studies.
Ann Epidemiol
2007
;
17
:
106
–11.
22
Chodick G, Kleinerman RA, Linet MS, et al. Agreement between diary records of time spent outdoors and personal ultraviolet radiation dose measurements.
Photochem Photobiol
2008
;
84
:
713
–8.
23
Klepeis NE, Nelson WC, Ott WR, et al. The National Human Activity Pattern Survey (NHAPS): a resource for assessing exposure to environmental pollutants.
J Expo Anal Environ Epidemiol
2001
;
11
:
231
–52.
24
Diffey BL. Solar ultraviolet radiation effects on biological systems.
Phys Med Biol
1991
;
36
:
299
–328.
25
Skin Cancer: questions and answers. Centers for Disease Control and Prevention [updated 2006 Dec 21]. Available from: http://www.cdc.gov/cancer/skin/chooseyourcover/qanda.htm.
26
Karagas MR, Zens MS, Nelson HH, et al. Measures of cumulative exposure from a standardized sun exposure history questionnaire: a comparison with histologic assessment of solar skin damage.
Am J Epidemiol
2007
;
165
:
719
–26.
27
Shelley K. Developing the American Time Use Survey activity classification system.
Mon Labor Rev
2005
;
128
:
3
–15.
28
Armstrong B, White E, Saracci R. Principles of exposure measurement in epidemiology. Oxford: Oxford University Press; 1992.
29
Landis JR, Koch GG. The measurement of observer agreement for categorical data.
Biometrics
1977
;
33
:
159
–74.
30
Fleiss J. Design and analysis of clinical experiments. New York: John Wiley & Sons; 1986.
31
Wolter KM. Introduction to variance estimation. New York: Springer-Verlag; 1984.