Test-retest reliability of health behavior indicators in Community Health Survey in South Korea

Community Health Survey has been conducted every year since 2008. Major indicators from the survey provided evidences for developing community-based health policy. This study was aimed to assess the test-retest reliability of selected indicators in the survey questionnaire among adults in Korea. A sample of 159 men and women aged 20-69 years old participated in a test-retest procedure with a two-week interval. A total of 23 items concerning smoking, alcohol drinking, physical activity, weight control, and mental health were selected. We evaluated items by using the percent agreement, and kappa from X² test. Percent agreement and kappa were ranging from 0.62 to 0.94 and from 0.40 to 0.89, respectively. We compared the mean of percent agreement and kappa value of items with domain (smoking, alcohol drinking, physical activity, weight control, and mental health), item characteristics(habit, recognition, and attitude), and response scale(two-, three-, four-, or five-point scale). Reliability had a different value for domain, item characteristic, and response scale. In domain, items asking smoking habit had a highest consistency (mean of percent agreement; 0.85, and mean of kappa; 0.70); whereas, items asking weight control had a lowest consistency (mean of percent agreement; 0.70, and mean of kappa; 0.52). In the item characteristics, items related to habit had the highest agreement (mean of percent agreement; 0.85, and mean of kappa; 0.70); whereas, items of attitude had the lowest agreement (mean of percent agreement; 0.55, and mean of kappa; 0.42). In response scale, items with two-point scale had the highest agreement (mean of percent agreement; 0.87, and mean of kappa; 0.64); whereas, items with five-point scale had the lowest agreement (mean of percent agreement; 0.69, and mean of kappa; 0.51). In linear regression model, two-point scale showed a better percent agreement, as compared to five-point scale after adjusting for domains and item characteristics(beta=0.152, p=0.002). Habit affected kappa positively, as compared to attitude after adjusting for domains and response scale (beta=0.308, p=0.015). Reliability of selected items in Community Health Survey was associated with item characteristics and response scale. Some items with low reliability might need to be modified regarding their item characteristics and response scale.