Lessons learned from multisite implementation and evaluation of Project SHARE, a teen health information literacy, empowerment, and leadership program

Background This case study describes the implementation and evaluation of a multisite teen health information outreach program. The objectives of the program were to increase health knowledge, health information literacy, interest in health careers, community engagement, and leadership skills of teens in disadvantaged communities. Case Presentation Teens at six sites across the country participated in a multi-week curriculum that focused on various aspects of health literacy, information literacy, and leadership. Lesson topics addressed personal health, social determinants of health, information quality, and communication and advocacy skills. Program evaluation included both quantitative and qualitative components and focused on multiple knowledge and skills outcome variables. Results suggested that while teens at all sites showed improvement, particularly with respect to engagement and interest in the topics, the degree of gains in knowledge and information literacy measures varied significantly from site to site. Conclusion On-site implementation planning, cohesive integration of added activities, and emphasis on retention can contribute to implementation and evaluation effectiveness. This work also underscores the limitation of a purely quantitative approach to capturing the impact of health information and stresses the importance of supplementing numerical scores and statistics with qualitative data.


Details of statistical analysis and results
To reduce the number of comparisons while assessing significance of variables across sites, continuous outcome variables from each cluster were included in repeated measure multivariate analysis of variance (MANOVA) accounting for participants being nested within sites. If MANOVA showed significant effect of time (pretest to posttest) or of time by site interaction for a cluster, the analysis identified variables that drove the difference. In addition, we also conducted paired sample t-tests for within-site comparisons. As the results specify which variables reached overall significant time or interaction effects, Bonferroni correction for multiple comparisons was not applied. Categorical variables were analyzed via McNemar tests. Due to data attrition, the analyses are underpowered and, likely, overly conservative.

A. Cluster 1, Knowledge: Knowledge of health disparities and social determinants of health
In this three-variable cluster-Number of factors recognized as social determinants of health, Average proportion of possible explanations per recognized health determinant, and Proportion of possible reasons explaining a local disparity-repeated measure MANOVA indicated significant effect of site, Wilks' Λ=0.40, F(18,124.94)*=2.63, p=0.01. Follow-up univariate tests showed that while Average proportion of possible explanations per recognized health determinant differed across sites, F(6,46)=8.44, p<0.01, there were no differences for the other outcome variables.
There was no significant overall pretest-posttest increase across measures, Wilks' Λ=0.95, F(3,44)=0.85, ns. However, there was a significant interaction between site and time of measurement, Wilks' Λ=0.38, F(18, 124.94)=2.82, p<0.01, with univariate tests indicating that the magnitude changes in Average proportion of possible explanations per recognized health determinant differed by site and contributed to the overall effect, F(6,47)=5.75, p<0.01, while there were no significant site by pretestposttest comparison interactions for the other 2 variables. Paired sample t-test comparison of pretestposttest group means was performed post-hoc to examine the results site-by-site (Table 1).   T-test analyses suggested that, in both cases, the change was significant for the E CT, but not the MT group. With regard to the factors one can control, E CT group's mean knowledge score increased from 0.75(STD=0.79) on the pretest to 1.25(STD=0.91) on the posttest (p<0.008). The corresponding knowledge scores for factors one cannot control changed from 0.30(STD=0.57) on the pretest to 0.85(STD=0.67) on the posttest (p<0.001).

D. Cluster 4, Knowledge: Knowledge of preventive health
A repeated measure MANOVA (MT and E CT participants only) with 2 outcome variables, Awareness of diseases that are public health concerns in the United States (Count) and Average number of known preventive health measures per disease, indicated no overall significant differences across sites, Wilks' Λ=0.60, F(4,34)=2.50, ns. There was significant pretest-posttest increase across measures, Wilks' Λ=0.48, F(2,17)=9.35, p<0.01. Univariate tests indicated that pretest-posttest increase was significant for the Awareness of disease (Count), F(1, 18)=6.88, p=0.02, but not Average number of known preventive health measures per disease.
There was also a significant interaction between site and time of measurement, Wilks' Λ=0.36, F(4, 34)=5.72, p<0.01. Univariate tests indicated that the magnitude changes in both the Awareness of disease (Count), (F(2,18)=5.00, p=0.02), and Average number of known preventive health measures per disease, (F(2,18)=3.46, p=0.05), differed by site and contributed to the overall effect. T-test analysis suggested that the effect of the Awareness of disease (Count) was significant for the MT group (pretestposttest mean increase from 2.75(STD=1.04) to 3.75(STD=0.46), p<0.033), but not the E CT group.
As a single multiple-choice question variable, Preventive measures recognition, could not be included in the MANOVA analysis and was analyzed separately via McNemar test. The proportion of students correctly answering this question differed significantly between pretest and posttest across groups. Examination of this effect site-by-site showed that the contrast was significant for the E CT group, in which the number of correct responses changes from 10% on the pretest to 65% on the posttest (p<0.0001).

E. Cluster 5, Knowledge: Knowledge of nutrition
Nine Knowledge of nutrition multiple-choice questions were summed to constitute this outcome variable (MT and E CT participants only). There was no overall significant pretest-posttest increase, F(1,25)=0.04, ns. However, there was a significant interaction between site and time of measurement, F(2, 25)=7.59, p<003. Paired t-tests revealed that the effects were significant for the E CT group, where the mean score increased from 7.05(STD=1.19) on the pretest to 8.80(STD=0.41) on the posttest (p<0.0001).

F. Cluster 6, Health information literacy: Information evaluation skills
A repeated measure MANOVA with the 3 outcome variables in this cluster-Recognition of information quality markers of a hoax site, Recognition of information quality markers of an authoritative site, and Knowledge of general online information quality criteria-indicated overall significant differences across sites, Wilks' Λ=0.29, F(18,124.94)=3.76, p<0.01. Follow-up univariate tests showed that difference held true for all 3 variables: Recognition of information quality markers of a hoax site, F(6,46)=3.92, p<0.01; Recognition of information quality markers of an authoritative site, F(6,46)=2.55, p=0.03; and Knowledge of general online information quality criteria, F(6,46)=5.97, p<0.01.
There was significant pretest-posttest increase across measures, Wilks' Λ=0.76, F(3,44)=4.62, p<0.01. Univariate tests indicated that pretest-posttest increase was significant for the Recognition of information quality markers of an authoritative site, F(1, 46)=9.50, p<0.01, and Knowledge of general online information quality criteria F(1,46)=7.61, p<0.01, but not Recognition of information quality markers of a hoax site score. There was also a significant interaction between site and time of measurement, Wilks' Λ=0.51, F(18,124.94)=1.86, p=0.02. Univariate tests indicated that the magnitude changes in Recognition of information quality markers of an authoritative site, F(6, 46)=3.44, p<0.01, but not the other two variables, differed by site and contributed to the overall effect. Results of paired sample t-test comparison of pretest-posttest group means are reported in Table 2.    Table 3.