Computerized versus hand-scored health literacy tools: a comparison of Simple Measure of Gobbledygook (SMOG) and Flesch-Kincaid in printed patient education materials
DOI:
https://doi.org/10.5195/jmla.2018.262Keywords:
Patient Education, Reading Grade Level, Health Literacy, Consumer Health, Simple Measure of Gobbledygook (SMOG), Flesch-Kincaid Grade Level, AppalachiaAbstract
Objective: The research compared and contrasted hand-scoring and computerized methods of evaluating the grade level of patient education materials that are distributed at an academic medical center in east Tennessee and sought to determine if these materials adhered to the American Medical Association’s (AMA’s) recommended reading level of sixth grade.
Methods: Librarians at an academic medical center located in the heart of Appalachian Tennessee initiated the assessment of 150 of the most used printed patient education materials. Based on the Flesch-Kincaid (F-K) scoring rubric, 2 of the 150 documents were excluded from statistical comparisons due to the absence of text (images only). Researchers assessed the remaining 148 documents using the hand-scored Simple Measure of Gobbledygook (SMOG) method and the computerized F-K grade level method. For SMOG, 3 independent reviewers hand-scored each of the 150 documents. For F-K, documents were analyzed using Microsoft Word. Reading grade levels scores were entered into a database for statistical analysis. Inter-rater reliability was calculated using intra-class correlation coefficients (ICC). Paired t-tests were used to compare readability means.
Results: Acceptable inter-rater reliability was found for SMOG (ICC=0.95). For the 148 documents assessed, SMOG produced a significantly higher mean reading grade level (M=9.6, SD=1.3) than F-K (M=6.5, SD=1.3; p<0.001). Additionally, when using the SMOG method of assessment, 147 of the 148 documents (99.3%) scored above the AMA’s recommended reading level of sixth grade.
Conclusions: Computerized health literacy assessment tools, used by many national patient education material providers, might not be representative of the actual reading grade levels of patient education materials. This is problematic in regions like Appalachia because materials may not be comprehensible to the area’s low-literacy patients. Medical librarians have the potential to advance their role in patient education to better serve their patient populations.References
Rubenstein E. From social hygiene to consumer health: libraries, health information, and the American public from the late nineteenth century to the 1980s. Libr Inf History. 2012;28(3):202–19.
Schneck JM. A bibliography on bibliotherapy and hospital libraries. Bull Med Libr Assoc. 1945 Jul;33(3):341–56.
Eakin D, Jackson SJ, Hannigan GG. Consumer health information: libraries as partners. Bull Med Libr Assoc. 1980 Apr;68(2):220–9.
Sollenberger JF, Holloway Jr. RG. The evolving role and value of libraries and librarians in health care. JAMA. 2013 Sep 25;310(12):1231–2.
McKnight M. Information prescriptions, 1930–2013: an international history and comprehensive review. J Med Libr Assoc. 2014 Oct;102(4):271–80. DOI: http://dx.doi.org/10.3163/1536-5050.102.4.008
Dee C, Stanley EE. Information-seeking behavior of nursing students and clinical nurses: implications for health sciences librarians. J Med Libr Assoc. 2005 Apr;93(2):213–22.
Hersh L, Salzman B, Snyderman D. Health literacy in primary care practice. Am Fam Physician. 2015 Jul 15;92(2):118–24.
US Department of Health and Human Services, Office of Disease Prevention and Health Promotion. National action plan to improve health literacy. The Department; 2010.
Mundt JC, Clarke GN, Burroughs D, Brenneman DO, Griest JH. Effectiveness of antidepressant pharmacotherapy: the impact of medication compliance and patient education. Depress Anxiety. 2001;13(1):1–10.
Neven P, Markopoulos C, Tanner M, Marty M, Kreienberg R, Atkins L, Franquet A, Gnant M, Neciosup S, Tesarova P, Barni S, Deschamp V. The impact of educational materials on compliance and persistence rates with adjuvant aromatase inhibitor treatment: first-year results from the compliance of aromatase inhibitors assessment in daily practice through educational approach (CARIATIDE) study. Breast. 2014 Aug;23(4):393–9.
Schaffer SD, Tian L. Promoting adherence: effects of theory-based asthma education. Clin Nurs Res. 2004 Feb;13(1):69–89.
Doak C, Doak LG, Root JH. The literacy problem. In: Teaching patients with low literacy skills. Philadelphia, PA: J.B. Lippincott; 1996. p. 1–8.
Roberts H, Zhang D, Dyer GS. The readability of AAOS patient education materials: evaluating the progress since 2008. J Bone Joint Surg Am. 2016 Sep 7;98(17):e70.
Badarudeen S, Sabharwal S. Assessing readability of patient education materials: current role in orthopaedics. Clin Orthop Relat Res. 2010 Oct;468(10):2572–80.
Joint Commission. Advancing effective communication, cultural competence, and patient-and family-centered care: a roadmap for hospitals [Internet]. The Commission; 2010 [cited 16 Oct 2017]. <https://www.jointcommission.org/assets/1/6/ARoadmapforHospitalsfinalversion727.pdf>.
Wallace LS, Lennon ES. American Academy of Family Physicians patient education materials: can patients read them? Fam Med. 2004 Sep;36(8):571–4.
McLaughlin GH. SMOG grading-a new readability formula. J Reading. 1969;12(8):639–46.
McGraw HC. SMOG conversion table 1. In: Office of Educational Research [class handout]. Townson, MD: Baltimore County Public Schools; 2016.
Rudd RE, Anderson JE. The health literacy environment of hospitals and health centers. Boston, MA: National Center of the Study of Adult Learning and Literacy; 2006.
DuBay WH. The principles of readability [Internet]. Costa Mesa, CA: Impact Information; 2004 [cited 16 Oct 2017]. <http://www.impact-information.com/impactinfo/readability02.pdf>.
Walsh TM, Volsko TA. Readability assessment of Internet-based consumer health information. Respir Care. 2008 Oct;53(10):1310–5.
Freda MC. The readability of American Academy of Pediatrics patient education brochures. J Pediatr Health Care. 2005 May–Jun;19(3):151–6.
US Department of Health and Human Services, Centers for Medicare & Medicaid Services. Toolkit for making written material clear and effective [Internet]. The Centers; 2012 [cited 16 Oct 2017]. <https://www.cms.gov/Outreach-and-Education/Outreach/WrittenMaterialsToolkit/index.html>.
Wang LW, Miller MJ, Schmitt MR, Wen FK. Assessing readability formula differences with written health information materials: application, results, and recommendations. Res Social Adm Pharm. 2013 Sep–Oct;9(5):503-16.
Shaw TC, De Young AJ, Rademacher EW. Educational attainment in Appalachia: growing with the nation, but challenges remain. J Appalachian Studies. 2004;10(3):307–29.
Wright D, Cunningham B, Stangle J. Identifying and addressing regional education needs. Appalachia Regional Advisory Committee; 2016. p. 1–52.
National Center for Education Statistics. State and county estimates of low literacy [Internet]. The Center [cited 16 Oct 2017]. <https://nces.ed.gov/naal/estimates/StateEstimates.aspx>.
Elsevier. Interactive patient education [Internet]. Elsevier; 2017 [cited 16 Oct 2017]. <https://www.elsevier.com/solutions/patient-engagement/interactive-patient-education>.
D’Alessandro DM, Kingsley P, Johnson-West J. The readability of pediatric patient education materials on the World Wide Web. Arch Pediatr Adolesc Med. 2001 Jul;155(7):807–12.
US Department of Health and Human Services, Centers for Medicare & Medicaid Services. Understanding readability formulas: a cautionary note. The Department; 2010.
Eltorai AE, Cheatham M, Naqvi SS, Marthi S, Dang V, Palumbo MA, Daniels AH. Is the readability of spine-related patient education material improving?: an assessment of subspecialty websites. Spine. 2016 Jun 15;41(12):1041–8.
Berkman ND, Sheridan SL, Donahue KE, Halpern DJ, Crotty K. Low health literacy and health outcomes: an updated systematic review. Ann Intern Med. 2011 Jul 19;155(2):97–107.
Moser DK, Robinson S, Biddle MJ, Pelter MM, Nesbitt TS, Southard J, Cooper L, Dracup K. Health literacy predicts morbidity and mortality in rural patients with heart failure. J Card Fail. 2015 Aug;21(8):612–8.
Mitchell SE, Sadikova E, Jack BW, Paasche-Orlow MK. Health literacy and 30-day postdischarge hospital utilization. J Health Commun. 2012;17(suppl 3):325–38.
Ryan L, Logsdon MC, McGill S, Stikes R, Senior B, Helinger B, Small B, Davis DW. Evaluation of printed health education materials for use by low-education families. J Nursing Scholarship. 2014 Jul;46(4):218–28.
Egeland M. Hospital librarians: from consumer health to patient education and beyond. J Hospital Librariansh. 2015;15(1):65–76.
Leonard K. Evaluating patient education materials for grade level. J Consumer Health Internet. 2017;21(1):87–94.
Blanck AW, Marshall C. Patient education materials from the layperson’s perspective: the importance of readability. J Nurses Staff Dev. 2011 Mar–Apr;27(2):E8–10; quiz E11–2.
