Health websites on COVID-19: are they readable and credible enough to help public self-care?

Objective: There are concerns about nonscientific and/or unclear information on the coronavirus disease 2019 (COVID-19) that is available on the Internet. Furthermore, people's ability to understand health information varies and depends on their skills in reading and interpreting information. This study aims to evaluate the readability and creditability of websites with COVID-19-related information. Methods: The search terms “coronavirus,” “COVID,” and “COVID-19” were input into Google. The websites of the first thirty results for each search term were evaluated in terms of their credibility and readability using the Health On the Net Foundation code of conduct (HONcode) and Flesch-Kincaid Grade Level (FKGL), Simple Measure of Gobbledygook (SMOG), Gunning Fog, and Flesch Reading Ease Score (FRE) scales, respectively. Results: The readability of COVID-19-related health information on websites was suitable for high school graduates or college students and, thus, was far above the recommended readability level. Most websites that were examined (87.2%) had not been officially certified by HONcode. There was no significant difference in the readability scores of websites with and without HONcode certification. Conclusion: These results suggest that organizations should improve the readability of their websites and provide information that more people can understand. This could lead to greater health literacy, less health anxiety, and the provision of better preventive information about the disease.


INTRODUCTION
As a large family of viruses, coronaviruses are responsible for multiple diseases. In humans, these include the common cold alongside severe diseases such as Middle East respiratory syndrome (MERS-CoV) and severe acute respiratory syndrome (SARS-CoV) [1]. Coronavirus disease 2019 (COVID- 19) was first reported in Wuhan, China, on December 31, 2019 [2]. According to a report published by the World Health Organization (WHO) in May 2020, COVID-19 spread across 216 countries very rapidly [3]. As a vaccine for this virus has yet to be developed, nonpharmacological interventions, such as increased hygiene, are crucial for controlling the virus and reducing the risk of infection [4,5].
Many people consider the Internet to be a useful and important source of health information [6][7][8][9] that can encourage the use of preventive strategies and consultation with physicians [10]. On March 13, 2020, Google Trends reported that the term "coronavirus" was searched five million times (supplemental appendix), underlining the importance of the Internet as a source for health information. However, multiple studies demonstrate that health websites may not be credible and may contain inaccurate or misleading information [11][12][13][14][15][16]. Therefore, the information found on health websites could increase uncertainty and anxiety and, thereby, serve to harm people's health [17][18][19].
Trust in online health information has recently been of great concern [20] due to deficiencies in people's ability to judge the quality of this information [21,22]. The current COVID-19 pandemic further raises concerns about nonscientific, misleading, and unclear information disseminated via the Internet or other media [23,24], which can See end of article for supplemental content.
increase anxiety among the population. This pandemic may be particularly problematic for people with cyberchondria, defined as compulsive searching for health information online [25], because the existence of inaccurate information on the Internet could serve to increase this behavior. Apart from individuals with health anxiety, the pandemic's high mortality rate causes widespread fear, anxiety, and stress for people around the globe [26].
Although different motivations can contribute to online health information-seeking, searching for information about COVID-19 online can be understood as one way to cope with the stress associated with the pandemic [27]. For instance, people may search for information about COVID-19 to gain clarity about their own or a relative's symptoms as an attempt to alleviate uncertainty [28]. However, the fear and anxiety caused by pandemics such as COVID-19 can serve as an additional barrier to understanding information correctly [29] and may lead to further anxiety, inappropriate use of information, and unnecessary referrals to medical centers.
The proper use of health information depends on people's ability to understand, interpret, and comprehend that information [30] and can be influenced by many factors, including its readability [31]. To be considered readable, a text should be easy to read and contain concepts that are easy to understand [32]. Organizations such as the American Medical Association (AMA) and National Institutes of Health (NIH) advise that the readability of health information should not exceed a sixth-grade level [33] and should be understandable to eleven years olds [34]. Moreover, it should not use medical jargon [35]. However, online health information is often written in at a level that is not easily understood by many people [36][37][38][39][40][41].
Considering the increased use of the Internet to obtain health information on COVID-19 [42], it is of great importance to understand the readability and credibility of health websites. The objective of this study was to evaluate the readability and credibility of publicly accessible health websites containing COVID-19-related information.

Website selection and categorization
The authors searched for "coronavirus," "COVID," and "COVID-19" using Google, the most frequently used Internet search engine [43,44], on February 26, 2020. The search was performed using Google Chrome, and the browser history, cached images, and cookies were removed before the searches were carried out. As 90% of all search engine users browse only the first 3 pages of results pages (i.e., the first 30 results) [45], the first 30 websites returned by Google for each of the 3 keywords were examined, resulting in a total of 90 websites.
We excluded non-English-language websites, irrelevant websites, scientific papers, duplicate websites, inaccessible websites, primarily non-textbased websites (e.g., YouTube), and advertisementsponsored links. Based on these criteria, forty-three websites were excluded, and forty-seven unique websites were selected for evaluation ( Figure 1). These websites were then divided into the following five categories according to their statement of affiliation: commercial, news, educational, governmental, and organizational.

Readability evaluation tools
Four readability scales, each of which uses different techniques, were used to evaluate the readability of the websites: Flesch-Kincaid Grade Level (FKGL), Simple Measure of Gobbledygook (SMOG), Gunning Fog, and Flesch Reading Ease Score (FRE). These scales have been used in many readability evaluation studies of health websites on various topics and are reliable [46][47][48][49][50][51][52][53], and NIH recommends using FKGL, Gunning Fog, and SMOG for readability evaluations [54].
The FRE scale produces a score between 0 and 100, with higher scores indicating higher readability levels: scores of 90-100, 60-70, and 0-30 indicate that text can be understood by fifth-grade students, eighth-or ninth-grade students, and university graduates, respectively. The Gunning Fog scale produces scores of 5, 10, 15, or 20, which indicate that text is easy to read, hard to read, difficult to read, or very difficult to read, respectively. The SMOG and FKGL scales estimate the years of education a person needs to have completed to understand a written text; for example, a score of 7.4 indicates that a seventh-grader can understand the text. To apply these scales, a free online readability checker (Readability Formulas) was utilized [55]. This web-based tool has been used in readability evaluations of a wide variety of health-related websites [36,46,50,[56][57][58][59].

Credibility evaluation tool
Various tools-including JAMA [60], DISCERN [61], and Health On the Net Foundation code of conduct (HONcode) [62]-are available for evaluating the credibility of health websites. The HONcode consists of nine criteria: authority, complementarity, privacy, attribution, justifiability, transparency, financial disclosure, and advertising policy. The nonprofit HON Foundation, which is officially related to WHO, checks the credibility of websites at the request of the institutions hosting the websites. If they meet the criteria, the HON logo is placed on the website, which indicates that the website has been officially certified and is a reliable source of health information. The HON Foundation also provides a toolbar extension compatible with Chrome and Firefox browsers that helps people easily identify HONcode-certified websites while browsing [63]. HONcode is the oldest and most-used ethical and trustworthiness code for medical and health-related information available on the Internet [64]. It is reliable and has been used in multiple studies to assess the credibility of health websites [11,12,[14][15][16][65][66][67]. Therefore, we used the HONcode toolbar to identify certified websites containing information on COVID-19.

Statistical analysis
We tested whether HONcode-certified websites were more readable than non-HONcode-certified websites using independent t-tests. We also tested whether readability scores differed among website categories and depended on the website position on the search results pages using one-way analysis of variance (ANOVA). Statistical analysis was conducted using SPSS 18.

RESULTS
We analyzed forty-seven unique websites containing information on COVID-19 retrieved via Google. The retrieved websites appearing on the first page of the search results included more HONcode-certified websites than those on the second and third pages (Table 1). However, even on the first page of results, most websites were not officially approved. In total, only six websites were HONcode-certified, and all were commercial and/or organizational.
ANOVA showed no significant effect of website category on readability scores (Table 2). There was also no significant effect of search results page number on readability scores (Table 3).
T-tests showed no significant differences in readability scores between HONcode-certified and non-HONcode-certified websites (Table 4).
Among prominent international and national organizations, the readability of website content published by the National Health Service (NHS) and Centers for Disease Control and Prevention (CDC) scored highest and lowest, respectively ( Table 5). The readability level of content available through the WHO and NIH websites was "difficult to read."

DISCUSSION
Low literacy levels are a barrier to health knowledge. One solution to this problem is to provide content in plain language that is easy to read [68]. Using plain language can help convey information to a wider population [69] and allows users to find what they need, understand what they have found, and then use this information to meet their needs [70]. On a global level, individual reading abilities and the readability levels of consumer information contribute to the overall health of all people and society [71]. Due to the importance of readability in the context of health literacy, health promotion, and patient self-care, we evaluated the readability and credibility of COVID-19 information available on websites intended for the general public that were retrieved via Google searches.
Patient education materials should be easily understood by an average eleven-year-old or DOI: dx.doi.org/10.5195/jmla.2021.1020 Journal of the Medical Library Association 109 (1) January 2021 jmla.mlanet.org students in the sixth grade [34]. The results of this study show that the readability of COVID-19 information on websites is more advanced than the recommended level and is generally aimed at high school graduates or college students. This finding is consistent with studies examining online information on Ebola [72] and other diseases [73][74][75]. Moreover, our findings show that the readability level of website content published by international and national health organizations such as WHO and CDC was far above the recommended sixth-grade reading level. This is concerning because individuals consider these websites to be major sources of reliable health information, especially in health crises such as the current COVID-19 outbreak.
We also examined the readability levels of websites based on their category. Government websites were expected to be more readable than other types of websites, because their purpose is usually to educate the general public [76]. However, we found that the readability scores of websites in all categories, including governmental websites, were far above the recommended level. Although previous studies reported that governmental websites were more readable than other types of websites [73], we found that the readability of commercial websites was more suitable for a public audience than governmental websites. However, it should be noted that the information on commercial websites has been found to be of lower quality than other types of websites [77,78]. Consequently, people looking for information on the symptoms, prevention, treatment, and management of COVID-19 may come across websites that contain readable but inaccurate information or, conversely, accurate but unreadable information, meaning that decisions based on this information could increase their anxiety and even threaten their health [17][18][19].
Most people tend to browse search results presented on the first page [74,79]. Therefore, we expected that websites appearing on the first page of the search results would be more readable than those on the second and third pages. However, we found no significant difference in the mean readability scores of websites appearing on different pages of the search results. While organizations make efforts to increase the rankings of their websites in search engine results, we recommend that they also pay attention to the readability of their websites to ensure that their content can be understood. This would allow users to better understand the websites' content, satisfy their information needs, and help to prevent the spread of dangerous infectious diseases such as COVID-19.
We found that most websites that we examined were not officially certified by the HON Foundation. Therefore, individuals searching for information on COVID-19 may encounter websites that contain misinformation, which could lead to incorrect decision-making and anxiety. Moreover, both HONcode-certified and non-certified websites had poor readability scores. This finding contrasts with a similar study on prostate health, in which more credible websites were found to have better readability [80]. Therefore, we recommend that authoritative organizations providing health information about various infectious diseases, including COVID-19, pay more attention to increasing the readability of their websites to help people understand the information that is provided.

Limitations
There were some limitations in this study. For example, searches were conducted through Google Search, and using other search engines might have generated different results. In addition, due to the dynamic characteristics of websites, alternative results might have been obtained if the searches were conducted at different times.