Introduction 1 4 3 health 5 6 7 1 8 9 10 11 12 13 14 15 1 16 17 We conducted an observational study of low-literacy adults to assess how they searched for Internet health information in as close to a natural setting as possible. Our investigative questions include the following: if low- to mid-level literacy adults are given access to the Internet, can they find basic health information that they can understand? Will their search strategies be effective in identifying information that they can use and comprehend? How do they rate current health Web sites in relation to their needs and interests? Will they be able to conduct successful self-directed searches? In our investigation, we also categorized navigational strategies used by low-literacy adults and the reading level of materials they accessed. Methods We enrolled 13 adult literacy students (3rd to 8th grade reading levels) from Bidwell Training Center, a vocational school in Pittsburgh, Pennsylvania. The protocol used was approved by the University of Pittsburgh Institutional Review Board. Bidwell students are organized individually and/or in small groups for reading instruction; they meet together once a week for program announcements. The literacy program coordinator introduced the study to 20 students in this large-group setting. Thirteen interested students then self-selected into the study. All subjects participated in a computer skills workshop in May 2003, where they were presented with basic search and navigation strategies and learned how to use the Google search engine. We selected Google because it is a widely used search engine and has a "Did You Mean…" feature that corrects misspelled search terms. We anticipated that this might be a feature that low-literacy subjects would find particularly helpful. Among other topics, subjects were taught how to use the "Back" button and the "Forward" button, how to scroll down a page, how to identify links, and how to conduct basic searches. Each subject also filled out a brief questionnaire to give insight on their educational background, ethnicity, health insurance status, and previous experience with computers and the Internet. The questionnaire was written at a 3rd grade reading level (Flesch-Kincaid Reading Scale). 18 The investigator then asked the participant to use the Internet and Google search engine and think aloud while finding information on a subject of his or her choice. This preliminary question allowed participants to practice and review their Internet searching techniques. Participants were permitted to ask the investigator technical and navigation-related questions during this part of the study. These questions included, but were not limited to, whether to put spaces between words in search terms and how to initiate a search once a search term had been specified. Results The investigator read the 3 questions aloud and also provided them to the participant in written form (Arial font, 20 pt): 1. Think of a health question you are interested in for yourself or for someone you know. Find out information about this question on the Internet. 2. Imagine that someone you care about has lung cancer. This person would like to know about treatments for lung cancer. Can you find out the three main types of treatments using the Internet? 3. Imagine that you are at a doctor's office and you are told you have a disease called diabetes (sometimes called sugar). You are given a pill called Metformin to take for it. What does Metformin do? Subjects identified answers to the investigator, who then asked them to articulate the answers in their own words. Participants who seemed frustrated or unreceptive, or who asked to move to a new question were directed to the next task. Participants were allowed to use any Web sites they felt would help them answer the questions. Participants also were not provided with dictionaries-our objective was to examine how they navigated the Internet without assistance from external sources. Subjects were given up to 15 minutes to complete each task, as measured by the investigator. To minimize anxiety, they were not informed of the time limitation. After the 15-minute period, investigators used a series of prompts to gradually guide subjects, if necessary, to the next task. 19 Investigators wrote notes on each participant's progress, and asked for participant feedback about the Internet both before and after searching the Internet. Investigators did not coach subjects on proper technical or navigational techniques after the initial practice question until subjects had completed their tasks. In 2 cases, investigators directed subjects to Google's "Did You Mean…" search term correction option in order to adjust for spelling mistakes; these subjects had repeatedly demonstrated very poor spelling proficiency before this intervention. Camtasia Studio screen-capture software recorded individual keystrokes and think-aloud recordings. Questionnaires and think-aloud methods were used to ascertain the criteria used by participants in evaluating Internet health Web sites. Investigators also calculated the 1) literacy levels of Web sites accessed by the participants, 2) the amount of time spent on each Web site, 3) the number of questions answered thoroughly and correctly by each participant according to pre-determined standards, 4) the average number of sites used to answer each question, and 5) the number of participants who accessed sponsored sites, or paid advertisements appearing on the Google retrievals page, and how many used that information to answer questions. Results Qualitative and quantitative results were analyzed in this study. Participants In this study, the subject population was reduced from 13 to 8. Two participants were excluded because they did not attend the one-on-one searching session with the investigator. Two other participants were excluded because they were non-native English speakers who did not understand the tasks presented to them. One participant was later excluded because technical problems prohibited the retrieval of her computer searches. The average age of our 8 remaining participants was 41.5 years. Five subjects were male and 3 were female. Seven identified themselves as African Americans and 1 self-identified as of Asian descent. The Asian participant was an English-as-a-second language (ESL) speaker with a university education from his native country. Seven of the 8 participants reported having health insurance. Seven of the 8 also had at least some high school or trade school education; 1 participant did not report educational experience on the intake questionnaire. Table 1 Table 1 Table 1 Search Engine Usage Participants reviewed their navigational skills during their preliminary question, where they were encouraged to look for information on any subject that interested them. They used Google to search for a variety of topics, ranging from entertainment to health-related information. Participants occasionally searched for information on more than one topic. Table 2 Table 2 Table 1 Questions 1 to 3 were given to our participants in writing, as well as orally; this may have affected their selection of search terms. For question 2, one participant wrote "treatments for lung cancer" in the search term box, a phrase that is written explicitly in that question. Another participant was similarly prompted by the wording of question 3 to write "a pill called metformin" as his search term. Table 1 Physicians' Desk Reference Yeah, you do have to space [between words]… I had to remember if you had to space. Soul Food So it [search engine] must go to other Smiths I wonder if I was supposed to put in 'Will Smith the actor'? You find a lot of stuff on this thing [the Internet]. Oh boy, I've got a lot to choose from. I don't want to go to the other ten [pages of retrievals] because it might give me other information I don't really need the first page gives me just enough of what I need to know. I didn't answer the questions, but I looked up the information, and it [Internet] gave me what it wanted me to have. Sites Accessed Ability to Answer Questions Shucks, I can't think aloud. Question 2 required participants to locate the 3 main types of lung cancer treatments (acceptable answers: chemotherapy, surgery, radiation). This question models the navigation of a typical Internet health-information seeker who searches for disease-related information. Of all 8 participants, only subject 5 was able to answer this question accurately and completely. Subject 3 verbalized one viable option-chemotherapy-based on information accessed online. The remaining participants either did not answer the question or identified an alternative medicine as one of the principal types of lung cancer treatments available. Question 3 required participants to find out the role of metformin, or Glucophage, in diabetes treatment (one acceptable answer: metformin lowers sugar in the blood). This question models a doctor-patient interaction in which a patient who is prescribed an unfamiliar medication independently searches for information about its effects. Six of 8 participants were unable to find information on the Internet to answer the question. The 2 participants, subjects 3 and 7, who found the information, read directly from text on the site and did not articulate the information in their own words. Surprisingly, subjects who reported sustained prior Internet experience in the questionnaire were no more successful at answering questions than subjects with little Internet experience. This could have been a result of the generalized search terms that they used to look for answers. Prior Internet experience does not seem to lead to satisfactory search/navigation skills for members of this group in searching for health information. Information Accessed Sites used by subjects 3, 5, and 7 to successfully answer questions 2 and 3 were written at a 12th grade reading level (Flesch-Kincaid). It is noteworthy that these subjects were able to identify the answer in the text and read it aloud. In 2 out of 3 cases, they were unable to express these answers in their own words, which suggests a minimal comprehension of the material accessed. Seven of the 8 participants accessed sponsored site information while attempting to answer questions. Businesses pay a service fee to Google to have their site names appear as sponsored sites when triggered by a particular search term or keyword. Sponsored sites are outlined in color and/or appear in boxes on the right side and heading of the Google retrievals page. In general, alternative treatments and commercial therapies and medications appear under this listing; many of these sites may contain information that is uncorroborated by legitimate scientific sources. 20 21 General Site Profiles Observational logs and records on the Camtasia software show little correlation between our subjects' ability to identify answers and the amount of text on a page; analysis using the Camtasia software also showed little conclusive difference in the amount of time that the subjects spent on each site despite variances in the amount of text on the pages accessed. Therefore, subjects did not seem to prefer or navigate towards Web pages/sites with less text. Table 3 at least Participants spent an overall average of 8.2 minutes on individual sites. All participants voluntarily finished answering questions 1 to 3 before the 15-minute time limit was reached. After completion of these first 3 questions, subjects were directed to a specific site; question 4 was posed about information directly linked to that site. We chose to use the ACS colon and rectum cancer Web page site, which contains links to a variety of prevention resources written at 6.3-12.0 grade levels (Flesch-Kincaid Reading Scale). The page to which we directed subjects consists of a listing of links to defined topic areas, one of which was closely related in wording to question 4. On the ACS site, 5 out of 8 people were able to answer question 4 correctly. Three of the 5 reported prior Internet experience; 2 reported none. These subjects used 3.8 links on average to answer the question. The 3 subjects who did not access the material used 6.5 sites on average before they were either stopped by the investigator or quit voluntarily. Two of these subjects had never used the Internet prior to enrollment in the study. Attitudes and Self-reporting While most participants were unable to answer all of the questions asked, 7 out of 8 reported feeling very comfortable or comfortable with their Internet searching experience. The eighth participant felt moderately comfortable. Also, 5 out of 8 found it at least moderately easy to find readable and understandable information on the Internet. Two of the remaining participants found it very difficult to find readable information, and one participant reported that finding understandable information is easy if the Web user has strong reading skills. I believe that on the Internet, you have your shysters just like anything. I'm getting a computer it can help your typing skills. The computer is real interesting. I'm a see if I can get one so I can learn [how to use it]. Discussion 11 Table 2 Difficulty Generating Search Terms 22 23 24 Reluctance to Use Links 23 Another reason why subjects' generation of search terms and selection of links were so limited may have been because the subjects were not interested in the health materials or the questions. Subjects may have also found the Google retrievals page confusing and intimidating. While the think-alouds are inconclusive about which of these factors contributed most to the weak search strategies observed, the post-session questionnaire reveals that the majority of participants reported that it was easy to search the Internet. Future research may help to illuminate the factors that contribute to the inconsistencies between subjects' perceived unwillingness to explore the Internet's health resources and their positive feedback about navigating through these resources. High Literacy Levels of Health Web Sites health 3 25 This is a real good one 'cause it breaks it right down for you Despite the navigational difficulties observed on the ACS Web page, the ability of 5 subjects to correctly answer question 4 probably resulted from the fact that the information needed to answer question 4 was written at an 8th grade reading level-significantly lower than the11th grade reading level required on average to read information retrieved in the first 3 searches. This suggests that low-literacy individuals can identify and utilize easier-to-read materials on Web sites. The Internet may indeed be a useful health resource to this population if materials are written at an appropriate reading level. Considering the navigational struggles of our subjects, the actual process of locating low-literacy sites on the Web may prove a more daunting challenge to this population. Difficulty Measuring Participants' Comprehension of Information 1 In addition, several subjects seemed to compensate for their low literacy skills by using external information resources. One subject who examined a Web site on mental health law (12th grade level) expressed great enthusiasm about a particular topic that he said was presented on the site. A perusal of the site after the session showed that this topic was not addressed on any of the pages he had accessed. This participant may have compensated for his struggles in reading the site by citing facts with which he was personally familiar. Another subject used a similar approach when accessing a lung cancer site. When asked about the type of information he was reading, the subject responded that the page focused on smoking cessation. However, there were no smoking-related topics on the pages examined by the subject. The subject was able to correlate lung cancer with smoking, and may have relied on this information in order to answer the investigator's query. Overall, some subjects may have been able to rely less on actual comprehension skills and more on background knowledge in order to infer answers. 26 Inaccurate Self-assessment 27 Preference for Sponsored Sites 28 motivation ability 29 Limitations of Methodology 30 31 13 15 and Indications for Future Research One caveat to the present study is that our sample size precluded the analysis of factors besides low literacy that could influence the results we observed. We believe, however, that our findings with this sample group in an observational study were representative of the way low-literacy adults interact with the Internet. It will be important to validate and analyze in a larger study the appeal of sponsored sites (as opposed to other retrieved links) to low-literacy adults. It will also be worthwhile to determine the relative importance of limited literacy in comparison to socioeconomic and cultural factors in effective use of the Internet by this population. Future work will identify the exact components of sites that engage and promote learning by low-literacy adults. Greater understanding of these factors will hasten the day when the Internet becomes an effective vehicle for optimizing the health knowledge and acumen for those at high risk of poor health outcomes.