Introduction 1 Table 1 Table 1 Milestones in the development of NCI's website   Year Milestone 1995 CancerNet website is launched. 1999 cancerTrials website is launched. 1999 CancerNet website is redesigned. 2002 2 2004 NCI website is redesigned. ® 3 6 There was, however, a new dynamic in the development of the Web. Cancer patients were coming online in large numbers, seeking to be informed decision makers in their own care. Simultaneously, the patient advocacy community was becoming more vocal in requesting that NCI provide products geared to patients. NCI responded to this growing audience by organizing the CancerNet website by audience type, with entry points for patients, health professionals, and researchers, and with information categorized accordingly. The evolution and refinement of NCI's online cancer information has been notably “user driven.” NCI has adopted a multifaceted approach to gathering feedback and other information about how its information products are used. This has included pre- and post-design tests in usability labs, heuristic or expert review, informal user feedback, standard online user surveys, focus groups, analysis of site usage and search logs, and special user survey projects. Each generation of NCI's Web presence has been informed by user feedback. NCI staff members were crucial leaders in developing usability guidelines and standards that are now widely accepted in the industry, and NCI was one of the first federal agencies to conduct systematic usability testing with its CancerNet website. This article presents an overview of the methods NCI has used to gather input about its online information products and services. It is not the result of research projects that set out to test specific hypotheses about the impact of specific user-driven enhancements. Rather, it presents an approach to information architecture and design of a website that uses a variety of methods to gauge user behaviors and preferences. It highlights some of the many enhancements that have been implemented in response to user data and feedback. While NCI's website contains a wealth of additional information about cancer research opportunities, funding, NCI programs/initiatives, cancer statistics, and information for the news media, space limitations prevent a discussion of the role of user input in the design and implementation of these areas. The focus of this article will be enhancements to patient-oriented cancer information and information about clinical trials. How NCI Gathers User Input and Feedback Stakeholder Input NCI solicits user input prior to any major online system design or redesign. For example, in response to a growing need for clinical trials information, and prior to a major redesign of the backend database and the user interface of its CancerNet website, NCI organized the Clinical Trials Information System meeting in Chantilly, Virginia, USA, in 1998. Approximately 200 patients, advocates, clinicians, oncology nurses, clinical investigators, and health information providers representing the core users of NCI's online information resources came together to brainstorm the design of a clinical trials information system. Figure 1 Before and after screen shots of the CancerNet home page, showing a shift in focus from audience to topic with the 1999 redesign  Some of the key recommendations of the meeting were the following: (1) that the NCI website avoid segmenting information pathways according to type of user (patient, physician, researcher); (2) that information be customized to provide varying degrees of technical detail, complexity, and reading level; and (3) that users be able to easily move between these levels. It was also recommended that the NCI website integrate clinical trials information with the full spectrum of cancer information; include information about clinical trials, patient rights, and the informed consent process; and include a feature covering news topics related to clinical trials. One of the outcomes of this meeting was the development of a new NCI website, cancerTrials, to provide an educational context for the PDQ clinical trials registry that was offered on CancerNet. The cancerTrials website was launched in 1999. In addition to guidance on how to search the PDQ registry, visitors to the new site were offered original articles explaining what cancer clinical trials were, how they worked, and where to find them. They also were offered brief summaries of recently announced cancer trial results and other timely news related to the US clinical trials system. 7 Figure 1 Ongoing Feedback from CIS Information Specialists 8 User Surveys Table 2 Table 2 Type of information users were seeking on CancerNet (1999)   Percentage of Respondents (N = 780) Information Sought 22.8 Information on a specific type of cancer 18.6 Treatment information (general and specific) 11.3 Clinical trial information (specific trials, general information, trial results) 8.1 Symptoms of cancer, causes, risk factors, detection, diagnosis, prevention 6.8 Specific term (type of tumor or other term—not by name of cancer) 4.7 Cancer literature/articles 4.5 New treatments, news, recent findings, current research projects 3.5 Patient support (pain relief, diet/nutrition, survivorship, exercise, follow-up, questions to ask doctor) 3.1 Side effects 2.9 Statistics (incidence rates, survival rates, mortality rates) 2.8 Drug information 2.8 Access to other cancer resources (treatment facilities, physician names/specialties, national tumor registry, cost information, insurance coverage, patient support group) 2.2 Caregiver information (how to help patient, what to expect as disease progresses, how to talk to patient, etc) 2.1 History of cancer research, information for reports/projects 1.8 NCI publications (ordering information) 0.8 Alternative treatments 0.6 Genetic information (general and specific) 0.4 Search engine for the site 0.3 Information about oncology professions Figure 2 The “Types of Cancer” page on CancerNet  Figure 2 Figure 3 Figure 3 The “Breast Cancer” page on CancerNet, an example of a cancer-specific home page  In 2002, NCI's overarching website was redesigned, and the CancerNet and cancerTrials websites became the Cancer Information and Clinical Trials portal areas of the redesigned site. In 2004, the NCI site underwent another redesign, once again guided by extensive evaluation and user input. NCI's early decision to provide information tailored for patients and their families continues to be supported by surveys conducted during the past five years. Data from 1999 showed that 44% of visitors to the site described themselves as cancer patients or family members or friends of a cancer patient. Data from the American Customer Satisfaction Index (ACSI) survey posted on the NCI website in 2004 showed that more than 50% of respondents identified themselves as cancer patients or family or friends. The next largest audience in 2004 was health care providers, about 13%. NCI continues to keep the patient at the center of many of its online resources—PDQ's cancer information summaries and clinical trial abstracts, clinical trial results summaries, fact sheets and other information products, and the website's dictionary are all written for lay audiences. American Customer Satisfaction Index (ACSI Survey) 9 Appendix 1 The ACSI methodology provides continuous online feedback and is a uniform, national, cross-industry measure of customer satisfaction. A core set of ACSI questions measures overall satisfaction, and customized questions can be added regarding individual websites or pages. 10 Usability Testing and Expert Review 11 12 Appendix 2 Search Log Analysis Figure 4 Figure 4 The home page of the current NCI website, prominently featuring “Types of Cancer” and links to “Common Cancer Types”  Selected User-Driven Enhancements Best Bets on the Website When the NCI website was redesigned in 2002, the site's search tool was supplemented with a “Best Bets” feature that gives users a concise list of editorially selected NCI sites and pages that are displayed above the full set of search results. Whereas the full set of search results, which are generated by a free-text search of NCI's Web content, can number thousands of documents for a given search term, the Best Bets offer an average of two links, with a range of one to 18. There are currently 677 Best Bets categories (eg, lung cancer, mammography, cancer diagnosis program) with selected Spanish-language categories included. To populate Best Bets initially, a team of information experts identified cancer-related information categories, selected the most relevant NCI sites and pages for each, and created a table of related terms for each category name. When a search term is entered in the search box on the site and a category name or related term matches the term or any part of the term, the associated list of Best Bets is displayed. Search log analysis after the launch of the Best Bets feature in 2002 validated the choice of category names and related terms, the majority of which proved to be among the more popular search terms. Since 2002, the Best Bets database has been edited by NCI staff as needed, on the basis of periodic analysis of search logs and knowledge of new and changing NCI Web content. Log analysis has prompted a considerable expansion of the Best Bets database by suggesting new category names and related terms. In addition, there have been a few instances in which the large number of searches on a topic indicated the need for new content. These findings have already led to the creation of two important pieces of content (which, in turn, were classified as Best Bets), namely a fact sheet about cancer staging and a substantial resource on the NCI website called the “Tobacco and Cancer” home page. Best Bets categories, related terms, and links have also been added in response to comments from users. Clinical Trials Portal Redesign Figure 5 13 Figure 5 The Clinical Trials portal home page on NCI's website (2002)  Evaluation Methods The 2002 evaluation used six qualitative and quantitative methods: key informant interviews diary activity Focus groups in-depth interviews online surveys Usability testing log analysis Key Findings 14 With regard to the Clinical Trials portal, most users found the information they needed, were able to understand it, and found it useful. However, they had difficulty finding their way to the Clinical Trial Results news summaries, even though this type of information was among the top three categories of information desired. When directed there (or when identified as having been there via the pop-up exit survey), users found the summaries to be useful, understandable, and well organized. Informed Changes Figure 6 15 Figure 6 The Clinical Trials portal home page on the 2004 redesigned NCI website  16 In addition, links to the Clinical Trial Results pages organized by type of cancer are more prominently displayed on the Clinical Trials portal home page, and teasers (brief description and link) for the two most recently posted Results articles are prominently displayed on the Clinical Trials portal home page. Improved Searching for Clinical Trials 3 Since its appearance on the Web on CancerNet, the complexity of the clinical trials search form has been a topic of discussion within NCI. The PDQ clinical trials registry began as part of a DOS-based, menu-driven system used almost exclusively by health professionals, medical librarians, and cancer information specialists. Developers were wary of transplanting the sophisticated search functionality of this system to NCI's website because many of the site's visitors had little familiarity with clinical trials, cancer staging, treatment choices, and other elements in the original system. Initial methods of clinical trial searching on NCI's CancerNet website included a form with limited search options and clinical trial descriptions written in technical language, a legacy from the original system. Simply written, patient-friendly descriptions of clinical trials were introduced in 1997. Two-Step Search Form The second-generation search form that was launched in 1999 was based on recommendations from the Chantilly meeting, data from an online feedback form on the website, analysis of the search form, and personal interviews. In addition, a prototype of the form was developed through iterative rounds of usability testing. A two-step search form was designed to allow users to search by common search parameters, such as type of cancer, type of trial, and geographic location. Users could then review their search results or choose to narrow their search with other parameters, such as stage of cancer, drug (including brand and generic names), type of treatment, and trial sponsor. Users were also given the option of viewing two descriptions of each clinical trial, one for patients and one for health professionals. Other changes to the search form based on usability testing included a user's guide for less experienced users, annotated labels for search parameters with links to more detailed explanations, and explanations of how to select multiple items per field (eg, selecting several stages of breast cancer). Audience-Focused Search Form In the 2002 redesign of the NCI website, the clinical trials search form was included in the Clinical Trials portal of the site, giving users a more integrated information pathway that grouped information such as patient safety, informed consent, and insurance issues with the listing of clinical trials. A major consideration in this redesign was the addition of a specific new group of users—information specialists from the CIS, whose duties include assisting patients, their families, and health professionals in identifying clinical trials of interest. Information specialists had previously used the DOS-based, menu-driven PDQ search system that allowed them to perform complex searches, review results, refine as needed, and then prepare an “information packet” that could be emailed or mailed to callers. Web designers visited a CIS regional office to understand the needs of this group of users and did extensive usability testing with them prior to launching the revised form. Given the diversity of users, it became clear that a single search form was not ideal. Some users found the detailed choices on the form confusing and beyond what they needed. An interactive search form that guided users through the search process was considered, but such a form would require JavaScript, which does not meet Web accessibility requirements for federal government websites. It was determined that the best approach was to develop two search forms with different levels of complexity. Web accessibility requirements could be satisfied by creating one form without JavaScript, and a more complex, interactive form could be created with JavaScript. The basic search form, designed for the patient, caregiver, or busy health professional, provided three search options—cancer type, type of trial, and zip code proximity. The results were also presented in a format more suited to the casual Web user, who was accustomed to clicking on a search result link to go to a page that contained more information. Usability testing had also indicated that users did not normally click on the check boxes that were provided with the search results in order to prepare a “package” for viewing or printing as a batch. 17 In addition, for the CIS users, the search results display was developed to enable information specialists to read a preliminary result set, so they could identify the most appropriate trials for their callers and prepare an “information packet.” Better Visibility for Search Forms 15 Patient-Oriented Clinical Trial Abstracts In the summer of 1996, NCI collaborated with the National Alliance of Breast Cancer Organizations (NABCO) to develop patient-oriented abstracts (summaries) of clinical trial protocols for breast cancer trials. By October 1996, these clinical trial abstracts were available on the NABCO and CancerNet websites in a one-paragraph format. After seeking input from many advocacy organizations, the patient-oriented clinical trial abstract format was redesigned, writing guidelines were developed, and the project was expanded to include all cancer types. By September 1998, patient-oriented abstracts for all active clinical trials were available on CancerNet. Since that time, clinical trial abstracts have been written according to the original guidelines. In November 2001, selected patient-oriented and corresponding health professional clinical trial abstracts were evaluated. As a result of this evaluation, several problems were identified in the guidelines for writing the patient-oriented abstracts, including a lack of specificity in some respects and inconsistent application and interpretation of the guidelines. These findings led to the recommendation that the guidelines be redefined and expanded. Consequently, a qualitative and quantitative evaluation of the needs and preferences of users of the patient-oriented abstracts was undertaken. This evaluation included the following two elements: (1) a written survey of advocacy organizations, members of NCI's Consumer Advocates in Research and Related Activities (CARRA) Program, members of the NCI Director's Consumer Liaison Group (DCLG), comprehensive cancer center directors and administrators, cancer cooperative group chairs/administrators, and oncology nurses; and (2) in-depth interviews with CIS information specialists. Written Survey A 10-question survey was mailed to nearly 400 organizations and individuals, with a 43% return rate. A key question focused on whether or not users could understand and act on the information provided in the clinical trial abstracts for patients. Results showed that 82% of users could explain the rationale or purpose of the clinical trial, 93% could determine if basic eligibility requirements were met, and 73% could understand the treatment plan. The organization and layout of the clinical trial abstracts were rated “excellent” or “good” by 72% of the respondents. Three samples of text written at 5th-, 8th-, and 12th-grade reading levels were included with the survey. The different reading levels were preferred by 37%, 42%, and 20% of the respondents, respectively. The results of the survey were better understood when viewed in the context of comments from individual respondents. Taken as a whole, the respondents' comments were varied and, at times, contradictory. Several themes, however, emerged related to language and readability, access to other resources, and pursuing participation in a clinical trial. Although 27% of the respondents indicated they could not understand the treatment plan, few specific suggestions were offered for improvement. Interviews with CIS Information Specialists Structured interviews were conducted with staff in six CIS offices in different geographic areas of the United States in order to obtain their perceptions of users' needs, preferences, and comprehension of the standard elements (title, rationale, purpose, eligibility criteria, treatment, and study contacts) of the patient-oriented clinical trial abstracts. The CIS information specialists interact directly with users of the abstracts by answering their questions and by guiding their use of the abstracts online during a phone call or through LiveHelp. The information specialists emphasized the need to use consumer-oriented language and the fact that users “skip” disclaimer-type information. 18 Table 3 19 Table 3 Selected improvements to the patient-oriented clinical trial abstracts   Criteria Improvement Use of Language     Readability     Content Display       Web-Friendly Cancer Information Summaries for Patients The PDQ cancer information summaries are descriptions of the latest cancer information on treatment, supportive care, screening, prevention, genetics, and complementary and alternative medicine that are reviewed and updated monthly by cancer experts. Most of the summaries are available in two versions: one written for health professionals and a corresponding patient version written in lay language. (A small number of the summaries are available only in the health professional version.) In 2000, in response to the Chantilly meeting, work was initiated to reformat the patient-oriented information summaries. The goal was to present the information in a format and style of language that was easier to read and understand, to provide more detailed information, and to take advantage of features afforded by new Web technology. Based on design concepts that enhance readability, as well as on strategies used in information mapping, the process of reformatting and reorganizing the patient-oriented summaries was begun. “Key Points” boxes that highlighted critical concepts and linked to explanatory information in the body text were added. Links to pop-up definitions from the website's dictionary and to clinical trials information were included. For users who wished to print documents, a printer-friendly version was added that included dictionary terms and their definitions as an appended glossary. 18 20 Conclusion NCI's website is a leading resource for cancer information on the Web, consistently appearing high on the list of retrievals using search engines such as Google, Yahoo, MSN, and AltaVista. It has been awarded the Freddie Award in the website category of the 2004 International Health and Medical Media Awards, and it placed first or as an honorable mention in seven out of eight categories in the 2005 Medicine on the Net Web Excellence Awards. Its success can be at least partly attributed to NCI's efforts to make the site highly responsive to the needs of its users. The large volume of traffic that the site receives offers tremendous opportunities to study user patterns, gather feedback, and test new ideas and designs. Online surveys are an efficient way to solicit opinions from users, and analysis of website logs provides insight into user needs. NCI's relationships with members of the cancer research and advocacy communities also facilitate the gathering of advice, suggestions, and other feedback related to NCI information products. The growing body of Web-design literature and advice from usability experts are important to the development of new Web features, but input from the site's wide range of users promises to have the greatest impact on shaping online information from the National Cancer Institute.