Introduction The Integrator 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 17 18 19 20 21 In this sense, PubMed’s provision of tools and services (such as Related Articles and Bookshelf) can be seen to help users easily and immediately augment, in a limited way, shortcomings in their knowledge base. The tools and services compensate for shortcomings not only in searching for articles but also when comprehending, evaluating, and utilizing abstracts and full-text articles. The tools and services (such as Full-Text Access, Email Alert, Send To) also appear to be poised to support the user’s interest in and engagement with the indexed material (which is the other critical factor Alexander has identified for domain learning). By introducing CAM practitioners to the tools and services, one by one, and gaining from them a sense of the perceived value and contribution of each, this study seeks to determine the potential contribution of these tools and services to the knowledge-base scaffolding and the personal interest of health care workers who lie outside of PubMed’s originally intended audience of researchers and physicians. This work has obvious implications for the design of the tools, in PubMed’s effort to serve as broad an audience of health care practitioners as possible, and as a way, in turn, of extending the impact and benefits of life science research. Methods Sample The participants in this study were drawn from among those practising CAM. They were recruited from three sources: (1) participants 1-5 were drawn from among students taking an online professional development course at a community college on research literacy for the health professions, (2) participants 6-9 were recruited from the Massage Therapy Association of British Columbia, and (3) participants 10-18 were drawn from the Canadian Memorial Chiropractic College. The eight women and 10 men who agreed to participate in the study ranged in age from their 20s to their 50s. Of the 18 participants, 10 were pursuing or possessed a chiropractic degree, seven were registered massage therapists, and one practised homeopathy. A total of 13 participants had a university degree, with one holding a master’s degree and another, a PhD. In terms of background with online research, participants 1-5 were taking a course on research literacy, while participants 13-18 had been introduced to the topic in a session with a librarian, with some one-on-one follow-up. Design Table 1 Multimedia Appendix 1 Multimedia Appendix 2 Table 1 PubMed tools and services introduced to participants   Tool/Service Function Boolean Search Search terms can be entered using Boolean operators AND, OR, NOT. Search Limits Allows the user to limit the search by language; type of journal; gender; human or animal subjects; age; type of article; date; inclusion into PubMed. Related Articles This appears as a button/link with each citation and brings up a list of matching articles based on the original article. Full-Text Access This is provided through the publisher’s link on the abstract page and on the LinkOut page, with publisher, aggregator, PubMed Central, and/or subscribing libraries providing open access for roughly 15% of articles. Send To Send To allows users to email citations, convert citations to a text file, or save citations for editing and further action. Search History This tracks terms, time, and number of results in searches, while allowing Boolean searches combining different searches from the Search History. Bookshelf This provides links to relevant passages in a set of full-text life science books that range from basic science to clinical practices. MeSH Vocab MeSH is a controlled vocabulary database for ascertaining the most commonly used terms in PubMed. Email Alert The National Center for Biological Information (NCBI) of the National Library of Medicine (NLM) offers email notification of new work in specified topics. Author Links Authors’ names have links leading to a listing of all of their work in PubMed. Analysis Multimedia Appendix 3 Results Table 2 Table 2 Value of PubMed tools and services for participants (N = 18)               Participants ‡ Further Use of PubMed †        Search History Bookshelf    MF1 yes yes * * + ++ + * − * *  yes CM2 yes yes ++ + ++ + *  −    yes HF3 no yes * * * + + ++ −   + no MF4 yes yes * * * * ++  ++     MF5 yes yes * + + ++   ++ +    MM6 no no + ++ ++ * −  + −  −  MF7 no no ++ − ++ + +  ++    yes MF8 no no − − + ++ −  − +  − yes MM9 yes no + − − + −   ++  − no CF10 no no ++ ++ − + − +  −   yes CM11 yes no ++ − − ++ − ++  + ++ + yes CF12 yes no + + − ++ ++   ++   yes CM13 yes yes + − + + − + −  + + no CM14 yes yes + − * + * * −  +   CM15 yes yes − ++ ++ ++ ++ * + −   yes CM16 yes yes * + ++ * ++ + −     CM17 yes yes * * + * ++ + − * +   CM18 yes yes − ++ * * + − +     * † ‡ Blank cells indicate that tool or service was not covered in the participant’s session or participant did not respond to email follow-up. Boolean Search The basic starting point with PubMed, as with any online index or database, is the search, which typically allows the combining of search terms with AND, OR, or NOT for more accurate searching. These Boolean operators were familiar to a number of participants, but were a little vague for others: “I might have heard of it” (CF12). Learning about them made a considerable impression on a number of participants: CM2 I will use the Boolean search methods even when using the different features of the database. This is incredibly important to know, use, and understand. It’s like the building blocks of searches. I’ve got to practise that one afterwards.  At least one participant recognized that the search results were profoundly influenced by the terms of the search:  CM11 sic Search Limits Figure 1 Figure 1 Limits tab on PubMed  Related Articles Figure 2 Full-Text Access Figure 2 Figure 2 Partial results of a PubMed search on “complementary medicine,” showing icons (from top to bottom) for “no abstract,” “abstract only,” and “open access to full-text article” (although no abstract is available) and hyperlinks to “Related Articles” and “Links” (to the full text on publisher and/or library websites)  22 24 For most participants, the abstract was necessary as a starting point but was insufficient as source of information on its own. It served as a screen, as it would for a researcher:  CM2 So an abstract is really useful. If I felt the abstract was interesting, I would definitely need the full text to actually find the article useful to my work. An abstract only provides enough information to pass the first screening test. This chiropractor initially felt, on first looking through the list of PubMed search results, that “This information is not what I am looking for; it is too technical, too specialized.” However, by the time he had had the chance to download and look at an open-access article, he was expressing regrets: “This is incredible; it is so easy; I wish I could just finish reading the article right now” (CM2). Frustration over not being able to view full-text articles was a common theme: “There was an article that I was interested in looking at, and it said that it would not be available for about a year or 6 months” (CM14). “The only problem,” as a registered massage therapist put it, “is that the ones that are free are not the ones I was wanting” (MF8). For some, visiting a research library was not an option, and the cost of purchasing a single article posed a serious obstacle: “It deters me, big time, having to pay for articles” (MF8), and “Oh, you can order papers; I don’t have money to order papers” (CM14). For others, using a credit card online was an issue: “I don’t think that that’s a safe measure, with my credit card online” (MF5). Yet the interest in seeing an article was sufficient that a number of participants were prepared to give serious consideration to paying to download it: “It is incredibly important to have access to the full text of the article. I would even pay to access a full-text article” (CM2); “I think I’ll look for mostly free articles so I could actually read them. If I was desperate to find information and couldn’t find anything, then I might actually buy it” (HF3), and “depending on how valuable the article is I might be willing to do that” (MF7). For one chiropractor, who administered a professional chiropractic organization, having access to the full-text article was a must. He felt that the full text offered a more accurate picture of the research and provided the language he needed to use for his own work, which was to lobby the Canadian government to further integrate chiropractic into standard public health policies. As he explained, CM11 No one who has ever done research has ever done an abstract that exactly covers the detail of what’s done inside. So when you actually read the research paper you see more…. The abstract…never provides that robust meaty quote that I need. New England Journal of Medicine Another chiropractor conveyed the frustration of coming across a very good source of research, only to find access to the resource restricted. In this case, it was a systematic review conducted by the Cochrane Collaboration, which are not freely available: CM14 It says [reading from the abstract] “evidence which is based on the Cochrane review of the subject.” This review is like topnotch.... It’s called a Cochrane collaboration, and we don’t have free access to that.... It would be phenomenal if we had access to that.... It’s all based on a strict protocol, like how many people were in the study.... If you want to know the vast information about a subject, you should go to Cochrane reviews. Even in the case of a chiropractor who felt that “often times I get most of the information I need from the abstract” (CM17) and thus did not need the full text, there was a recanting after a few moments: CM17 Well, that’s not true. I do [need access to the full text]. Often times you are stuck because they tell you “in the article we describe” such and such. And then you do everything to get it. And we have different ways to get articles. A registered massage therapist noted how much she appreciated it when the researcher sent, prior to her session, the full text of an article of interest:  MF7 I especially valued that it was not just a summary.... It is very relevant to my work as a therapist, and as an instructor. [I] also found the references very helpful. And finally, a registered massage therapist who had recently obtained her MSc from a Canadian university, and thus had lost her ready access to the scholarly literature upon graduation, had learned about the issue of access in terms of how libraries were signing a new type of agreement with the journal publishers, which carefully controlled access: MF1 The contracts that the academic institutions sign with the publishers, limiting access to students and faculty only, mean less access for the public. When hard copies were available, anyone could read, digest, copy relative sections, etc. Now, depending on the institutional policy, it may be difficult even to get access to view a full text. Needless to say, the cost of individual articles is exorbitant. The library would allow the public access to its online journals at a small number of “public” terminals (for those designated as “walk-ins” in the contract), but as this participant noted,  MF1 [The university libraries] control [public access] very, very, very tightly. I went in and asked if I could use the computer.... I had 6 articles I wanted to see, and they let me do that. She concluded that this represented “the publisher’s stranglehold on the universities” (MF1). Another participant had caught wind of a hopeful agreement that would greatly increase access to medical research, not only for him, but the public at large: MM6 I was talking to a librarian here at the public library and she said in [British Columbia, Canada] they were...trying to link the public library with the university library so that anybody that has a library card [could view the journals]. It was PubMed’s clear identification of, as well as links to, the open-access articles that was the one aspect which participants consistently valued. The participants felt strongly about the importance of being able to read the full text, and while a few had access to research libraries (McGill and University of British Columbia cited), others were willing to consider paying for that access. But in all cases, the value of open access to this literature was judged as a critical aspect in becoming better informed about what the research had to offer their professional practice. Send-To Tools Users are able to select, organize, and save the citations they find by sending the selected citations to a file, text, printer, clipboard, email, or RSS feed. Five of the participants were very impressed by how this tool allowed one to build a personal set of research resources:  CM2 I’ve done this before [email search results to self] when I have found research on the Web. I think it’s great; it is very easy, and I like to keep things in my inbox. This is a great tool that I would use. Another chiropractor commented on how it enabled him to integrate PubMed into his work:  CM11 Over the course of the day, I can sort it and then I have my list in order. Now I can take that display list and send it to text. Search History PubMed keeps a record of each search that a user conducts, under History, allowing users to not only review their previous searches but to combine them using Boolean operators. “Wow. That’s amazing. If I ever want to go back to what I did, this is how. It’s like you don’t have to write anything down because it is all recorded automatically. This is really, really great” (HF3). While this tool only made it into the sessions of 10 participants, only one of those participants proved relatively indifferent to its powers: “That makes sense” (CM18). Bookshelf PubMed has integrated a number of full-text medical and life science books that can be searched separately or in conjunction with an article abstract to gain background information about key terms in the abstract (which are highlighted with links to the relevant passages in the books). Only one participant saw how access to the books might be used for providing background: “I really like it; if I am writing a thesis or a paper, I would see this as very useful” (MF5). More frequently, the participants in the study made a number of observations about the currency and focus of the research articles as the reasons they preferred them over books: “Well that’s interesting [referring to Bookshelf], but I like articles because they are more current” (MF1), and “I like journal articles because they are more specific” (HF3). A chiropractor reinforced this idea, to a degree that suggested how the background role of PubMed’s Bookshelf program was being missed by some: CM14 That’s interesting [referring to Bookshelf]. But...once you get into reading research articles, they are much better than reading books. Books are always behind a few years.... There is a lot of bias behind the author of the book—reading an article, you get the actual truth. Such responses suggest that the participants were picking up ideas about the importance of currency and the leading edge role of journals in the life sciences, ideas that need to be refined, certainly, but that are not far removed from common thinking within the research culture. MeSH Vocabulary Database The MeSH Database enables readers to look up the controlled vocabulary used to index the research literature. Among the 10 participants introduced to MeSH, there were challenges initially in comprehending it, to a degree not experienced with the other tools and services: “I don’t understand; what is this [MeSH Database]? Is this everything with the keywords?” (CF12), and “Well, I have taken the tutorial [on the MeSH Database], and I’ve tried to use it, and I know it provides you with headings” (MF1). However, with further explanation and a chance to work with it, at least one participant found it “totally awesome and very useful” (MF8), and another appreciated that “it helps to distinguish things, and so you can go in one direction or another depending on what you want” (MM9). However, that same participant pointed out that what he felt was an element of bias: MM9 The word “subluxation”...means something quite different to a medical doctor than it does to a chiropractor when it comes to small tiny restriction in motion or small tiny misalignment. But the medical profession still doesn’t believe such things can occur, so I see here that there is no such thing here as subluxation in the sense that the chiropractor means it—very interesting. PubMed’s sharing of its vocabulary used for indexing also spoke to the value of this service as a source of engagement and reflection. Email Alert and Notification Services The National Center for Biotechnology Information (NCBI) of the National Library of Medicine (NLM) is responsible for PubMed and offers users a personalized service that includes email alerts when materials of interest to them are indexed. While this tool came up in only five of the sessions, there was recognition of its value by all the participants: “I have an automatic referral for [from] NCBI every so often about updates of new research in this area” (MF1). A chiropractor had signed up with a similar service, PubCrawler, developed by a genetics research lab at Trinity College, Dublin, for alerting readers to new articles in areas of interest: “I’ve signed up to PubCrawler, which alerts me to new research in my area, and so [I] quite often scan these [almost daily]” (CM11). Author Links The ability to find other works by the same author, simply by clicking on the author’s name, was not generally valued by the participants with the exception of the homeopath and one of the chiropractors: “So I have heard of him before. I would think it would be interesting; it would be interesting to know what his perspective was on other things” (CF12). With the six participants who encountered this feature divided between interest in and indifference toward it, it may seem that this way of focusing on the work of an individual researcher is not as valued as the other means of tracking work. Comprehending Research In the course of reflecting on PubMed tools and services, some of the participants reflected on their understanding, as well as their critical reading and use, of the research they were searching. For example, a number of participants commented on the value of reading the abstract as a necessary first step in reading research: “I wouldn’t read a paper without an abstract” (HF3). By the same token, the abstract was not enough: “Definitely need to read the article. [Abstract] does not tell us enough here” (MF8). A chiropractor noted that the figures and illustrations available in the full text were needed to make sense of the orthopedic tests used in the study: CM15 [The abstract is] a good summary, but...if the article had pictures and stuff, because maybe these are new tests that have come out that the school hasn’t taught us yet, and they might be better than the ones we’ve learned...[then access] would be important. Another chiropractor spoke of the importance of seeing the research methodology described in detail: CM14 The abstracts usually don’t show enough about the methods of an article.... It depends on the method, if the paper is actually good. Similarly, a third chiropractor, who was satisfied with the abstract alone when it came to “advice or recommendations,” felt that when the “treatment” of a client was at issue, then more than the abstract was needed:  CM18 I can’t break down the methods they used [with the abstract alone], and if there’s any flaws in it...I won’t get it from the abstract. So I would like to see the whole thing. p sic If participants felt somewhat overwhelmed by the research at times—“It’s a little above my head” (CM13)—they also knew how to direct their reading in relation to their strengths. As a registered massage therapist put it: “This is a little too theoretical and general...[while I’m] looking for protocols for treating low back pain.... I’d be more on ‘protocol,’ ‘orthopedic,’ ‘massage therapy’” (MM9). Or, as a chiropractor explained: “It’s a bit too heavy; it is going to go beyond what I need clinically” (CF10). What was perhaps most notable about the participants’ responses to the research was their perception of the value of this public resource to their professional practice: “It’s pretty encouraging to find this [material in PubMed]; it’s not intimidating” (MF7), and “I use the database and feel quite comfortable with it” (CF10). A Growing Research Culture This research on CAM practitioners’ use of a life sciences research index reflects a growing influence of a research culture among the health professions and the public. Two participants made reference to their clients bringing in online health information: MF5 My problem is telling [my clients] that what they are finding in Google is not that sound scientific information because any practitioner out there, anybody can put something out onto Google or their office website, but it’s not necessarily valid. I always have clients who are asking me about information that has come out on certain treatments, so that is primarily what I would tend to use this [PubMed] for. That same increased interest in recent research had been expressed within the CAM community as well, as the administrator of a chiropractic organization stated:  CM11 Our membership expects us to know things instantly if they are in the public domain. They say, "Why don’t you know? Everyone else knows”. The increasing prominence of research in the health services sector was also reflected in the training received by graduates of the Canadian Memorial Chiropractic College: CM17 This school is very evidence-based. So a lot of the assignments are pretty much looking up research: sensitivity, specificity, tests, prognosis.... If you are efficient with any of these search engines, you can find the information quite quickly. I have used it in university as well, but not to a great extent. When I got here I had to use it much more. It depends on the clinician that you get; some are really, really, really, evidence-based, meaning they follow everything the research says. Another participant noted: CM16 They [at the Canadian Memorial Chiropractic College] talk so much about evidence-based care in medicine and chiropractic, and you have to have research articles in order to attempt to that. You need a way to do it and PubMed is the only way I know. He went on to add that “it is free.” This was further confirmed by a classmate: “They [the school] thrust that down our throats: ‘Make sure you are up to date on that’” (CM15). In turn, CAM practitioners are participating in the debates on the growing prominence of certain forms of medical research but in the following example also demonstrate limited understanding of the methodology: CM11 I wrote an article about [how]...it is more or less impossible to do a [randomized clinical trial] of chiropractic. You can’t blind a practitioner; it’s more or less impossible to blind the patient, so any effect of randomization is nonsense. Lymphology Continuing Use of PubMed In response to follow-up emails that were sent out some weeks after the session with the researcher, seven participants affirmed that they were using or were planning to use PubMed: CM2 Yes, I have used PubMed since our last conversation. I use it on a weekly basis. I use it to find information for patients and for myself. I am in the process of developing an acupuncture-related website and writing a book on health-related topics. Issues of access to research articles persisted in the use of PubMed for this chiropractor:  CM2 I have not purchased any articles, although I have been tempted. I tend to look at the full-text articles, and I have used the links LinkOut and Bookshelf with success. As well, he had begun to see how he could best use PubMed in his practice: CM2 I think I’ll use it to get research, to get on-the-fly research to answer questions. But I wouldn’t send my patients to use the database or conduct searches on this database after a consult with them because I think it would be too complicated. I would not refer them to PubMed to find their own research. There were also indications from the participants of PubMed raising the quality of knowledge that informs their practice: “I’m using [PubMed] now to look something up instead of Google” (CF12). Other participants spoke in the follow-up emails of their plans for continuing use of PubMed:  CM15 Now that I know more about it, I think I’ll use it more.... Now that we have had this session, I’ll be able to understand it a little bit better, especially this "Related Articles". This same participant had explained in his session with the researcher how the Limits tool would serve his practice: “[This] would be great if I had a patient come in with a problem, and I had to really specify my search according to his needs” (CM15).  Another chiropractic student spoke of PubMed use in terms of its public access as a source of professional development: CM17 I am not going to be in academia much longer. It would be great to know as much as I could before I leave. I plan on using this when I leave as well. Of course, not everyone who participated in this study went on to use PubMed afterward: HF3 Medical Post A similar theme was sounded by a massage therapist:  MM9 I have no urgent reason to [use PubMed], and my time is too occupied to browse.... Currently, I do not [need PubMed]. And a chiropractor wrote to say that while he is sometimes “directed to PubMed as a resource” at his school, the problem is that he “could never really get the entire PDF files” that he wanted (CM13). Discussion This study confirmed our hypothesis that CAM practitioners would find that certain PubMed tools and services, on being introduced to them, had the potential to contribute to their engagement with and understanding of the research that interested them. This study did not seek to measure the actual contribution of these tools and services to the users’ learning. It was designed to establish whether these users perceived the tools and services as having value, as a first step to subsequent studies that will assess the differences that individual tools might make to the participants’ learning. Among the PubMed tools and services to which the participants were introduced, only the service of full-text access proved a positive asset for engagement, comprehension, evaluation, and utilization. As well, the Boolean Search and Related Articles were also strong contributors to the participants’ work with PubMed, and all tools and services impressed more than a few participants as to their value and contribution. 25 26 27 28 29 30 31 The participants’ consistent interest in having access to the full text of the articles indexed in PubMed leads to a strong and immediate recommendation to improve that access: The NLM should make every effort to capture an otherwise missing and substantial source of open access to research and scholarship, namely, the published health sciences research that has been posted by authors in institutional repositories and on websites. While PubMed has an excellent system for identifying and connecting to open-access articles made available by publishers, it needs to develop similarly effective systems for tapping the published work that authors have posted, with the publisher’s permission, in archives and on websites. With numerous archiving mandates, both in place and pending, for this form of open access to research that has been funded by governments and foundations, PubMed needs to ensure that it is able to take advantage of this movement to greater openness. For example, PubMed could include a means of searching the more than 800 repositories worldwide. For those who study the use of health information, we see a need to push beyond the obvious limitations of this study. There are sufficient grounds for further studies carefully designed to assess the contribution that particular aspects of these tools provide for different types of users, with a focus on how the design of the tools and services augments the users’ comprehension, evaluation, and utilization of the materials they encounter. The prospect of the ongoing development of resources such as PubMed, along with an increasing degree of public and professional expectation of access to research and scholarship, holds much promise for the continuing educational quality of the Internet and society at large. While only a fraction of CAM practitioners and a proportion of their clients will pursue these forms of knowledge, there is a larger point to this. Through the combined efforts and commitment of the NLM, a good number of life science journals, and the researchers themselves (who post open-access copies of their work), the quality of knowledge that is publicly and universally available is increasing, adding to people’s understanding as well as to the health care practices that affect their lives. This study is one small demonstration of how the benefits of this greater access extend to a larger community than has been commonly considered when it comes to public resources such as PubMed.