Introduction 1 2 3 4 5 6 7 8 9 10 11 12 13 http://www.medmatrix.org/ http://www.HealthAtoZ.com/ 14 12 14 15 6 15 16 17 The ideal method for assessing the quality of medical websites should provide a means of rating great amounts of medical web resources while respecting the World Wide Web peculiarities, such as its multimedia capabilities and changing contents. At the same time, it should at least be as reliable as systematic reviews of those resources by editorial boards. In summary, it should be a method born in the Internet but with the efficacy of those used in the printed media. In this study, we evaluated the reliability of four websites characteristics as medical websites quality indicators. The four characteristis used: their authors' impact factor, their grade of updating, their daily visits and inbound links. The evaluations of a sample of pediatric websites by a number of Internet rating systems was the gold standard with which these websites characteristics were compared. Methods During March 1998 a subset of websites rating systems were compiled. From these, we selected a sample of websites that were studied during the first week of April 1998. 13 14 Table 1 Only three rating systems (Medical Matrix, Physician Choice, and Six Senses) gave information about their editorial boards. Most of their members were physicians. Two of the web rating systems only gave a global result of their websites evaluation (Medical Matrix and Magellan), while the rest (HealthAtoZ, Argus Clearinghouse, Lycos Top 5%, Sympatico Health, Physician Choice, and Six Senses) gave a result for each considered criterion. Content was a common criterion to all the eight ranking systems. Therefore, the results of the evaluation of each web site were divided in two categories, content and non-content (design) aspects. In order to make comparisons, the results of the evaluations of the websites supplied by each rating system were transformed to a one hundred scale. When provided, the daily visits registered by the websites visits counters were recorded. In some websites the date from which the counter was started was not available. Thus, their webmasters were asked for this information by electronic mail, and it was included in the statistical study if provided before the end of the observation period, 15th April 1998. 18 When provided, the time since the last update was also recorded. 19 20 S Results After excluding 93 non-accessible websites, a total of 363 pediatric websites were compiled. On average, the websites of our sample received links from 470 other sites on the Internet (range, 0 to 3574). In 48% of the websites, information on their last update was given. On average, they had been updated 47.5 weeks before (range, 0 to 395). Only 10% of the websites had a visit counter, and the average daily visits were 470 (range, 1.2 to 3145). Seven visit counters did not distinguish among different visitors, that is, they registered any visit to their websites. In 137 websites (38%) the editor/author's name was given, but only 60 of them had published at least one article since January 1997 in the journals included in MEDLINE database. Their average impact factor was 2.14. Figure 1 Figure 2 Table 2 Table 3 Table 3 Finally, no correlation was demonstrated between the average impact factor of the websites authors and the other variables. Table 4 Discussion 13 Some aspects of our method are open to discussion. Firstly, the reliability of the data regarding the daily visits and the updating frequency depends on the accuracy of the information that the websites editors offer in their sites. In this sense, we considered the grade of updating of the websites by the dates of their last changes. Clearly these changes could involve very different aspects and in different grades, and not necessarily provide more current contents. However, we believe that it could demonstrate the editor's efforts in maintaining or increasing the interest of his web site for the visitors. The results regarding the number of daily visits to the websites must be considered with caution when comparing one web site to another, because some visit counters were set to register every visit, instead of every distinct visitor. Nevertheless, both can be considered usage indexes of a given web site. total 21 22 23 24 We could not find statically significant correlations among the evaluations of the websites by the different rating systems. This is probably due to the small size of the subset of websites indexed and evaluated by all the systems, and their different evaluation criteria. However, some interesting data were found when we considered the correlations among the four websites characteristics and the evaluations. We found that the best websites for HealthAtoZ, the largest analyzed rating system, were the most updated and the most linked ones. On the other hand, the most valuable websites for Medical Matrix, the second rating system by size, were the most visited ones. In any case, both the number of daily visits and the time since the last update highly correlated with the number of inbound links. The lack of correlation among the four variables and the evaluations by the other rating systems could be due to their little contribution to our sample. 25 16 15 16 26 27 authoritative websites hub sites 28 16 29 16 Webometrics 30 a posteriori 31 In summary, although the Internet provides a very different publishing medium, traditional means borrowed from printed journals could also be used with this electronic media for achieving minimal levels of quality. These include certain peer review processes, that enhance the rigor of the documents submitted for publication taking in account the peculiarities of this media, and linking analysis as a measure of the citation on the World Wide Web.