|
|
||||||||
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
,
,1
* Alfa Institute of Biomedical Sciences (AIBS), Athens, Greece;
Department of Medicine, Tufts University School of Medicine, Boston, Massachusetts, USA;
Department of Medicine, Henry Dunant Hospital, Athens, Greece; and
Network of Scientometrics Studies for Higher Education, National Scientific Research Center, Havana, Cuba
1Correspondence: Alfa Institute of Biomedical Sciences (AIBS), 9 Neapoleos St., 151 23 Marousi, Greece. E-mail: m.falagas{at}aibs.gr
| ABSTRACT |
|---|
|
|
|---|
Key Words: bibliometric analysis quality of publications bibliographic databases mathematical computing scientometrics
| INTRODUCTION |
|---|
|
|
|---|
Although the journal impact factor has been widely regarded as the best instrument for the evaluation of the quality of scientific journals, it has not been spared from criticism(5
6
7
8)
. Main points of consideration regarding methodological aspects in the calculation of this index include the lack of assessment of the quality of citations (9)
, the inclusion of self-citations (10
11
12)
, the poor comparability between different scientific fields (13)
, and the analysis of mainly English-language publications (14
15
16)
.
In fact, many researchers have proposed different approaches in the evaluation of the quality of scientific journals. The common point in most of these approaches is the assessment of the quality of citations received by a journal (17
18
19
20)
. The quality of citations can be estimated analyzing the networks of scientific papers with sophisticated mathematical algorithms (21)
. The PageRank algorithm, used in the evaluation of webpages by the popular Google search engines, has been proposed as an appropriate model for the evaluation of the quality of citations in scientific journals (22
, 23)
. In fact, a group of researchers at the University of Washington developed a similar algorithm for the evaluation of the influence of scientific journals included in the Thompson Scientific Journal Citation Reports (JCR) dataset (24)
. Furthermore, another research group from Spanish Universities developed an indicator, named SCImago Journal Rank (SJR) indicator, for the assessment of the quality of scientific journals, applying the PageRank algorithm on the Scopus database (25)
. In this evolving context, we sought to identify and evaluate the main characteristics and differences between the widely used journal IF by ISI and the newly introduced SJR indicator.
| MATERIALS AND METHODS |
|---|
|
|
|---|
In addition, we listed the journals with the top 100 journal IFs and retrieved information regarding their ranking in the SJR indicator list by matching their international standard serial number (ISSN). We also listed the journals with the top 100 SJR indicators and found their ranking in the list of journal IFs. Finally, we calculated the median value as well as the first and third quartile values of the absolute change in ranking order of the journals in the two top 100 lists (with the use of each one of the compared indices vs. the other).
| RESULTS |
|---|
|
|
|---|
|
The SJR indicator of a specific journal for a three-calendar-year period is calculated through an iteration process that computes the "prestige" gained by the journal through the transfer of prestige from all the other journals included in the network of journals, by their citations during the past 3 yr to all articles of the specific journal published in the past 3 yr, divided by the total number of articles of the specific journal during the 3 yr period in regard. The amount of prestige of each journal transferred to another journal in the network is computed by considering the percentage of citations of the former journal that are directed to articles of the latter journal (26)
.
Table 2
presents the first 20 ranked journals of all categories with each one of the compared indices and their corresponding rank using the other index. Of the 20 journals with the highest journal IFs, 13 retain a position in the top 20 journals with the use of the SJR indicator, and vice versa. Regarding the top 100 journals in the 2006 journal IF ranking order, the median absolute change in their ranking position with the use of the SJR indicator is 32 (1st quartile: 12; 3rd quartile: 75). Conversely, regarding the top 100 journals with the current SJR indicator, the median absolute change in their ranking position with the use of the journal IF is 29 (1st quartile: 10.5; 3rd quartile: 65.5).
|
The journals with the greatest relative increase in their ranking order in the SJR indicator compared to the journal IF were the journals Immunity, Molecular Cell, Annual Review of Cell and Developmental Biology, Cell, and Current Opinion in Cell Biology. The journals with the greatest relative stability in their ranking order from the journal IF to the SJR classification were Annual Review of Physiology, Annual Review of Biophysics and Biomolecular Structure, PLoS Biology, Annual Review of Plant Biology, and Blood. The journals that exhibited the greatest relative decrease in their ranking order in the SJR indicator compared to the journal IF were Annual Review of Astronomy and Astrophysics, JAMA–Journal of the American Medical Association, Nature Physics, Advances in Catalysis, and Behavioral and Brain Sciences. It should be mentioned that all five of the latter journals exhibited a fall of more than 4000 positions in their ranking order with the use of the SJR indicator compared to the journal IF. Exploring the considerable discrepancy in the rankings of the latter five journals, we noted as potential causes a computational error, the presence of another journal with the same title but different ISSN, lack of source data, and a divergence between the number of citable articles and of total articles or documents. Furthermore, one of the top 100 journals according to the journal IF classification (Trends in Ecology & Evolution) was not included in the SJR database.
| DISCUSSION |
|---|
|
|
|---|
Regarding the differences in the scientific databases, on which the compared indices of evaluation of scientific journals are applied, Scopus includes a substantially larger collection of journals, originating from remarkably more countries and published in a greater variety of languages (27)
. Thus, in this regard it can be assumed that SJR may provide a more comprehensive estimation of the scientific value of journals, particularly so for those published in non-English languages (16)
. This is why the latter category of journals receives a great percentage of the overall incoming citations from non-English journals (14
, 15)
. The latter are rather underrepresented in the Web of Science database. Furthermore, the Web of Science takes into consideration citations originating from a subset of source journals (28)
, potentially excluding some journals published in non-English languages, a factor that may influence the evaluation of this category of journals (14)
. Yet, it should be mentioned that although Scopus includes a larger collection of non-English journals, the latter are still underrepresented, constituting
15% of the total number of included journals (29)
. Moreover, the apparent advantage of Scopus in citation analysis originating from the breadth of its database is limited to the time period after 1996 for which citation analysis is available (30)
.
Regarding the methodology of the calculation of the two compared indices, the most significant difference lies in the fact that the SJR indicator takes into account not only the absolute number but also the "quality" of citations received by a journal, whereas the journal IF considers incoming citations only in a quantitative manner. It is plausible that the articles of a journal have a greater impact in science if they are cited by journals of higher scientific quality. Such an analysis may not have been feasible in the past, though in todays electronic era, powerful computational systems provide the opportunity to apply sophisticated algorithms for the evaluation of interactions between journals in a huge network or universe of scientific publications. However, the simple and comprehensible methodology used in the calculation of the journal IF, despite the controversy raised over the years for the output data (31
, 32)
, has been regarded as one of the most favorable attributes of this long-used standard of reference in the field of analysis of scientific citations.
It should be noted though that the simple methodology used in the calculation of the journal IF has allowed editors to use various practices aiming to increase the impact factor of their journals (5
, 33
, 34)
. The principal ones are probably the promotion of self-citations (10
11
12
, 35)
, the predilection for review articles (36)
, and the decrease in the total number of included articles (37)
. Notably, the use of the SJR indicator allows for the estimation of a journals impact without the influence of self-citations, since prestige can be transferred to a journal by all other journals but not by itself. Instead, in JCR providing the journal IFs, self-citation analysis for each journal can be separately performed. However, this factor is not incorporated in the calculation of the journal IFs. Regarding the weight of the different types of articles in the process of calculating the two compared indices of scientific journals, no provision has been made for differentiating between the value assigned to original research articles compared to review articles in any of the indices.
One of the major shortcomings of the SJR indicator may be the fact that it divides the prestige gained by a journal, through the citations of its articles, to the total number of articles included, rather than to the number of citable articles, as is used in the calculation of the journal IF. Although the strategy used in the calculation of the SJR indicator may be mathematically valid, since in theory any published article can be cited and all citations are taken into account in the numerator of the fraction, in practice, article types such as correspondence articles, letters to the editor, commentaries, perspectives, news, obituaries, editorials, interviews, and tributes are unlikely to receive a significant number of citations (3)
. In this regard, the scientific quality of journals that contain a large number of the latter types of articles, which may otherwise be of interest to the reader, are expected to be appreciably underestimated with the SJR indicator. Yet, it should be mentioned that neither is the journal IF an optimal index of quality of scientific journals regarding this issue, since it does not adjust for the fact that a great number of total citations addressed to a journal are received by a relatively short portion of the included articles (3
, 38
, 39)
.
The effect of the differences in the time periods used for the assessment of the two herein compared indices of quality of scientific journals on the accuracy of the estimates cannot be directly inferred in the context of the present study. It should be noted though that both of the indices refer to the past three calendar-year periods, although through a different approach. This temporal window may be relatively short, particularly for journals with an appreciable interval between receipt of an article and publication (3
, 40)
.
An indisputable advantage of journal IF over any new index of evaluation of quality of scientific journals is tradition. On the one hand, authors compete for publishing the products of their research in highly ranked journals (41)
. On the other hand, journal editors elect to publish scientific articles that are expected to be highly cited (37)
. Thus, it can be assumed that journal IF rankings would have influenced accordingly the quality of the journals over the years in which they have been considered as the sole standard of reference (3)
.
Although our study did not aim to systematically assess the comparability in journal rankings between the novel SJR indicator and the well-established journal IF index, we observed that the introduction of the SJR indicator does not bring about radical changes in this regard. In fact, half of the journals in the top 100 journal IF list are placed within a reasonable range of 32 ranking places in the SJR indicator journal list. Some striking inconsistencies observed in the rankings of the new SJR indicator, compared to the journal impact factor, should be attributed to the "precocity" of the new indicator rather than to systematic differences in the methodologies used. In fact, the two methodologies have been systematically compared in previous studies. A weighted PageRank algorithm applied in the same dataset used for the derivation of the journal impact factor provided results that correlated significantly with the later index (23)
. However, this correlation was moderately strong and differed between various scientific fields, for instance, for medical journals (9
, 23)
.
The SCImago journal rank indicator is a novel instrument for the evaluation of scientific journals that may challenge the established premiership of the journal IF in ranking scientific journals. It provides unrestricted (open) access, is based on a larger source journal database, and focuses on the quality of citations that a journal receives by other journals, rather than the absolute number. However, the sophisticated methodology used in the calculation of the SJR indicator needs to be adequately validated, and certain characteristics may need to be reconsidered before definitive conclusions for its applicability could be drawn. It appears, though, that the election of one index or the other would be mostly a matter of whether the popularity or the quality of a journal is considered as the primary criterion for the evaluation of scientific journals.
| ACKNOWLEDGMENTS |
|---|
| REFERENCES |
|---|
|
|
|---|
Related Articles
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | SUBSCRIPTIONS | ARCHIVE | SEARCH | TABLE OF CONTENTS |