Citation analysis of orthopaedic literature; 18 major orthopaedic journals compared for Impact Factor and SCImago

Abstract

Background

One of the disadvantages of the Impact Factor (IF) is self-citation. The SCImago Journal Rank (SJR) indicator excludes self-citations and considers the quality, rather than absolute numbers, of citations of a journal by other journals. The present study re-evaluated the influence of self-citation on the 2007 IF for 18 major orthopaedic journals and investigated the difference in ranking between IF and SJR.

Methods

The journals were analysed for self-citation both overall and divided into a general group (n = 8) and a specialized group (n = 10). Self-cited and self-citing rates, as well as citation densities and IFs corrected for self-citation (cIF), were calculated. The rankings of the 18 journals by IF and by SJR were compared and the absolute difference between these rankings (ΔR) was determined.

Results

Specialized journals had higher self-citing rates (p = 0.01, Δmedian = 9.50, 95%CI -19.42 to 0.42), higher self-cited rates (p = 0.0004, Δmedian = -10.50, 95%CI -15.28 to -5.72) and greater differences between IF and cIF (p = 0.003, Δmedian = 3.50, 95%CI -6.1 to 13.1). There was no significant correlation between self-citing rate and IF for both groups (general: r = 0.46, p = 0.27; specialized: r = 0.21, p = 0.56). When the difference in ranking between IF and SJR was compared between both groups, sub-specialist journals were ranked lower compared to their general counterparts (ΔR: p = 0.006, Δmedian = 2.0, 95%CI -0.39 to 4.39).

Conclusions

Citation analysis shows that specialized orthopaedic journals have specific self-citation tendencies. The correlation between self-cited rate and IF in our sample was large but, due to small sample size, not significant. The SJR excludes self-citations in its calculation and therefore enhances the underestimation in ranking of specialized journals.

Background

The IF [1] is considered the best reference utensil for evaluation of scientific journals although its limitations have already been described extensively [2–7]. Major points of criticism are the lack of quality assessment for citations [5], poor comparability between different domains of interest per journal [6] and the mainly English language in publications. A major problem with the IF for journals is self-citation, defined in Journal Citation Reports as referring to articles from the same journal [1]. Due to these limitations and the simplicity of calculating the IF, it would be relatively easy for editors to manipulate it. A journal's IF can be increased artificially by using self-citations, publishing relatively many review articles and limiting the number of articles included.

The internet search engine Google™ uses a PageRank algorithm to determine page ranking after a specific search query. Several authors have advocated a similar algorithm for the evaluation of scientific journals. Falagas et al. [5] recently described the application of a PageRank algorithm to the Scopus database to produce the SCImago Journal Rank (SJR) indicator. The SJR represents awarded prestige per article in the analysed year and is calculated using a complicated iterative formula. Self-citations do not contribute to the SJR, since a journal can receive prestige only from other journals, not from itself [8]. The SJR has several other benefits [5], such as the greater number of journals and languages included in its database and the fact that it is open-source software (free of charge).

In the present study we first aimed to evaluate the influence of self-citations on the 2007 IF for 18 major orthopaedic journals, including sub-specialist journals. These journals publish specialized articles concerning a narrow field of interest and as such are expected to receive relatively few citations from other journals. We therefore hypothesized that self-citation rates are higher for specialized journals than for the general orthopaedic literature. Secondly, we investigated ranking of orthopaedic journals with the SJR. This indicator excludes self-citations, therefore we hypothesized that the SJR ranks specialized journals comparatively lower than does the IF.

Methods

We included and analyzed 18 orthopaedic journals, making a distinction between general and specialized orthopaedic journals. Two orthopaedic clinicians participating in this study individually allocated journals to either the general or the specialized group.

The general group included the following journals: Acta Orthopaedica (Acta), Archives of Orthopaedics and Trauma (AOTS), BMC Musculoskeleletal Disorders (BMCMD), Clinical Orthopaedics and Related Research (CORR), International Orthopaedics (Int Orthop), the American Volume of Journal of Bone and Joint Surgery (JBJS [Am]), the British volume of Journal of Bone and Joint Surgery (JBJS [Br]) and Orthopaedic Clinics of North America (OCNA).

The specialized orthopaedic journals included in the study were: American Journal of Sports Medicine (AJSM), Arthroscopy, European Spine Journal (ESJ), Foot and Ankle International (FAI), Journal of Arthroplasty (JOA), Journal of Orthopaedic Trauma (JOT), Journal of Pediatric Orthopaedics (JPO), Journal of shoulder and elbow surgery (JSES), Knee surgery sports traumatology arthroscopy (KSSTA) and Spine.

For both groups self-citation was analysed for the 2007 IF. The self-citing rate, self-cited rate and citation density, parameters known to influence the IF [2], were calculated for each of these journals as defined by the ISI Web of Science [1]. Also, all IF were corrected for the influence of self-citations (cIF) [1]; for individual journal evaluation, an absolute change of >0.5 in IF (ΔIF = IF - cIF) was considered substantial. To estimate the influence of self-citation on the IF, journals in both groups were correlated for self-citing rate and IF.

The self-citing rate[1] was calculated by dividing the number of self-cited articles in journal X in 2007 by the total amount of citations by that journal in 2007. For example, in 2007 AJSM contained 1542 citations of AJSM articles on a total of 8089 citations; the self-citing rate was 1542/8089 = 19% (Table 1).

Table 1 Self-citing rate, self-cited rate, citation density and IF 2007

The ratio between the number of self-cited articles in journal X in 2007 and the total amount of citations received in 2007 for articles in journal X is the self-cited rate[1]. For example, AJSM was cited 10,711 times in total, of which 1542 by itself; the self-cited rate was 1542/10711 = 14% (Table 1).

The citation density[1] was determined by dividing the total number of references in journal X in 2007 by the total number of articles published in that journal in 2007. For example, the 239 articles published in AJSM in 2007 contained a total of 8089 references; the citation density was 8089/239 = 34 citations per article (Table 1).

We retrieved information regarding ranking by SJR via the SCImago journal- and country-rank website developed by the SCImago research group [8]; the journals were selected by matching international standard serial number (ISSN) found in the JCR. Since the IF ranks 6426 journals and the SJR 15,922, absolute rankings by IF and SJR are not comparable. Therefore only the 18 journals included in our analysis were mutually ranked for IF (R_IF) and SJR (R_SJR). The difference between R_IF and R_SJR, expressed as ΔR, was calculated to check for a possible difference between both rankings.

Statistical analysis

Statistical analysis was performed using Prism v5.00 for Windows (Graphpad Software Inc, San Diego, CA, USA). Analysing differences between general and specialized journals for the IF, the following specifics were compared using a Mann-Whitney test: self-citing rate, self-cited rate, citation density, ΔIF, R_IF, R_SJR and ΔR. We determined the medians, the first- and third-quartile values, and the difference between the medians (Δmedian = median_general - median_specialized) and their confidence intervals. A Spearman rank correlation coefficient was computed to estimate the correlation between self-citing rates and IF, as well as between the journal rankings by IF and by SJR. For all tests, p values < 0.05 were considered significant.

Results

Self-citation rates

The journals with the highest self-citing rates were CORR in the general group (15%) and Spine in the specialized group (29%) (Table 1); the lowest rates were for BMC MD/OCNA (general group, 1%) and KSSTA/ESJ (specialized group, 5%). The highest self-cited rates were for CORR/Int Orthop (general, 14%) and FAI (specialized, 38%), the lowest rates for OCNA (1%) and JPO (9%). Arthroscopy (0.75), AJSM (0.67) and Spine (0.63), all experience substantial declines in IF when corrected for self-citation (cIF).

Table 2 shows the medians and first- and third-quartile values of the citation analysis. Specialized journals have significantly higher self-citing rates (Δmedian = 9.50, 95%CI -19.42 to 0.42, p = 0.01) and self-cited rates (Δmedian = -10.50, 95%CI -15.28 to -5.72, p = 0.0004). Both types of journal tend to use similar numbers of citations per published article (Δmedian = 3.50; 95%CI -6.1 to 13.1, p = 0.35). The absolute difference between IF and cIF proved to be greater for specialized journals (Δmedian = -0.22; 95%CI -0.45 to -0.01; p = 0.003). There was no significant correlation between the self-cited rate and the IF for either the general (r = 0.46; p = 0.27) or the specialized group (r = 0.21; p = 0.56) (figure 1A).

Table 2 Citation analysis for general and specialized journal groups

Figure 1

Correlation Analysis. A: Correlation between self-cited rate (%) and impact factor B: Correlation between IF and SJR

Comparative Ranking: IF versus SCImago (Table 3)

The rankings by IF and SJR for the 18 journals in our analysis are presented in Table 3. For the SJR, an orthopaedic ranking is possible for the sixteen journals represented in the subcategory Medicine: Orthopaedics and Sport Medicine in the SJR database, but not for BMC MD (included in the subcategory Medicine: Miscellaneous) and Spine (included in the subcategory Biochemistry, Genetics and Molecular Biology).

Table 3 Comparative rankings of orthopaedic journals by journal impact factor and SCImago journal rank indicator

In comparing SJR rank relative to IF rank, seven journals maintained their rank (OCNA, JOT, JSES, JPO, FAI, AOTS, Int Orthop), six improved their rank (JBJS [Am], CORR, JBJS [Br], JoA, Acta, BMC MD) and five experienced a decline in rank (AJSM, Spine, Arthroscopy, ESJ, KSSTA). The greatest differences between SJR and IF rank were seen for BMC MD (seven places up in SJR) and ESJ/KSSTA (four places down). Despite these changes, a strong correlation was found between IF and SJR (general: r = 0,98, p < 0.0001) (specialized: r = 0.93, p < 0.0001) (figure 1B).

Table 4 presents the differences in ranking between both groups, IF (Δmedian = 1.00, 95%CI -7.13 to 9.13, p = 0.41) and for SJR (Δmedian = -3.50, 95%CI -12.2 to 5.2, p = 0.76). There was a significant difference in ΔR between both groups (Δmedian = 2.0, 95%CI -0.39 to 4.39, p = 0.006).

Table 4 Comparison of IF and SJR rankings

Discussion

Key findings

Our study revealed the following: (1) Specialized journals receive proportionally more self-citations with a strong influence on the IF. (2) The SJR shows a strong relation with the IF. (3) Correcting for self-citation with the newly introduced SJR results in substantial individual changes for journal ranking. (4) Specialized journals tend to drop, whereas general journals will climb, in rank when the SJR is applied instead of IF.

Group-specific characteristics were clearly visible. Sub-specialist journals have higher self-citing (p = 0.01) and self-cited rates (p = 0.0004), also expressed in a greater difference between IF and corrected IF for self-citations (p = 0.003). Despite the elaborate iterative calculation of the SJR, there exists a very strong correlation between IF and SJR ranking (general: r = 0.98, p < 0.0001) (specialized: r = 0.93, p < 0.0001). Comparing the mutual difference in ranking for SJR and IF between both groups showed an increased contrast (p = 0.003).

Strength and weaknesses of our study

Our study is strengthened by the analysis of two separate groups of orthopaedic literature: general and specialized orthopaedic journals. Hakkalamani et al. [9] excluded several journals because of a suspected increased self-citation because of subject subspecialty. Applying standard definitions for self-citing rate, self-cited rate and citation index from the ISI Web of Science to these journals, we were able to identify specific citation patterns for specialized journals. Comparison of the Impact Factor with the newly introduced SCImago journal rank indicator elucidated whether citation analysis based on a PageRank algorithm provides improved quality assessment of orthopaedic literature over journal indicators.

A limitation of this study is the evaluation of journal citations for 2007 only, and not for other years of publication. Furthermore, the distinction made between general and specialized journals is arbitrary; however, distinguishing journal categories by domain of interest is plausible and is coherent with work published previously [9]. Therefore, we believe that applying widely accepted methods for citation analysis to our analysis groups generated reliable data.

Previous Literature

Self-citation indicators that affect the IF are not limited to the orthopaedic literature [9]. Similar relations have been demonstrated for anaesthesia [10] and radiology [11]. The high self-citation indexes found within these specialized fields of medical science prompted suggestions that self-citations should be eliminated from the calculation of the IF. However for the 2007 IF in orthopaedic literature we did not find a similar distinct influence of self-citations. The high self-citation indexes for specialized journals relative to general journals rather indicate that these journals serve a small and isolated field within the orthopaedic literature [10]. The question can be posed whether high self-citation rates within a specialized domain of interest are indications for low-quality publications or rather reflect overall high quality? Papers published in these specialized journals are likely to have a high impact in their field of interest [6].

Implications of our study

Postma has stated that the impact of evolutionary papers published in multidisciplinary journals is substantially overestimated by their overall impact factor and, on the other hand, that the impact of papers in more specialized journals is significantly underestimated [6]. We agree that the IF has evolved for many users into an indicator of quality for articles, but one should always be aware of the fact that article citation rates determine the journal impact factor, not vice versa [3]. The differences found in our analysis of two groups of orthopaedic journals do not represent a difference in quality of articles published in these journals. They confirm that these groups publish different types of research, that is, either multidisciplinary or sub-specialized orthopaedic literature.

The SJR, based on an PageRank algorithm, provides a more sophisticated alternative for the IF and eliminates the effect of self-citations, which might be desirable if self-citation is seen as a negative aspect. Our analysis showed that the SJR bears a high resemblance with the IF, but further enhances differences between general and specialized literature. As a result of the iterative calculation of the SJR, the contrast between both groups of literature will increase over time.

Conclusions

Even though there was no significant correlation between self-cited rate and IF, there is a strong relation between these both indices. Self-citation indexes tended to be higher for specialized orthopaedic journals. The SJR corrects for self-citations, which increases the gap in overall ranking between specialized and general journals, this reflects a difference in field of interest rather than quality.