Search

Research Impact

Library


1. Journal Impact

If you are a scholar planning to publish a journal article, you will be looking for a reputable journal to provide the best exposure for your work. For the qualitative aspects of this problem see Avoiding Predatory Publishers. The quantitative aspects are addressed by various forms of journal rankings, which attempt to estimate a journal’s relative importance compared with other journals in the same field.

Systems based on the rate at which a journal’s content is cited are all subject to certain inherent limitations and vulnerabilities:

  • Publication and citation practices vary among disciplines, making comparisons across different subject areas difficult.
  • Rates of citation are not necessarily stable over time, and can be affected by such factors as the rise of multiple authorship of papers.
  • Generally, no distinction is made between favorable and unfavorable citations, so they are not a sure indication of merit.
  • Rankings may be affected by certain editorial practices, such as publishing numerous review articles.
  • Journal rankings can be manipulated by excessive self-citation, or by groups of authors colluding to cite each other’s work.

For all their imperfections, journal rankings nonetheless provide valuable information. A list of the best-known systems follows.

Journal Impact Factor

The University of Winnipeg Library presently does not subscribe to any database that includes the Journal Impact Factor (JIF), such as the Journal Citation Reports (JCR) from ISI, so only a brief explanation of it will be given here. The JIF is the most established indicator of journal ranking, and measures the frequency with which the “average article” published in a journal has been cited in a particular period. The default period is two years. For example,

  • 2015 impact factor = (number of citations in 2015 of articles published in 2013-2014) ÷ (number of articles published in 2013-2014)
  • Five-year 2015 impact factor = (number of citations in 2015 of articles published in 2010-2014) ÷ (number of articles published in 2010-2014)

The JIF has been criticized for various limitations, such as inclusion of citations of materials (such as editorials and letters) that are not included in the denominator of the calculation formula, the inclusion of self-citations, and the lack of evaluation of the quality of the origin of the citation.

SCImago Journal Rankings

SCImago Journal Rankings (SJR) is a service similar to, though not yet as widely adopted, as the Journal Citation Reports mentioned above. The site identifies top-ranking journals by subject field, drawing its data from citations in Elsevier’s Scopus database. The SJR addresses some of the criticisms directed at the Journal Impact Factor, as it includes more journals, limits self-citation, and  weighs citations according to the importance of the journal where they were published, using an algorithm similar to that of Google PageRank.

Journal Metrics from Elsevier

Journal Metrics offers the SCImago Journal Rankings (SJR) as above, and the Source Normalized Impact per Paper (SNIP) ratio, for over 20,000 journals. These are downloadable as Excel files. The SNIP is a journal’s citation count per paper, divided by the total number of citations in a subject field, so that a single citation is scored higher in subject areas where citations are less likely, and vice versa.

Eigenfactor

The Eigenfactor Project, founded in 2007, presently covers literature published from 1997 to 2013. The Eigenfactor gives a ranking for journals indexed by ISI in Web of Knowledge, based for each on the number of citations its articles receive, weighting citations that come from other influential journals more heavily, and excluding self-citations. The Article Influence scores give the average ranking for a journal (the Eigenfactor divided by the number of articles in the journal), and are thus similar to the JIF. The site also provides data on journals’ Cost-Effectiveness.

Google Scholar Metrics

While based on a metric originally designed to measure author impact (the h-index described below), the list of journals indexed in Google Scholar also serves as an indicator of the impact of journals, giving five-year h-index and h-median values.

2. Author Impact

Author statistics can be as simple as a count of the number of times a single article has been cited, or as comprehensive as a metric that attempts to encapsulate the cumulative influence of entire body of work.

Citation counts

The impact of an individual article can be measured by the number of times it has been cited. Many databases, and a number of software applications, tally the number of times a specific article has been cited. Like journal metrics, citation counts are subject to certain inherent limitations:

  • Citation counts will vary from one source to another, depending on the number of and type of sources indexed by a database.
  • Journal articles are the predominant form of publication of original research in the Sciences, but not in the Social Sciences and Humanities, and databases which do not index books or book chapters under-represent activity in those disciplines.
  • The literature of some disciplines is more heavily cited than others.
  • Few sources of citation counts make it possible to identify self-citations, or to distinguish unfavorable citations from favorable ones.

Citation counts appear in a number of databases, including:

  • Google Scholar: In lists of the search results, look at the bottom of an entry for “cited by,” followed by a number. If you have a Google Scholar account and are logged in, the My Citation button will bring up a list, with citation counts, for all your publications discovered by the Google spider.
  • PLOS One: Lists of the search results provide not only citation counts, but number of views, and number of shares in social media. Learn more about PLOS article-level metrics.
  • Web of Science: A database of all of the Web of Science Citation indexes, including 8,500 international journals in the Sciences, Social Sciences, and Arts & Humanities. Results-lists report “Times Cited.”

The h-index (Hirsch index)

The Hirsch index, more widely known as the h-index, is a measure of an author’s scholarly impact. The h-index reflects both the number of publications and the number of citations per publication. An author with an h-index of n has at least n papers that have each been cited n times. In other words, their work has at least n2 citations in total. Limitations of the h-index include the following factors:

  • As with the JIF, the h-index is not a fair means of comparing authors across subject areas, as some disciplines naturally publish and cite more than others.
  • Because the h-index is calculated using the count of a researcher’s publications, and therefore reflects an author’s “scholarly age,” persons with shorter career spans are at a disadvantage, regardless of the importance of their discoveries.
  • An author’s h-index will be invalid if another researcher has the same name; for this reason, the only truly reliable means of deriving an author’s h-index is to use a list of publications provided by the author.
  • The h-index does not account for self-citations or negative citations, and may, therefore, misrepresent an author’s importance.

Sources for the h-index include:

  • Google Scholar: note that this source only provides an author’s h-index if he or she has created a user profile. To set up a user profile, click on the link at the top of the web page for “My citations.” For the h-index of journals, see Google Scholar Metrics.
  • Web of Science: a database of all of the Web of Science Citation indexes, including 8,500 international journals in the sciences, social sciences, and arts and humanities. To find an h-index, perform an Author Search, then on the search results page, click on “Create Citation Report.”

Altmetrics

As researchers increasingly publish and communicate about their work through web-based environments such as blogs, social networks, and institutional repositories, some scholars have begun to question the primacy of journal-based metrics in the assessment of scholarship. Altmetrics track new indicators of significance and impact outside the academy, using the social web — download counts, page views, mentions in news reports, social media, and blogs — to measure and represent the amount of attention an article receives.

Altmetrics can provide a more immediate measure of impact, and can measure the impact of non-traditional publications, like datasets and code. Moreover, Altmetrics can provide the context and reasons for a citation.

However, Altmetrics only measure how much attention a publication receives, not its quality. Other limitations include the fact that some disciplines or subjects naturally receive more attention than others, and the fact that older publications are not under-represented in social media.

Sources for altmetrics include: