Field-Weighting Readership: How Does it Compare to Field-Weighting Citations?

Recent advances in computational power and the advancement of the Internet, mean that we now have access to a wider array of data than ever before. If used appropriately, and in conjunction with peer evaluation and careful interpretation, metrics can inform and enhance research assessment through the benefits of being impartial, comparable, and scalable . There have been several calls for a “basket of metrics” to be incorporated into research evaluation. However, research is a multi-faceted and complex endeavour. Its outputs and outcomes vary, in particular by field, so that measuring research impact can be challenging. In this paper, we reflect on the concept of field-weighting and discuss field-weighting methodologies. We study applications of field-weighting for Mendeley reads and present comparative analyses of field-weighted citation impact (FWCI) and field-weighted readership impact (FWRI). We see that there is a strong correlation between the number of papers cited and read per country. Overall, per subject area for the most prolific countries, FWCI and FWRI values tend to be close. Variations per country tend to hold true per field. FWRI appears as a robust metrics that can offer a useful complement to FWCI, in that it provides insights on a different part of the scholarly communications cycle.


Introduction
The measurement of research impact is a growing and dynamic area, which is becoming more and more important. National assessment exercises such as the UK REF (Research Excellence Framework) and the Australian ERA (Excellence in Research for Australia) use research impact metrics, such as the h-index [1] and the journal Impact Factor [2] as part of their assessment criteria to distribute billions of pounds / dollars of research funding. Couple that with the growing number of global and national university rankings and the way that funding bodies assess funding applications, the growing use of research impact metrics is clear.
One of the most widely used traditional metrics, is the number of citations an entity has received. Be it a paper, journal, researcher, group of researchers or an institution, citation counts give an indication of how many times the research was referred to in other scientific publications. As with all metrics, the citation count has some major weaknesses, one of them being that you cannot compare the citation counts of documents from different subject fields. Studies have shown that fields such as mathematics are likely to be cited far less than fields such as biochemistry & molecular biology [3]. Even if when comparing papers within the same field, you must also take document age into consideration. For example, when published in the same field, does a paper published in 2005 with 40 citations have a bigger impact than one published in 2014 with 10 citations? The 2005 paper has had 9 additional years with which to build up citations, so it is not a fair comparison. Some researchers have employed PageRank algorithms in an attempt to overcome the weaknesses of citation counts and provide an alternative indicator to represent the academic influence of scientific papers. [4][5][6] Another way to compare outputs in fields of differing citation densities and ages is to use field normalisation.
The key role of such indicators, is to remove the effect that variables such as the age and subject area of a document have on a citation analysis, so that you can freely compare different documents against each other. At an article level, there are 2 commonly used normalisation methods -normalisation based on average citation counts and based on highly cited publications [7]. To calculate a normalisation metric for an article based on average citation counts, you need to know the expected number of citations for that publication. The expected number of citations for a paper is defined by calculating the average number of citations that similar document types in the same subject areas and year have received. For normalisation calculations done using Scopus data, the subject areas are defined as the 334 ASJCs (All Science Journal Classifications). Examples of a normalisation metric based on average citation counts is the field-weighted citation impact (FWCI) and the category normalised citation impact (CNCI) [8]. Both indicators are calculated at a paper level and can be aggregated to provide normalised values for an author, group of authors, institution etc, by calculating the average for all normalised values for all documents in the set. For journals or serial titles, dedicated normalised indicators have been created to allow cross discipline comparisons. One example is SNIP (source normalised impact per paper), which measures a journal's contextual citation impact. Using Scopus data, it's calculated annually by the Centre for Science and Technology Studies (CWTS) at Leiden University [9].
Another form of normalisation is based on highly cited publications. Here, field-dependent thresholds are used to determine if a publication is deemed to be highly cited [7]. The CiteScore Percentile metric is an example of a journal metric that uses such a methodology, which allows comparison between titles in different subject areas [10].
With the appearance of PlumX metrics and Altmetric.com, alternative metrics, or altmetrics, so named to differentiate themselves from the traditional metrics of citation and document indicators [11], are available to compliment traditional metrics. In PlumX for example, the metrics available include usage (clicks, views, downloads, library holdings, video plays), captures (bookmarks, favourites, reference manager saves), mentions (blog posts, news mentions, comments, reviews, Wikipedia mentions), social media (tweets, +1s, likes, shares) and citations (citation indexes, patent citations, clinical citations, policy citations) [12]. Many studies have been done on the advantages and disadvantages of altmetrics and potential correlations to citation counts [13,14] and if researchers have an appetite and willingness to use more metrics such as usage data [15]. Whilst this paper does not go into the pros and cons of alternative metrics, we realise a growing willingness to use such metrics in the available "basket of metrics". In a recent randomised study of data collected from Impact Story, Mendeley readership counts provided the most metrics [16]. As such we wanted to investigate if it was possible to create a normalised version of the Mendeley readership.

Methodology a. Data sources
• Scopus (https://www.scopus.com) is the world's largest abstract and citation database of peerreviewed literature, delivering a comprehensive overview of global research output in the fields of science, technology, medicine, social science, and arts and humanities. Scopus includes abstracts and citation information from more than 70 million records including peer-reviewed journals, books and conference papers. Content coverage of peer-reviewed literature in Scopus includes research articles in 22,800 peer-reviewed journals published by over 5,000 publishers. Scopus covers approximately 6,400 titles from North-America, 11,800 from Europe, 2,500 from Asia-Pacific, and 1,500 from Latin-America and Africa.
• Mendeley (https://www.mendeley.com) is a free reference manager and academic social network that helps researchers organize their research, collaborate with others online, and discover the latest research. It has millions of users, including not only students, post-doctoral researchers, professors/lecturers, and other academic researchers but also commercial R&D professionals, government/NGO researchers, and other professionals. Mendeley can capture all types of reference information, however, in this paper we only analyse the publications that can be found in Scopus.

b. Indicators
• Citations are formal references to earlier work made in an article, frequently to other journal articles.
A citation is used to credit the originator of an idea or finding and is usually used to indicate that the earlier work supports the claims of the work citing it. The number of citations received by an article from subsequently-published articles is a proxy of the quality or importance of the reported research.

c. Field-weighting
In order to study and compare the different indicators of impact previously defined, the field-weighted indicators have been computed. Field-weighted indicators allow the comparison of different publications because they take into account the differences due to the different document types, publication years and subject areas. The same methodology has been applied to the citations (FWCI, field-weighted citation impact) and the readers (FWRI, field-weighted readership impact). They are indicators of mean citation/ readership impact, and compare the actual number of citations/readers received by an article with the expected number of citations/readers for articles of the same document type, publication year and field. If the article is classified in two or more fields, the harmonic mean of the actual and expected citation rates is used. The indicator is therefore always defined with reference to a global baseline of 1.0 and intrinsically accounts for differences in citation/ reader accrual over time, differences in citation/reader rates for different document types (reviews typically attract more citations than research articles, for example) and subject fields.  Figure 1 shows that at country level, there is a strong correlation between the number of papers cited and read per country. This resonates with the findings from other Mendeley based analyses [16][17][18]. Among prolific countries, established research nations such as the US and European countries tend to have relatively more publications read than cited, compared to the trend line. Conversely, several emerging research nations, such as BRIC and some Asian countries, tend to have relatively fewer publications read than cited, compared to the trend line. To some extent, this might be explained by the global distribution of Mendeley readers: as shown in  The correlation between field-weighted citation impact and field-weighted readership impact (thereby removing the any size effects, as well as any specialisation effects) is lower but still strong as demonstrated in Figure 3, and the same for countries, regardless of their publication output. Interestingly, several prolific countries (Brazil, India, Japan, Russia) that have relatively fewer publications read than cited, compared to the trend line, have a relatively higher than expected FWRI in relation to their FWCI. Conversely, China and several prolific established research nations have a relatively lower than expected FWRI in relation to their FWCI.  Figure 4 shows that for the world and for the most prolific countries, overall and per subject area, the FWCI and FWRI values tend to be close to the trend line, with a very slight FWRI advantage. Nevertheless, variations per country observed overall tend to hold true per OECD field. For instance, China's FWCI advantage holds true for each of the six OECD subject categories. Looking at absolute comparisons of FWCI vs FWRI as in Figure 5 however reveals different patterns. While global values are close to 1 overall and for most subject areas, in most of fields except for the Humanities, there is a very slight FWRI advantage. The FWRI is however notably higher than the FWCI in Agricultural Sciences and especially in the Social Sciences, in which the FWRI is 20% higher than the FWCI. There are also differences per country: in absolute terms, FWRI values tend to be higher than FWCI values across fields for prolific established western nations like the USA, the UK, Germany, and France, while the reverse is true for prolific Asian countries like China, Japan, or India.

Conclusions
Our analyses show there is a strong correlation between number of papers cited and read per country, which doesn't appear to be a function of size. Specific regional patterns may be influenced by the geographic distribution of Mendeley readers, data which was not available at the time of writing this paper, but that could be used in future studies. The correlation between field-weighted citation impact and field-weighted readership impact, is lower but still strong. Overall and per subject area, and for most prolific countries, FWCI and FWRI values tend to be close. Variations per country observed overall tend to hold true per field. FWRI appears as a robust metrics that can offer a useful complement to FWCI, in that it provides insights on a different part of the scholarly communications cycle. More detailed analyses are welcome to further test the metrics at different aggregation levels. It would also be interesting to see how it compares to other indicator types (e.g. downloads, views, altmetrics).