Applying the Leiden Manifesto Principles in Practice – Commonalities and Differences in Interpretation
The Leiden Manifesto (LM) is changing how we think about and use metrics [1]. Bibliometric evaluation is explained as a combination of quantitative and qualitative methods, allowing the use of different metrics, disciplinary knowledge and research performance strategies. Both bibliometricians and consumers of bibliometrics are encouraged to communicate and use the LM principles to acknowledge what they know and do not know, what is measured and what is not measured, thus legitimizing the use of the metrics.
However, in our previous study, we observed that it is unclear how the LM principles should be interpreted [2, 3]. We suspect that subjective interpretations of the principles do not correlate. To investigate the reliability and validity of the LM, the present study presents a systematic review of bibliometric reports that apply the LM principles. Reports are retrieved from the LM blog [4], Scopus, Web of Science and Google Scholar. Each principle and its interpretation is coded in NVivo, whereafter we explore the degree of agreement in the interpretations across the reports.
We find that for some principles, e.g. principle 1, the interpretations are well aligned. For other principles, e.g. principle 3, the interpretations differ but may be seen as complementary. We also observe that interpretations can overlap and thus the redundancy of the principles needs to be further investigated, e.g. principle 3 and 6.
We conclude that at least for some of the LM principles, the reliability appears weak as the range of interpretations are wide, however complementary. Furthermore, some of the interpretations are applied for more principles, which may point to weak validity.
Further research on the reliability and the validity of the LM will be essential to establish guidance in implementing the LM in practice.
