The experimental research funder’s handbook (RoRI Working Paper No.6)

Across the international research funding community, there is a growing appetite for more sophisticated approaches to evidence gathering and experimentation, to inform evaluation, allocation and decision-making.

Different research funders are at different stages of this journey, and don’t always benefit as much as they might from the insights and experiences of others, or from the latest academic studies in this field. This Handbook aims to provide a practical resource for funders looking to move further or faster down the experimental path.

Here we have collated and synthesised insights from funder partners in the Research on Research Institute (RoRI) consortium that have conducted trials with aspects of peer review and evaluation.

From these accounts, we have assembled practical descriptions of designing, implementing, and evaluating changes to peer review processes. By supplementing these with more foundational information about experimental processes and theories of peer review, we aim to present a comprehensive description of end-to-end processes that might be used by funders considering similar experiments in future.

Detailed descriptions of the implementation of such processes and of their effects have been largely hidden from those outside of the organisations carrying out the experiments. By providing these here, thanks to direct input from those funders, we highlight a variety of approaches that funders may take, challenges they have experienced and lessons learned.

In line with RoRI’s goal to achieve more efficient, dynamic, diverse and inclusive research systems through rigorous evidence and experimentation, our hope is that this Handbook will support funders to design effective interventions that can be properly evaluated, and yield robust findings of relevance to others.

This is an initial working paper version of the Handbook, for discussion at a workshop in December 2021, after which we will update it for final publication in spring 2022. We warmly invite comments at hello@researchonresearch.org, and look forward to seeing a growing diversity of experiments with research funding in the months and years to come.