Practical Guide for Research Evaluators

• July 23, 2024

In the quest to foster innovation and excellence in research, the evaluation process for funding proposals plays a pivotal role. Recognizing the need for a more comprehensive approach, DORA (Declaration on Research Assessment) has introduced “Balanced, Broad, Responsible: A Practical Guide for Research Evaluators.” This initiative comprises a short, informative video and a one-page brief designed to serve as invaluable resources for both public and private research funders.

The Need for a New Approach

The guide underscores the importance of research assessment in shaping the research culture. Traditionally, there has been an overreliance on quantitative metrics, which can limit evaluators’ ability to fully appreciate the breadth and impact of an applicant’s work. Just as research itself is dynamic and evolving, the methods used to assess research should also be subject to continuous improvement and innovation. This shift is essential to ensure that evaluations are holistic and reflective of the diverse contributions researchers make to their fields.

Practical Suggestions for Holistic Evaluation

To aid funders in adopting more responsible assessment practices, the guide provides a practical checklist with six key suggestions:

1. Align Decision-Making: Ensure that decisions align with the strategic objectives and specific criteria of the funding institution or instrument.
2. Balance Metrics: Clearly communicate the context and limitations of any quantitative metrics used and balance them with qualitative assessments of the funding proposal.
3. Broaden the Scope: Look beyond traditional metrics such as publications and grants to capture the full range of a researcher’s contributions, including activities like mentoring, community engagement, and collaborative projects.
4. Address Biases: Be mindful of biases that can arise from scientific and cultural stereotypes, striving to create a fair and inclusive evaluation process.
5. Promote Accountability: Encourage personal and group accountability for responsible research assessment throughout the evaluation process.
6. Manage Conflicts of Interest: If there is any uncertainty about potential conflicts of interest, seek guidance from the funding institution to ensure transparency and impartiality.

Implementing the Guide

The one-page brief offers practical suggestions for various contexts, including sharing the video and brief with grant evaluators, institutions, and researchers. By disseminating these resources widely, funders can promote a more balanced and responsible approach to research evaluation.

Collaborative Development

These resources are the result of a partnership between DORA and the Luxembourg National Research Fund (FNR). The development process included iterative feedback from the DORA Steering Committee, the DORA Funder Discussion Group, active researchers, and other funding agencies. This collaborative effort has ensured that the checklist is both practical and grounded in the real-world experiences of those involved in research funding and assessment.

To sum up, “Balanced, Broad, Responsible: A Practical Guide for Research Evaluators” represents a significant step towards improving how research proposals are evaluated. By embracing a more holistic approach, funders can better recognize and support the full spectrum of contributions that researchers make to their fields and society at large.

More at DORA