OPUS Research Assessment Framework (RAF): Changing the Way We Evaluate Research

• September 12, 2023

The OPUS team has been hard at work for months, creating the first version of the OPUS Research Assessment Framework (RAF). This innovative tool is set to change the way we evaluate research. The framework provides a range of indicators and metrics specifically designed for research organizations and funding bodies to assess researchers for career advancement and grant applications. By using the OPUS RAF, research organizations can effectively evaluate the quality and impact of research while promoting the principles of Open Science. It’s important to note that the RAF also includes a focus on Open Science practices, providing recognition and incentives for these important aspects.

As part of the OPUS project, two key documents have been finalized to drive this initiative forward. Deliverable D.2.1 outlines interventions to test in pilot programs, focusing on rewards and incentives for researchers. Deliverable D.3.1 presents indicators to test in these pilot programs, covering metrics for Open Science practices.

The development of the RAF takes inspiration from influential policies and existing frameworks like the San Francisco Declaration on Research Assessment (DORA), Leiden Manifesto for Research Metrics, Hong Kong Principles, and Recommendations by the Open Science Policy Platform (OSSP), among others. By incorporating elements from these important documents, the RAF aims to create a comprehensive assessment tool that combines both quantitative and qualitative approaches, covers research and non-research activities, and meets the unique needs of research organizations and funding bodies.

Throughout the development process, key stakeholders in research assessment and Open Science have played a crucial role. Collaboration with partners within the OPUS project, pilot organizations, the Advisory Board, and external stakeholders has helped shape the RAF. Input from pilot initiatives, targeted stakeholder feedback, and an open consultation with the wider research community will guide future improvements to the RAF.

The RAF offers a comprehensive set of indicators and metrics to assist in researcher assessment within research organizations and funding bodies. These activities are categorized into research, education, leadership, and valorization, allowing organizations to customize their assessment systems to their specific needs. The framework covers a broad range of activities and includes a dedicated focus on Open Science to ensure that both types of activities receive appropriate recognition and rewards.

Important Deliverables

Two separate deliverables under the OPUS project, Work Package 2 (WP2) – Interventions for Open Science (Rewards and Incentives for Researchers), and Work Package 3 (WP3) – Indicators and Metrics for Open Science (Rewards and Incentives for Researchers), have been finalized:

• D.2.1 Interventions to Test in the Pilots to support the implementation of the RAF, Deliverable D2.1 outlines various interventions available for testing during the pilot phase of the project. These interventions provide practical support for research organizations and funding bodies and can be used alongside the RAF framework. The indicators and metrics from the RAF will undergo rigorous testing in selected pilot organizations to ensure their effectiveness.
• D.3.1 Indicators to Test in the Pilots: Deliverable D3.1 provides in-depth insights into the guiding principles, structure, and implementation of the RAF. It features generic indicators and metrics in Section 3 and Open Science indicators and metrics in Section 4. The report concludes with a discussion on future steps and the role of the OPUS project. For a detailed overview, including all indicators and metrics, you can refer to Appendix 1, which presents the full RAF.

What’s Next?

With the finalization of these two critical deliverables, the OPUS project has achieved a significant milestone in research evaluation. To enhance its effectiveness, the developers will conduct comprehensive testing and gather feedback over the next eighteen months. Valuable feedback on potential areas for improvement, as well as the appropriateness of chosen categories and indicators, will refine the framework.

Pilot institutions will play a crucial role in thoroughly testing the framework and providing valuable feedback on both the indicators and interventions. This iterative process ensures that the OPUS RAF becomes a practical and user-friendly tool for European research organizations and funding bodies.

A sector-wide consultation will engage key organizations, projects, and sector partners, gathering insights and perspectives from various stakeholders in the research community. By involving diverse voices, the final version of the OPUS framework will be more comprehensive and relevant.

As the OPUS RAF undergoes further refinement and collaboration, it holds the promise of shaping research evaluation practices in a profound way. The project’s dedication to continuous improvement and engagement with stakeholders ensures that the final version of the OPUS RAF will meet the needs of the research community, driving positive change and progress.

Stay tuned for updates as the OPUS project takes research assessment into a new era of transformation. We will provide detailed insights into each deliverable, giving you a comprehensive understanding of the progress made and the significance of these developments. Keep an eye out for more information on each deliverable in the coming days.