OPUS Research Assessment Framework (RAF): Revolutionizing Research Evaluation

• September 13, 2023

The OPUS team has dedicated months of work to develop the first version of the OPUS Research Assessment Framework (RAF). This cutting-edge framework offers a comprehensive suite of indicators and metrics, designed specifically for research-performing organizations (RPOs) and research-funding organizations (RFOs), to assess researchers for career progression and grant applications. By implementing the OPUS RAF, RPOs and RFOs can now effectively evaluate the impact and quality of research while promoting the principles of Open Science.

The RAF incorporates an Open Science dimension, providing explicit recognition and incentives for Open Science practices. It comprises a carefully crafted set of indicators and interventions aimed at fostering Open Science practices within research organisations: RPOs and RFOs. As part of the OPUS project Work Package 2  (WP2) – Interventions for Open Science (Rewards and Incentives for Researchers) and Work Package 3 (WP3) – Indicators and Metrics for Open Science (Rewards and incentives for Researchers), two separate deliverables have been finalized to propel this transformative initiative forward:

• D.2.1 Interventions to Test in the Pilots
• D.3.1 Indicators to Test in the Pilots

The developers of the RAF have outlined ten guiding principles that will shape its implementation, ensuring a transformative impact on research evaluation practices across geographical boundaries and disciplinary constraints. Unlike traditional approaches, the RAF focuses on assessing individual researchers rather than teams or groups and combines both quantitative and qualitative assessment methods.

The RAF itself offers a comprehensive array of interventions, indicators and metrics to facilitate researcher assessment in RPOs and RFOs, covering a wide range of activities undertaken by researchers. These activities are categorized into research, education, leadership, and valorization, allowing organizations to tailor their assessment systems to their specific requirements. The framework encompasses two dimensions: a generic approach and a dedicated Open Science focus, ensuring the appropriate recognition and reward of both types of activities.

The OPUS Project Revolutionizes Research Evaluation with Groundbreaking Research Assessment Framework (RAF) and Innovative Interventions (Deliverable D2.1: Pilots Put to the Test)

The OPUS project marks a significant milestone in research evaluation, promising to transform the research landscape. By offering a robust framework and practical interventions, the project empowers RPOs and RFOs to embrace Open Science practices, fostering a culture of transparency, collaboration, and progress. 

To support the implementation of the RAF, Deliverable D2.1 presents a range of interventions to be tested during the project’s pilot phase. These interventions, outlined in detail, offer practical support for RPOs and RFOs and can be used along with the RAF framework. Through pilot initiatives at three RPOs and two RFOs within the OPUS project, selected interventions, indicators and metrics from the RAF will undergo testing to ensure their effectiveness and suitability.

The interventions within the OPUS RAF are structured based on the dimensions of the main framework, which consists of five assessment categories with subcategories. This structured approach enables a clear and cohesive implementation of the RAF. 

To ensure the successful implementation of the RAF, five categories of interventions have been identified: Policy, Resource, Repository, Awareness Raising, and Training. These interventions emphasize the need for senior management support, clear strategies, procedural guidelines, adequate resources, easily accessible repositories, education for researchers on data collection, and training initiatives to equip researchers with necessary skills.

A Holistic Approach to Research Evaluation with the OPUS Research Assessment Framework (RAF) Indicators and Metrics (Deliverable D3.1: Putting Indicators to the Test)

Deliverable D3.1 provides in-depth insights into the guiding principles, structure, and implementation of the RAF, featuring generic indicators and metrics in Section 3 and Open Science indicators and metrics in Section 4. The report concludes with a discussion on future steps and the role of the OPUS project. For a detailed overview, including all generic and Open Science indicators and metrics, Appendix 1 presents the full RAF.

A notable strength of the OPUS RAF is its ability to encompass the entire spectrum of researchers’ activities, extending beyond research-related endeavors. The framework accommodates both open and non-open activities undertaken by researchers, emphasizing the importance of transparency and collaboration in the research landscape. It even includes a dedicated framework for evaluating Open Science activities, reflecting the growing significance of openness and collaboration in the scientific community.

To capture the comprehensive nature of researchers’ activities, the OPUS RAF incorporates three types of indicators: process, output, and outcome indicators. Process indicators track ongoing or developing activities, while output indicators measure tangible products or clear endpoints resulting from those activities. Finally, outcome indicators assess the immediate or short-term results generated by the outputs.

Figure 1 showcases the categories, subcategories, and indicator groups that form the Researcher Assessment Framework.

Each indicator group within the framework is accompanied by quantitative metrics and comprises three types of indicators. Process indicators track activities in progress, output indicators measure concrete outcomes, and outcome indicators provide insight into immediate or short-term results stemming from outputs.

The OPUS RAF empowers RPOs and RFOs to select, refine, and prioritize indicators and metrics based on their specific contexts and goals. By adhering to the guiding principles and utilizing the structured interventions, organizations can confidently adopt the OPUS RAF, leading to a more comprehensive and accurate assessment of researchers.

A Collaborative Journey Towards RAF Enhancement Persists

While the OPUS Research Assessment Framework (RAF) is ready for initial implementation, it is important to note that it is still in its early draft stage. 

The development of the RAF draws inspiration from influential policy developments and existing frameworks such as the San Francisco Declaration on Research Assessment (DORA), Leiden Manifesto for Research Metrics, Hong Kong Principles, and Recommendations by the Open Science Policy Platform (OSSP), among others. By incorporating elements from these important documents, the RAF aims to create a comprehensive assessment tool that combines both quantitative and qualitative approaches, encompasses research and non-research activities, and caters to the unique needs of RPOs and RFOs.

Throughout the development process, key stakeholders in research assessment and Open Science have played an integral role. Partners within the OPUS project, pilot organizations, and the Advisory Board have provided valuable insights and expertise. Collaboration with external stakeholders, including the GraspOS project and other relevant entities, has further contributed to shaping the RAF. Additionally, input from pilot initiatives, targeted stakeholder feedback, and an open consultation with the wider research community will inform future iterations and enhancements of the RAF.

The OPUS team members recognize the need for further refinement to enhance its effectiveness. Over the next eighteen months, pilot institutions will have the opportunity to thoroughly test the framework and provide valuable feedback on both the indicators and interventions.

Simultaneously, a comprehensive sector-wide consultation will be conducted, involving key organizations, projects, and sector partners. This ongoing collaboration will ensure that the final output of the OPUS framework becomes a practical and user-friendly tool for European RPOs and RFOs.

During the development process, certain points for consideration and improvement have emerged. The framework aims to be comprehensive yet customizable, allowing organizations to prioritize specific areas while still covering all essential aspects. Feedback on whether any important areas have been overlooked and whether the chosen categories and indicator groups are appropriate will be crucial.

As the implementation of the OPUS RAF commences, it holds the potential to transform research evaluation practices, promoting a culture of excellence, accountability, and transparency. Institutions embracing this groundbreaking framework will drive positive change in the research community.