OPUS helps reform the assessment of research

Interventions for Open Science/Researcher Assessment

OPUS has developed a set of interventions for Open Science toward a system that incentivises and rewards researchers to take up practices of providing open access to research outputs, early and open sharing of research, participation in open peer-review, measures to ensure reproducibility of results, and involving all stakeholders in co-creation.

The RAF is designed to comprehensively cover the full spectrum of researcher activities and offer researchers the possibility to be assessed on all relevant activities in a given assessment period, rather than solely on the traditional metrics of the number of peer-reviewed publications and citations and the journal impact factor (JIF). The inclusion of Open Science metrics in a given RAF at an RPO or RFO ensures that Open Science activities are explicitly recognised in the assessment. It remains the prerogative of RPOs and RFOs to determine how exactly to reward researchers for Open Science.

1. Generic Researcher Assessment Framework: Indicators and Metrics

The RAF is structured around 4 main categories of activities, which are divided into subcategories of these activities and consist of indicator groups of related indicators/metrics.

1.1 Interventions

The interventions are divided by the indicator that they support and then are subdivided into the following categories.

Each indicator group further consists of 3 types of indicators with associated quantitative metrics:

  • Process: Activity which is in development or is ongoing
  • Output: Clear endpoint or tangible product of a process
  • Outcome: Immediate or short-term result of an output

They are supported by five categories of interventions:

2. Meta-Interventions

Research Assessment Framework and Open Science Career Assessment Matrix

Meta Interventions: This set of meta-interventions has been developed as an overarching set of interventions for any RPO or RFO that wishes to implement the RAF or OSCAM Framework. They are intended as a starting point for consideration and implementation at a the top level of an organisation.

Research Assessment Framework Open Science Career Assessment Matrix
Senior management decision and approval to use the framework at their institution.

Senior management decision on the set of indicators they wish to use in their institution.

Senior management develop policies and procedures that support the collection of metrics and data relevant to the indicators with clear guidelines and expectations.

Senior management approval and decision, involving all relevant parties to collect and make open metrics and data that are specific to the selected indicator.

Senior management decision on which indicators they wish to use in their institution.

Senior management develop policies and procedures that support collection of metrics and data relevant to the specific indicators and advocate making open access with clear guidelines and explanation.

 

Research Assessment Framework Open Science Career Assessment Matrix
Allow researchers time (and supporting budget) to record the relevant metrics and upload the relevant evidence for the selected indicators.

Ensure there is a member (or members) of staff responsible for monitoring, assisting and understanding any ethical queries that arise form collection of data relevant to the selected indicators.

Provide expertise and support relevant to the topic of the selected indicators.

Allow researchers time (and supporting budget) to record, upload and make open metrics and data relevant to the selected indicators.

Ensure there is a member (or members) of staff responsible for monitoring upload of metrics and data relevant to the selected indicator are open with an understanding of any ethical issues this may create.

Provide expertise and support relevant to the topic of the selected indicators and how to make the topic open.

 

Research Assessment Framework Open Science Career Assessment Matrix
Ensure there is an appropriate and easily accessible database or repository to record the metrics and data relevant to the selected indicators. Ensure there is a suitable and easily accessible database or repository to record data and metrics relevant to the selected indicators and that this is open access.
Research Assessment Framework Open Science Career Assessment Matrix
Ensure researchers are aware of what they should be recording for the selected indicators and where they can get training in how to do this and who they should go to for help.

Ensure researchers understand the benefit to them as individuals and the institution of recording the metrics and data relevant to the indicators and that they understand the link to researcher assessment.

Ensure researchers know that they should record and ensure open access for all documents relevant to the selected indicators and where they can receive training in how to do this and who they should go to for help.

Explain to researchers the benefit of making the selected indicators open to them as individuals and the institution and that they understand the link to researcher assessment.

Research Assessment Framework Open Science Career Assessment Matrix
Train researchers in where and how to log metrics and data relevant to the selected indicators.

Train researchers in the skills relevant to the topic of the selected indicators

Ensure there are a range of best practice proposal examples that researchers can access and draw upon relevant to the selected indicators.

 

Train researchers in where and how to log and ensure open access documents relevant to selected indicators.

Train researchers and develop skills relevant to specific indicators.

Train researchers in how to make documents relevant to specific indicators open access including issues specific to academic discipline and any ethical concerns.

Ensure there are best practice open access examples relevant to the specific indicators selected.

Research

The research category consists of 6 subcategories for proposals, methods, data, software, publications, and materials with associated generic indicators/metrics for researcher assessment.

Generic Interventions for Research – Subcategory Proposals
Indicator Type Quantitative Metric
Process # of Project Proposals Being Developed
Output # of Project Proposals Submitted
Outcome # of Project Proposals Granted
Category RAF Proposal Development Interventions
Policy Senior management approval and decision to collect the number of project proposals being developed, submitted, and granted across the institution.

Senior management develop policies and procedures to support the collection of the number of proposals being developed, submitted and granted across the institution including clear guidelines and explanations.

Resources Allow researchers time (and supporting budget) to record the number of project proposals being developed, submitted and granted and upload the relevant evidence.

Ensure there is a member (or members) of staff responsible for monitoring, assisting and understanding any ethical queries that arise from the collection of the number of project proposals being developed, submitted and granted.

Provide expertise and support in proposal writing and development

Repository Ensure there is an appropriate and easily accessible database or repository to record the number of project proposals being developed, submitted and granted.
Awareness Raising Ensure researchers know that they should record all project proposals being developed, submitted, and granted and where they can get training in how to do this and who they should go to for help.

Ensure researchers understand the benefit to them as individuals and the institution of recording the number of project proposals being developed, submitted, and granted, that they have trust in the process and that there is a clear link to researcher assessment.

Training Train researchers in where and how to log project proposals being developed, submitted, and granted.

Train researchers in proposal writing and development.

Ensure there are a range of best practice proposal examples that researchers can access and draw upon.

Generic Interventions for Research – Subcategory Methods

Indicator Type Indicator
Process #Method Sets Being Developed
Output # Method Sets Finalised
Outcome # Method Sets Implemented
Outcome # Method Sets Accessed

# Method Sets Cited

Category RAF Methods Development Interventions
Policy Senior management approval and decision to collect number of methods sets being developed, finalised and implemented across the institution.

Senior management approval and decision to collect the number of method sets accessed and cited across the institution.

Senior management develop policies and procedures to support the collection of method sets being developed, finalised and implemented across the institution including clear guidelines and explanations.

Senior management develop policies and procedures to support the collection of method sets access and cited across the institution including clear guidelines and explanations.

Resources Allow researchers time (and supporting budget) to record the number of methods sets, being developed, finalised and implemented across the institution and upload the relevant evidence.

Allow researchers time (and supporting budget) to record the number of methods sets accessed and cited across the institution and upload the relevant evidence.

Ensure there is a member (or members) of staff responsible for monitoring, assisting, and understanding any ethical queries with the recording of the number of method sets being developed and finalised.

Ensure there is a member (or members) of staff responsible for monitoring, assisting and understanding any ethical queries with the recording of the number of methods sets accessed and cited across the institution.

Provide expertise and support in development and management of method sets.

Repository Ensure there is a suitable and easily accessible database or repository to record method sets being developed, finalised and implemented across the institution.

Ensure there is an appropriate and easily accessible database or repository to record method sets accessed and cited across the institution.

Awareness Raising Ensure researchers know that they should record all method sets, being developed, finalised, and implemented and where they can receive training in how to do this and who they should go to for help.

Ensure researchers know that they should record all method sets accessed and cited and where they can receive training in how to do this and who they should go to for help.

Ensure researchers understand the benefit of recording the number of methods sets being developed and utilised to them as individuals and the institution, that they trust the process and that there is a clear link to research assessment.

Training Train researchers in where and how to log the number of methods sets developed and finalised across the institution.

Train researchers in where and how to log the number of methods sets accessed and cited across the institution.

Train researchers in the development of method sets.

Ensure there are a range of best practice examples of method sets that researchers can access and draw upon.

Generic Interventions for Research – Subcategory Data

Data Planning
Indicator Type Indicator
Process #(FAIR) Data Management Plans Being Developed
Output # (FAIR) Data Management Plans Finalised
Outcome # (FAIR) Data Management Plans Implemented
Category RAF Data Planning Interventions
Policy Senior management approval and decision to collect the number of data management plans being developed, finalised, and implemented across the institution.

Senior management develop policies and procedures to support the collection of the number of data management plans being developed, finalised and implemented including clear guidelines and explanations.

Resource Allow researchers time (and supporting budget) to record the number of data management plans being developed, finalised and implemented across the institution and upload the relevant evidence for the selected indicators.

Ensure there is a member (or members) of staff responsible for monitoring, assisting and understanding any ethical queries with that arise from monitoring the number of data management plans developed, finalised and implemented across the institution.

Provide expertise and support in the development of quality data management plans.

Repository Ensure there is an appropriate and easily accessible database or repository to record the number of data management plans being developed, finalised and implemented across the institution.
Awareness Raising Ensure researchers know that they should record the number of data management plans being developed, finalised, and implemented and where they can receive training in how to do this and who they should go to for help.

Ensure researchers understand the benefit of them as individuals and the institution of recording the number of data management plan, that they trust the process and that there is a link to researcher assessment.

Training Train researchers in where and how to record the number of data management plans being developed, finalised, and implemented.

Train researchers in development of quality data management plans.

Ensure there are best practice examples of data management plans that researchers can access and draw upon.

Data Management
Indicator Type Indicator
Process # (FAIR) Data Sets Being Developed
Output # (FAIR) Data Sets Finalised
# (FAIR) Data Sets Archived Outcome
Outcome # (FAIR) Data Sets Accessed
# (FAIR) Data Sets Cited
Category RAF Data Management Interventions
Policy Senior management approval and decision to collect the number of FAIR data management plans being developed, finalised and archived across the institution.

Senior management approval and decision to collect the number of FAIR data management plans accessed and cited across the institution.

Senior management develop policies and procedures to support the collection of the number of FAIR data management plans being developed, finalised and archived across the institution with clear guidelines and expectations.

Senior management develop policies and procedures to support the collection of the number of FAIR data management plans accessed and cited across the institution with clear guidelines and expectations.

Resource Allow researchers time (and supporting budget) to record the number of FAIR data management plans being developed, finalised, and archived across the institution.

Allow researchers time (and supporting budget) to record the number of FAIR data management plans accessed and cited across the institution.

Ensure there is a member (or members) of staff responsible for monitoring, assisting with and understanding any ethical queries that arise for the number of data management plans being developed, finalised, and archived across the institution.

Ensure there is a member (or members) of staff responsible for monitoring, assisting with and understanding any ethical queries that arise for the number of FAIR data management plans accessed and cited across the institution.

Provide expertise and support in the development of FAIR data management plans.

Repository Ensure there is a suitable and easily accessible database or repository to record the number of FAIR data management plans being developed, finalised and archived across the institution.

Ensure there is an suitable and easily accessible database or repository to record the number of FAIR data management plans accessed and cited across the institution.

Awareness Raising Ensure researchers know that they should record the number of FAIR data management plans being developed, finalised, and archived and where they can receive training in how to do this and who they should go to for help.

Ensure researchers know that they should record the number of FAIR data management plans accessed and cited and where they can receive training in how to do this and who they should go to for help.

Ensure researchers understand the benefit of recording the number of FAIR data plans in these ways and that they understand the benefit to them as individuals and the institution, that the trust the process and the link to researcher assessment.

Training Train researchers in where and how to record the number of FAIR data management plans being developed, finalised, and implemented.

Train researchers in where and how to record the number of FAIR data management plans developed accessed and cited.

Train researchers in the development of FAIR data management plans.

Ensure there are a range of best practice examples of FAIR data management plans that researchers can access and draw upon.

 

Data Review
Indicator Type Indicator
Process # (FAIR) Data Set Peer Reviews Being Drafted
Output # (FAIR) Data Set Peer Reviews Submitted
Outcome # (FAIR) Data Set Peer Reviews Accepted
Category RAF Data Review Interventions
Policy Senior management approval and decision to collect the number of peer reviews being drafted, submitted, and accepted across the institution.

Senior management develop policies and procedures to support the collection of the number of peer reviews being drafted, submitted and accepted across the institution, with clear guidelines and expectations.

Resource Allow researchers time (and supporting budget) to record the number of peer reviews being drafted submitted and accepted across the institution.

Ensure there is a member (or members) of staff responsible for monitoring, assisting with and understanding any ethical queries for collecting the number of peer reviews being drafted, submitted and accepted.

Provide expertise and support in the development of peer reviews

Repository Ensure there is an appropriate and easily accessible database or repository to record the number of peer reviews being drafted, submitted and accepted.
Awareness Raising Ensure researchers know that they should record all peer reviews being drafted, submitted, and accepted and where they can receive training in how to do this and who they should go to for help.

Ensure researchers understand the benefit to them as individuals and to the institution of collecting the number of peer reviews being drafted, submitted and accepted that the trust the process and understand the link to researcher assessment.

Training Train researchers in where and how to record peer reviews being drafted, submitted, and accepted.

Train researchers in the peer review process.

Ensure there are best practice examples of peer reviews that researchers can access and draw upon.

Generic Interventions for Research – Subcategory Software
Indicator Type Indicator
Process # Software Sets Being Developed
Output # Software Sets Finalised
# Software Sets Archived
Outcome # Software Sets Accessed
# Software Sets Cited
Category RAF software Development Interventions
Policy Senior management approval and decision to collect the number of software sets being developed, finalised, and archived across the institution.

Senior management approval and decision to collect the number of software sets accessed and cited across the institution.

Senior management develop policies and procedures to support the collection of the number of software sets being developed, finalised and archived across the institution with clear guidelines and explanation.

Senior management develop policies and procedures to support the collection of the number of software sets accessed and cited across the institution with clear guidelines and explanation.

Resource Allow researchers time (and supporting budget) to record and upload the relevant evidence for the number of software sets being developed, finalised, and archived across the institution.

Allow researchers time (and supporting budget) to record and upload the relevant evidence for the number of software sets accessed and cited across the institution.

Ensure there is a member (or members) of staff responsible for monitoring, assisting with and understanding any ethical queries for recording the number of software sets developing, finalised and archived.

Ensure there is a member (or members) of staff responsible for monitoring, assisting with and understand any ethical queries with recording the number of software sets accessed and cited.

Provide expertise and support in the development of software sets.

Repository Ensure there is an appropriate database or repository to record the number of software sets being developed, finalised and archived across the institution.

Ensure that there is an appropriate database to record the number of software sets accessed and cited across the institution.

Awareness Raising Ensure researchers know that they should record all software sets being developed, finalised, and implemented and where they can receive training in how to do this and who they should go to for help and support.

Ensure researchers know that that they should record all software sets accessed and cited and where they can receive training in how to do this and who they should go to for help and support.

Ensure researchers understand the benefit to them as individuals and to the institution of recording the number of software sets utilised, that the trust the process and understand the link to researcher assessment.

Training Train researchers in where and how to record software sets being developed, finalised, and implemented.

Train researchers in where and how to record software sets cited and accessed.

Train researchers in development of software sets.

Ensure there are best practice examples of software sets that researchers can access and draw upon.

 

Software Review
Indicator Type Indicator
Process # Software Set Peer Reviews Being Drafted
Output # Software Set Peer Reviews Submitted
Outcome # Software Set Peer Reviews Accepted
Category RAF Software Review Interventions
Policy Senior management approval and decision to collect the number of software set peer reviews being drafted, submitted and accepted.

Senior management develop policies and procedures to support the collection of software set peer reviews that are being drafted, submitted and accepted including guidelines and explanation

Resource Allow researchers time (and supporting budget) to record the number of software set peer reviews being drafted, submitted, and accepted across the institution and upload relevant supporting evidence.

Ensure there is a member (or members) of staff responsible for monitoring, assisting with and understanding any ethical queries for the recording the number of software set peer reviews being drafted, submitted and accepted.

Provide expertise and support in the development of software set peer reviews

Repository Ensure there is an appropriate and easily accessible database or repository to record the number of software peer reviews being drafted, submitted and accepted.
Awareness Raising Ensure researchers know that they should record all software set peer reviews being drafted, submitted, and accepted and where they can receive training in how to do this and who they should go to for help.

Ensure researchers understand the benefit of recording the number of software sets peer reviews being drafted, submitted and accepted to them as individuals and the institution, that they trust the process and understand the link to researcher assessment.

Training Train researchers in where and how to record software set peer reviews being drafted, submitted, and accepted.

Train researchers in the software set peer review process.

Ensure there are best practice examples of software peer reviews that researchers can access and draw upon.

Generic Interventions for Research – Subcategory Publications
Indicator Type Indicator
Process # Publications Being Drafted
Output # Publications Submitted
Outcome # Publications Published
# Publications Accessed
# Publications Cited
Category Research Assessment Framework
Policy Senior management approval and decision to collect the number of publications being drafted, submitted, and published across the institution.

Senior management approval and decision to collect the number of publications accessed and cited across the institution.

Senior management develop policies and procedures to support the collection of the number of publications being drafted submitted and published across the institution with clear guidelines and explanation.

Senior management develop policies and procedures to support the collection of the number of publications accessed and cited across the institution with clear guidelines and explanation.

Resource Allow researchers time (and supporting budget) to record the number of publications being drafted, submitted, and published across the institution and upload the relevant evidence.

Allow researchers time (and supporting budget) to record the number of publications, accessed and cited across the institution and upload the relevant evidence.

Ensure there is a member (or members) of staff responsible for monitoring, assisting with and understanding any ethical queries for collecting the number of publications being drafted, submitted, and published across the institution.

Ensure there is a member (or members) of staff responsible for monitoring, assisting with and understanding any ethical queries with collecting the number of publications accessed and cited across the institution.

Provide expertise and support in quality publications.

Repository Ensure there is an appropriate and easily accessible database or repository to record the number of publications being drafted, submitted, and published across the institution.

Ensure that there is an appropriate and easily accessible database or repository to record the number of publications accessed and cited across the institution.

Awareness Raising Ensure researchers know that they should record all publications being drafted, submitted, and published and where they can receive training in how to do this and who they should go to for help and support.

Ensure researchers know that that they should record all publications accessed and cited and where they can receive training in how to do this and who they should go to for help and support.

Ensure researchers understand the benefit of recording the number of publications to both individuals and the institution, that they trust the process and understand the link to researcher assessment.

Training Train researchers in where and how to record publications being drafted, finalised, and implemented.

Train researchers in where and how to record publications cited and accessed.

Train researchers in how to publish.

Ensure that there are best practice examples of a range of quality publications that researchers can access and draw upon.

Publication Review
Indicator Type Indicator
Process # Publication Peer Reviews Being Drafted
Output # Publication Peer Reviews Submitted
Outcome # Publication Peer Reviews Accepted
Category RAF Publication Review Interventions
Policy Senior management approval and decision to collect the number of publication peer reviews being drafted, submitted, and accepted.

Senior management develop policies and procedures to support the collection of the number of publication peer reviews being drafted, submitted, and accepted, with clear guidelines and expectations.

Resource Allow researchers time (and supporting budget) to record the number of publication peer reviews being drafted, submitted, and accepted and to upload the relevant evidence.

Ensure there is a member (or members) of staff responsible for recording, assisting with and understanding any ethical queries for recording the number of publication peer reviews being drafted, submitted, and accepted.

Provide expertise and support in the development of publication peer reviews.

Repository Ensure there is an appropriate and easily accessible database or repository to record the number of publication peer reviews being drafted submitted and accepted.
Awareness Raising Ensure researchers know that they should record all publication peer reviews being drafted, submitted, and accepted and where they can get training in how to do this and who they should go to for help.

Ensure researchers understand the benefit of recording publication peer reviews to them as individuals and the institution, that they trust the process and understand the link to researcher assessment.

Training Train researchers in where and how to record publication peer reviews being drafted, submitted, and accepted.

Train researchers in the publication peer review process.

Ensure there are best practice examples of publication peer reviews that researchers can access and draw upon.

Generic Interventions for Research – Subcategory Materials
Indicator Type Indicator
Process # Material Sets Being Developed
Output # Material Sets Finalised
Outcome # Material Sets Implemented
# Material Sets Accessed
# Material Sets Cited
Category Research Assessment Framework Interventions
Policy Senior management approval and decision to collect the number of materials being developed, finalised, and implemented across the institution.

Senior management approval and decision to collect the number of materials accessed and cited across the institution.

Senior management develop policies and procedures to support the collection of the number of materials being developed, finalised, and implemented across the institution including clear guidelines and explanation.

Senior management develop policies and procedures to support the collection of the number of materials accessed and cited across the institution including clear guidelines and explanation.

Resource Allow researchers time (and supporting budget) to record the number of materials being developed, finalised, and implemented across the institution and upload the relevant evidence.

Allow researchers time (and supporting budget) to record the number of materials being developed, accessed, and cited across the institution and upload the relevant evidence.

Ensure there is a member (or members) of staff responsible for monitoring, assisting with and understanding any ethical queries for recording the number of materials developing, finalised and implemented across the institution.

Ensure there is a member (or members) of staff responsible for monitoring, assisting with and understanding any ethical queries for recording the number of materials accessed and cited across the institution.

Provide expertise and support in materials development.

Repository Ensure there is an appropriate and easily accessible database or repository to record the number of materials being developed, finalised, and implemented across the institution.

Ensure that there is an appropriate and easily accessible database to record the number of materials accessed and cited across the institution.

Awareness Raising Ensure researchers know that they should record all materials being developed, finalised, and implemented and where they can receive training in how to do this and who they should go to for help and support.

Ensure researchers know that that they should record all materials accessed and cited and where they can receive training in how to do this and who they should go to for help and support.

Ensure researchers understand the benefit of recording the number of materials being developed, finalised and implemented to both them as individuals and the institution, that they trust the process and understand the link to researcher assessment.

Training Train researchers in where and how to record materials being developed, finalised and implemented.

Train researchers in where and how to record materials cited and accessed.

Train researchers in materials development.

Ensure there are a range of best practice examples of quality materials that researchers can access and draw upon.

Education

The education category consists of 5 subcategories for courses, resources, teaching, supervision, and skills development with associated generic indicators/metrics for researcher assessment.

Courses

This subcategory focuses on educational courses. There is flexibility in the definition of ‘courses, which may include variation in the scope and duration of the courses. There is also flexibility in how the courses are ‘implemented’, such as by the researcher themselves or by other researchers.

Course Development
Indicator Type Indicator
Process # Courses Being Developed
Output # Courses Finalised
Outcome # Courses Implemented

Resources

This subcategory focuses on educational resources. There is flexibility in the definition of ‘resources, which may include articles, books, recordings, images, games, and digital tools. There is also flexibility in how the resources are ‘implemented’, ‘accessed’, and ‘cited’, which may dependon the type of resource and also on the standard practices for resources within a specific discipline.

Resource Development
Indicator Type Indicator
Process # Resources Being Developed
Output # Resources Finalised
Outcome # Resources Implemented
Outcome # Resources Accessed
# Resources Cited

Teaching

This subcategory focuses on teaching students and courses. There is flexibility in the definition of ‘courses’, which may include variation in course scope and duration, as well as ‘students’, which may include bachelor, master, and, if applicable, doctoral students. There is also flexibility in the number of students ‘passed in courses’, which may be the number per course or total across courses.

Student Teaching
Indicator Type Indicator
Process # Course Hours Assigned
Output # Courses Hours Taught
Outcome # Students Passed in Courses

Supervision

This subcategory on supervising students. There is flexibility in the definition of’ supervision’, which may include mentoring students and supervision of student theses, as well as in the definition of’ students’, which may include bachelor, master, and, if applicable, doctoral students.

Student Supervision
Indicator Type Indicator
Process # Students Being Supervised
Output # Students Supervised
Outcome # Supervised Student Theses
# Supervised Students Graduated

Skills

This subcategory focuses on skill development by researchers. There is flexibility in the definition of ‘courses’ and ‘certificates’, which may include variations in course scope and duration. The courses and certificates may cover any research, education, leadership, and valorisation skills.

Skills Development
Indicator Type Indicator
Process # Skills Courses Being Followed
Output # Skills Courses Completed
Outcome # Skills Certificates Obtained
Leadership

The leadership category consists of 4 subcategories for managing people, projects, organisational units, and recognition with associated generic indicators/metrics for researcher assessment.

People

This subcategory focuses on supervising staff. There is flexibility in the definition of ‘supervision’ which may include mentoring staff and supervision of (post)doctoral theses, as well as ‘staff’ which may include local and visiting (post)doctoral and senior researchers and other staff. There is also flexibility in the definition of ‘theses’, which may include (post)doctoral theses, as well as ‘projects’, which may include research and non-research projects managed by supervised staff.

Staff Supervision
Indicator Type Indicator
Process # Staff Being Supervised
Output # Staff Supervised
Outcome # Supervised Staff Theses
# Supervised Staff Projects

Projects

This subcategory focuses on managing projects. There is flexibility in the definition of ‘projects’ which may be funded or non-funded but should involve official management responsibility.A project which is ‘successfully evaluated’ has been formally reviewed and successfully closed.

Project Management
Indicator Type Indicator
Process # Projects Being Managed
Output # Projects Completed
Outcome # Projects Successfully Evaluated

Organisation

This subcategory focuses on managing organisational units. There is flexibility in the definition of ‘units’ which may include a team, group, institute, faculty, or university. The ‘positions’ should involve official management responsibility. There is also flexibility in the selection of ‘unit management outputs’ and ‘unit management outcomes’ which could be taken from relevant outputs and outcomes already identified in the RAF and should be formally agreed within the organisation.

Unit Management
Indicator Type Indicator
Process # Unit Management Positions Assigned
Output # Unit Management Positions Completed
# Agreed Unit Management Outputs
Outcome # Agreed Unit Management Outcomes

Recognition

This subcategory on the recognition of researchers through expert positions. There is flexibility in the definition of ‘expert positions’ including invited strategic, advisory, and honorary roles, which recognise researchers for their expertise and experience. There is also flexibility in the definition of ‘expert position outputs’ which may include meetings, presentations, and reports, as well as ‘expert position outcomes’, depending on the type of expert position. The ‘achievement awards’ may be for any recognised research, education, leadership, and valorisation contributions.

Expert Positions
Indicator Type Indicator
Process # Expert Positions Assigned
Output # Expert Positions Completed
# Expert Position Outputs
Outcome # Expert Position Outcomes
# Expert Achievement Awards
Valorisation

The valorisation category consists of 3 subcategories for communication, engagement, and innovation activities along with associated generic indicators/metrics for researcher assessment.

Communication

This subcategory focuses on research communication via public writing and public speaking. There is flexibility in the definition of ‘publications’, which may include print and social media and ‘appearances’ which may be physical or digital and include talks, conferences, workshops, and recordings. There is also flexibility in the definition of ‘accessed’ which may include readership, attendees, views, and downloads, and ‘cited’ which may include mentions in print and social media.

Public Writing
Indicator Type Indicator
Process # Publications Being Drafted
Output # Publications Published
Outcome # Publications Accessed
# Publications Cited
Public Speaking
Indicator Type Indicator
Process # Appearances Planned
Output # Appearances Made
Outcome # Appearances Accessed
# Appearances Cited

Engagement

This subcategory focuses on on intersectoral and citizen engagement. There is flexibility in the definition of ‘intersectoral’ which may include academic, public, and private organisations and ‘collaborations’ which may include research collaboration, staff secondments and exchanges, volunteering and advocacy, and policy development. There is also flexibility in the definition of ‘citizen science’, which may include any research activities with citizens and encompass activities in projects with a citizen science dimension or full projects on citizen science. There is also flexibility in the selection of ‘intersectoral outputs’ and ‘intersectoral outcomes’ as well as ‘citizen science outputs’ and ‘citizen science outcomes’ which could be taken from relevant outputs and outcomes in the RAF.

Intersectoral Engagement
Indicator Type Indicator
Process # Intersectoral Engagements
Output # Intersectoral Outputs
Outcome # Intersectoral Outcomes
# Organisations Engaged
Citizen Engagement
Indicator Type Indicator
Process # Citizen Science Activities Ongoing
Output # Citizen Science Activities Completed
# Citizen Science Outputs
Outcome # Citizen Science Outcomes
# Citizen Scientists Engaged

Innovation

This subcategory focuses on the innovation of research through research exploitation and entrepreneurial spirit. There is flexibility in the definition of ‘being legalised’, including defining Intellectual Property Rights (IPR), patenting, and licensing of research outputs, which may include innovative products and services. There is also flexibility in the size of spin-offs and start-ups as well as the definition of ‘employees’ in terms of number of people or Full Time Equivalent (FTE).

Research Exploitation
Indicator Type Indicator
Process # Research Outputs Being Legalised
Output # Research Outputs with Defined IPR
# Research Outputs Patented
Outcome # Research Outputs with Licenses
Entrepreneurial Spirit
Indicator Type Indicator
Process # Spin-offs/Start-ups Being Created
Output # Spin-offs/Start-ups Created
Outcome # Spin-off/Start-up Employees
# Spin-off/Start-up Products
# Spin-off/Start-up Service

2. Open Science Researcher Assessment Framework: Indicators and Metrics

Research

The research category consists of 6 subcategories for proposals, methods, data, software, publications, and materials with Open Science indicators and metrics for researcher assessment.

Proposal Development

This focuses on proposals for research projects to an RPO or RFO which are openly available. There is flexibility in how the proposals are made ‘openly available’.

Indicator Type Indicator
Process # Developing Project Proposals Openly Available
Output # Submitted Project Proposals Openly Available
Outcome # Granted Project Proposals Openly Available
Methods Development

This subcategory focuses on methods to conduct research which are openly available. There is flexibility in how the method sets are actually made ‘openly available’ for use by the public.

Indicator Type Indicator
Process # Developing Method Sets Openly Available
Output # Finalised Method Sets Openly Available
Outcome # Openly Available Method Sets Implemented
# Openly Available Method Sets Accessed
# Openly Available Method Sets Cited

Data

This subcategory focuses on research data planning, management, and peer review, which are openly available. There is flexibility in the inclusion or exclusion of a focus on FAIR and how data management plans, data sets, and data peer reviews are made ‘openly available’ for public use.

Data Planning
Indicator Type Indicator
Process #(FAIR) Developing Data Management Plans Openly Available
Output # (FAIR) Finalised Data Management Plans Openly Available
Outcome # (FAIR) Implemented Data Management Plans Openly Available
Data Management
Indicator Type Indicator
Process # Developing (FAIR) Data Sets Openly Available
Output # Finalised (FAIR) Data Sets Openly Available
# Archived (FAIR) Data Sets Openly Available
Outcome # Openly Available (FAIR) Data Sets Accessed
# Openly Available (FAIR) Data Sets Cited
Data Review
Indicator Type Indicator
Process # Draft (FAIR) Data Set Peer Reviews Openly Available
Output # Submitted (FAIR) Data Set Peer Reviews Openly Available
Outcome # Accepted (FAIR) Data Set Peer Reviews Openly Available

Software

This subcategory focuses on research software development and peer review which are openly available. There is flexibility in software sets and peer reviews being ‘openly available’.
Software Development
Indicator Type Indicator
Process # Developing Software Sets Openly Available
Output # Finalised Software Sets Openly Available
# Archived Software Sets Openly Available
Outcome # Openly Available Software Sets Accessed
# Openly Available Software Sets Cited
Software Review
Indicator Type Indicator
Process # Draft Software Set Peer Reviews Openly Available
Output # Submitted Software Set Peer Reviews Openly Available
Outcome # Accepted Software Set Peer Reviews Openly Available

Publications

This subcategory focuses on research publications and peer reviews which are openly available. There is flexibility in how the publications and peer reviews are made ‘openly available’. There is also flexibility in the version of the publication which may be a preprint, Author Accepted Manuscript (AAM), or Version of Record (VoR) as well as the type of open access such as green or gold. There is
further flexibility in whether the publicat ions adhere to the principles of Plan S.
Publication Drafting
Indicator Type Indicator
Process # Draft Publications Openly Available
Output # Submitted Publications Openly Available
Outcome # Published Publications Openly Available
# Openly Available Publications Accessed
# Openly Available Publications Cited
Publication Review
Indicator Type Indicator
Process # Draft Publication Peer Reviews Openly Available
Output # Submitted Publication Peer Reviews Openly Available
Outcome # Accepted Publication Peer Reviews Openly Available
Materials Development

This subcategory focuses on research materials which are openly available. There is flexibility in how the material sets aremade’openly available’ depending on the type of materials.

Indicator Type Indicator
Process # Developing Material Sets Openly Available
Output # Finalised Material Sets Openly Available
Outcome #Implemented Material Sets Openly Available
# Openly Available Material Sets Accessed
# Openly Available Material Sets Cited
Education

The education category consists of 5 subcategories for courses, resources, teaching, supervision, and skills development with related Open Science indicators/metrics for researcher assessment.

Courses

This subcategory focuses on educational courses which are on Open Science or openly available. There is flexibility in the definition of courses on ‘Open Science’ which may focus on many or specific Open Science practices as well as how the courses are made ‘openly available’.
Course Development
Indicator Type Indicator
Process # Open Science Courses Being Developed
# Developing Courses Openly Available
Output # Open Science Courses Finalised
# Finalised Courses Openly Available
Outcome # Open Science Courses Implemented
# Implemented Courses Openly Available

Resources

This subcategory focuses on educational resources which are on Open Science or openly available. There is flexibility in the definition of resources on ‘Open Science’ which may focus on many or specific Open Science practices as well as how the resources are made ‘openly available’.

Resource Development
Indicator Type Indicator
Process # Open Science Resources Being Developed
#Developing Resources Openly Available
Output # Open Science Resources Finalised
# Finalised Resources Openly Available
Outcome # Open Science Resources Implemented
# Implemented Resources Openly Available
 

Teaching

This subcategory focuses on teaching students and courses which are on Open Science or openly available. There is flexibility in the definition of courses on ‘Open Science’ which may focus on many or specific Open Science practices as well as how the courses are ‘openly available’.
Student Teaching
Indicator Type Indicator
Process # Open Science Course Hours Assigned
Output # Open Science Course Hours Taught
Outcome # Students Passed in Open Science Courses
# Students Passed in Openly Available Course

Supervision

This subcategory focuses on supervising students in Open Science and making student theses openly available. There is flexibility in the definition of ‘Open Science’ which may include many or specific Open Science practices and how thestudent theses are ‘openly available’.

Student Supervision
Indicator Type Indicator
Process # Students Being Supervised in Open Science
Output # Students Supervised in Open Science
Outcome # Supervised Student Theses Openly Available
# Supervised Students in Open Science Graduate

Skills

This subcategory focuses on skills development in Open Science by researchers. There is flexibility in the definition of ‘courses’ and ‘certificates’ which may include variation in course scope and duration and ‘Open Science’ which may focus on many or specific Open Science practices. Courses and certificates may cover research, education, leadership, and valorisation skills.

Skills Development
Indicator Type Indicator
Process # Open Science Skills Courses Being Followed
Output # Open Science Skills Courses Completed
Outcome # Open Science Skills Certificates Obtained
Leadership

The leadership category consists of 4 subcategories for people, project, and organisational unit management and recognition with Open Science indicators/metrics for researcher assessment.

People

This subcategory focuses on supervising staff in Open Science. There is flexibility in the definition of ‘Open Science’ which may include many or specific Open Science practices. There is also flexibility in how these are ‘openly available’ and how projects are ‘involving Open Science’.
Staff Supervision
Indicator Type Indicator
Process # Staff Being Supervised in Open Science
Output # Staff Supervised in Open Science
Outcome # Supervised Staff Theses Openly Available
# Supervised Staff Projects involving Open Science

Projects

This subcategory focuses on managing projects involving Open Science. There is flexibility in the definition of ‘involving Open Science’ which may include variation in the scope and duration of projects and ‘Open Science’ which may include many or specific Open Science practices.

Project Management
Indicator Type Indicator
Process # Projects involving Open Science Being Managed
Output # Projects involving Open Science Completed
Outcome # Projects involving Open ScienceSuccessfully Evaluated

Organisation

This subcategory focuses on managing organisational units involving Open Science. There is flexibility in the definition of ‘Open Science’ which may include many or specific Open Science practices. There is also flexibility in the selection of ‘unit management outputs’ and ‘unit management outcomes’ involving Open Science, which could be taken from relevant outputs and outcomes already identified in the RAF and should be formally agreed within the organisation.

Unit Management
Indicator Type Indicator
Process # Unit Management Positions in Open Science Assigned
Output # Unit Management Positions in Open Science Completed
#Agreed Unit Management Outputs involving Open Science
Outcome # Agreed Unit Management Outcomes involving Open Science

Recognition

This subcategory focuses on the recognition of researchers through expert positions in Open Science. There is flexibility in the definition of ‘Open Science’ including many or specific practices and how ‘expert position outputs’ and ‘expert position outcomes’ are ‘openly available’.

Expert Positions
Indicator Type Indicator
Process # Expert Positions in Open Science Assigned
Output # Expert Positions in Open Science Completed
# Open Science Expert Position Outputs
# Expert Position Outputs Openly Available
Outcome # Expert Position Outcomes Openly Available
# Expert Achievement Awards for Open Science
Valorisation

The valorisation category consists of 3 subcategories for communication, engagement, and innovation along with associated Open Science indicators/metrics for researcher assessment

Communication

This subcategory focuses on research communication via public writing and public speaking, which is openly available. There is flexibility in how the publications and appearances are made ‘openly available’ and also in the definition of ‘Open Science’ including many or specific practices.

Public Writing
Indicator Type Indicator
Process # Draft Publications Openly Available
Output # Published Publications Openly Available
Outcome # Openly Available Publications Accessed
# Openly Available Publications Cited
Public Speaking
Indicator Type Indicator
Process # Appearances on Open Science Planned
Output # Appearances on Open Science Given
Outcome # Appearances on Open Science Accessed
# Appearances on Open Science Cited
# Appearances Openly Available

Engagement

This subcategory focuses on intersectoral and citizen engagement involving Open Science. There is flexibility in the definition of ‘Open Science’ including many or specific Open Science practices. There is also flexibility in the selection of ‘intersectoral outputs’ and ‘intersectoral outcomes’ as well as ‘citizen science outputs’ and ‘citizen science outcomes’ for Open Science, which could be taken from
relevant outputs and outcomes for Open Science already identified in the RAF.
Intersectoral Engagement
Indicator Type Indicator
Process # Intersectoral Engagements involving Open Science
Output # Intersectoral Outputs involving Open Science
Outcome # Intersectoral Outcomes involving Open Science
# Organisations Engaged for Open Science
Citizen Engagement
Indicator Type Indicator
Process # Citizen Science Activities involving Open Science Ongoing
Output # Citizen Science Activities involving Open Science Completed
# Citizen Science Outputs involving Open Science
Outcome # Citizen Science Outcomes involving Open Science
# Citizen Scientists Engaged in Open Science

Innovation

This subcategory focuses on the open innovation of research including research exploitation and entrepreneurial spirit involving Open Science as in Table 36. There is flexibility in the definition of ‘Open Science’ including many or specific Open Science practices and how research outputs are ‘openly available’. Spin-offs and start-ups may be focused on supporting or exploiting Open Science.
Research Exploitation
Indicator Type Indicator
Process # Openly Available Research Outputs Being Legalised
Output # Openly Available Research Outputs with Defined IPR
# Openly Available Research Outputs Patented
Outcome # Research Outputs with Open Licenses
Entrepreneurial Spirit
Indicator Type Indicator
Process # Open Science Spin-offs/Start-ups Being Created
Output # Open Science Spin-offs/Start-ups Created
Outcome # Open Science Spin-off/Start-up Employees
# Open Science Spin-off/Start-up Products
# Open Science Spin-off/Start-up Services
Research Assessment Framework (RAF) Purpose

The Research Assessment Framework (RAF) serves as a dynamic tool to reform the evaluation of researchers, encouraging and rewarding Open Science practices in project applications, career development, and progression at Research Performing Organisations (RPOs) and Research Funding Organisations (RFOs). RAF encompasses the entire spectrum of research, education, leadership, and valorisation activities. It is designed to be adaptable across countries, disciplines, and organisations, offering both quantitative and qualitative approaches to implementing indicators.

Privacy Preferences

When you visit our website, it may store information through your browser from specific services, usually in the form of cookies. Our Privacy Policy can be read here.

Here you can change your Privacy preferences. It is worth noting that blocking some types of cookies may impact your experience on our website and the services we are able to offer.

Click to enable/disable Google Analytics tracking code.
Click to enable/disable Google Fonts.
Click to enable/disable Google Maps.
Click to enable/disable video embeds.
Our website uses cookies, mainly from 3rd party services. Define your Privacy Preferences and/or agree to our use of cookies.