OPUS helps reform the assessment of research

Indicators/Metrics for Open Science/Researcher Assessment

OPUS has developed a set of indicators and metrics, and interventions for Open Science toward a system that incentivises and rewards researchers to take up practices of providing open access to research outputs, early and open sharing of research, participation in open peer-review, measures to ensure reproducibility of results, and involving all stakeholders in co-creation.

The RAF is designed to comprehensively cover the full spectrum of researcher activities and offer researchers the possibility to be assessed on all relevant activities in a given assessment period, rather than solely on the traditional metrics of the number of peer-reviewed publications and citations and the journal impact factor (JIF). The inclusion of Open Science metrics in a given RAF at an RPO or RFO ensures that Open Science activities are explicitly recognised in the assessment. It remains the prerogative of RPOs and RFOs to determine how exactly to reward researchers for Open Science.

1. Generic Researcher Assessment Framework: Indicators and Metrics

The RAF is structured around 4 main categories of activities, which aredivided into subcategories of these activities and consist of indicator groups of related indicators/metrics.

Research

The research category consists of 6 subcategories for proposals, methods, data, software, publications, and materials with associated generic indicators/metrics for researcher assessment.

Proposal Development

This focuses on proposals for research projects to an RPO or RFO. The qualitative description of the proposal may include the associated funding call and roles, activities, and efforts of the researchers involved in the proposal development and in the to-be-granted project.

Indicator Type Quantitative Metric
Process # of Project Proposals Being Developed
Output # of Project Proposals Submitted
Outcome # of Project Proposals Granted
Methods Development

This subcategory focuses on methods to conduct research. There is flexibility in the definition of ‘methods, which may include research methodologies and protocols. There is also flexibility in how the method sets are ‘implemented’, such as by the researcher themselves or by others; how the method sets are ‘accessed’, which may include restricted forms of access; and how the method sets are ‘cited, which may be dependent on standard practices within a specific discipline.

Indicator Type Indicator
Process #Method Sets Being Developed
Output # Method Sets Finalised
Outcome # Method Sets Implemented
Outcome # Method Sets Accessed

# Method Sets Cited

Data Planning, Management, and Review

There is flexibility in the definition of ‘data management plan, which may be dependent on standard practices within a specific discipline. There is also flexibility in the inclusion or exclusion of a focus on adopting the Findable, Accessible, Interoperable, and Reusable (FAIR) principles for research data. There is further flexibility in how data sets are ‘archived’, which may be in a trusted repository with(out) long-term preservation; how data sets are ‘accessed’, including restricted forms of access; and how data sets are ‘cited’, which may depend on standard practices within a specific discipline. Data set peer reviews involve formal peer review requested by an academic venue or publisher.

Data Planning
Indicator Type Indicator
Process #(FAIR) Data Management Plans Being Developed
Output # (FAIR) Data Management Plans Finalised
Outcome # (FAIR) Data Management Plans Implemented
Data Management
Indicator Type Indicator
Process # (FAIR) Data Sets Being Developed
Output # (FAIR) Data Sets Finalised
# (FAIR) Data Sets Archived Outcome
Outcome # (FAIR) Data Sets Accessed
# (FAIR) Data Sets Cited
Data Review
Indicator Type Indicator
Process # (FAIR) Data Set Peer Reviews Being Drafted
Output # (FAIR) Data Set Peer Reviews Submitted
Outcome # (FAIR) Data Set Peer Reviews Accepted

Software

This subcategory focuses on research software development, and peer review. There is flexibility in the definition of software’, ‘which may include algorithms, code, and packages. There is also flexibility in how the software sets are ‘archived’, which may be in a trusted repository with(out) long-term preservation; how the software sets are ‘accessed’, including restricted forms of access; and how the software sets are ‘cited’, which may be dependent on standard practices within a specific discipline. Software set peer reviews may include non-editorial board and security check requests.

Software Development
Indicator Type Indicator
Process # Software Sets Being Developed
Output # Software Sets Finalised
# Software Sets Archived
Outcome # Software Sets Accessed
# Software Sets Cited
Software Review
Indicator Type Indicator
Process # Software Set Peer Reviews Being Drafted
Output # Software Set Peer Reviews Submitted
Outcome # Software Set Peer Reviews Accepted

Publications

This subcategory focuses on research publications and peer reviews. There is flexibility in the definition of ‘publications’, which may include books, chapters, and articles. There is also flexibility for the recognition of the JIF of the academic venue or publisher. There is further flexibility in how publications are ‘archived’, which may be in a trusted repository with(out) long-term preservation; how publications are ‘accessed’, including restricted forms of access; and how publications are ‘cited’, which may depend on standard practices within a discipline. Publication peer reviews always involve formal peer review, which is requested by an (editorial board of an) academic venue or publisher.

Publication Drafting
Indicator Type Indicator
Process # Publications Being Drafted
Output # Publications Submitted
Outcome # Publications Published
# Publications Accessed
# Publications Cited
Publication Review
Indicator Type Indicator
Process # Publication Peer Reviews Being Drafted
Output # Publication Peer Reviews Submitted
Outcome # Publication Peer Reviews Accepted
Materials Development

This subcategory focuses on research materials. There is flexibility in the definition of ‘materials, which may include artwork, tools, instruments, and hardware. There is also flexibility in how the material sets are ‘implemented’ and ‘accessed’, which may be dependent on the type of materials,and how the material sets are ‘cited, which may be dependent on standard practices in a discipline.

Indicator Type Indicator
Process # Material Sets Being Developed
Output # Material Sets Finalised
Outcome # Material Sets Implemented
# Material Sets Accessed
# Material Sets Cited
Education

The education category consists of 5 subcategories for courses, resources, teaching, supervision, and skills development with associated generic indicators/metrics for researcher assessment.

Courses

This subcategory focuses on educational courses. There is flexibility in the definition of ‘courses, which may include variation in the scope and duration of the courses. There is also flexibility in how the courses are ‘implemented’, such as by the researcher themselves or by other researchers.

Course Development
Indicator Type Indicator
Process # Courses Being Developed
Output # Courses Finalised
Outcome # Courses Implemented

Resources

This subcategory focuses on educational resources. There is flexibility in the definition of ‘resources, which may include articles, books, recordings, images, games, and digital tools. There is also flexibility in how the resources are ‘implemented’, ‘accessed’, and ‘cited’, which may dependon the type of resource and also on the standard practices for resources within a specific discipline.

Resource Development
Indicator Type Indicator
Process # Resources Being Developed
Output # Resources Finalised
Outcome # Resources Implemented
Outcome # Resources Accessed
# Resources Cited

Teaching

This subcategory focuses on teaching students and courses. There is flexibility in the definition of ‘courses’, which may include variation in course scope and duration, as well as ‘students’, which may include bachelor, master, and, if applicable, doctoral students. There is also flexibility in the number of students ‘passed in courses’, which may be the number per course or total across courses.

Student Teaching
Indicator Type Indicator
Process # Course Hours Assigned
Output # Courses Hours Taught
Outcome # Students Passed in Courses

Supervision

This subcategory on supervising students. There is flexibility in the definition of’ supervision’, which may include mentoring students and supervision of student theses, as well as in the definition of’ students’, which may include bachelor, master, and, if applicable, doctoral students.

Student Supervision
Indicator Type Indicator
Process # Students Being Supervised
Output # Students Supervised
Outcome # Supervised Student Theses
# Supervised Students Graduated

Skills

This subcategory focuses on skill development by researchers. There is flexibility in the definition of ‘courses’ and ‘certificates’, which may include variations in course scope and duration. The courses and certificates may cover any research, education, leadership, and valorisation skills.

Skills Development
Indicator Type Indicator
Process # Skills Courses Being Followed
Output # Skills Courses Completed
Outcome # Skills Certificates Obtained
Leadership

The leadership category consists of 4 subcategories for managing people, projects, organisational units, and recognition with associated generic indicators/metrics for researcher assessment.

People

This subcategory focuses on supervising staff. There is flexibility in the definition of ‘supervision’ which may include mentoring staff and supervision of (post)doctoral theses, as well as ‘staff’ which may include local and visiting (post)doctoral and senior researchers and other staff. There is also flexibility in the definition of ‘theses’, which may include (post)doctoral theses, as well as ‘projects’, which may include research and non-research projects managed by supervised staff.

Staff Supervision
Indicator Type Indicator
Process # Staff Being Supervised
Output # Staff Supervised
Outcome # Supervised Staff Theses
# Supervised Staff Projects

Projects

This subcategory focuses on managing projects. There is flexibility in the definition of ‘projects’ which may be funded or non-funded but should involve official management responsibility.A project which is ‘successfully evaluated’ has been formally reviewed and successfully closed.

Project Management
Indicator Type Indicator
Process # Projects Being Managed
Output # Projects Completed
Outcome # Projects Successfully Evaluated

Organisation

This subcategory focuses on managing organisational units. There is flexibility in the definition of ‘units’ which may include a team, group, institute, faculty, or university. The ‘positions’ should involve official management responsibility. There is also flexibility in the selection of ‘unit management outputs’ and ‘unit management outcomes’ which could be taken from relevant outputs and outcomes already identified in the RAF and should be formally agreed within the organisation.

Unit Management
Indicator Type Indicator
Process # Unit Management Positions Assigned
Output # Unit Management Positions Completed
# Agreed Unit Management Outputs
Outcome # Agreed Unit Management Outcomes

Recognition

This subcategory on the recognition of researchers through expert positions. There is flexibility in the definition of ‘expert positions’ including invited strategic, advisory, and honorary roles, which recognise researchers for their expertise and experience. There is also flexibility in the definition of ‘expert position outputs’ which may include meetings, presentations, and reports, as well as ‘expert position outcomes’, depending on the type of expert position. The ‘achievement awards’ may be for any recognised research, education, leadership, and valorisation contributions.

Expert Positions
Indicator Type Indicator
Process # Expert Positions Assigned
Output # Expert Positions Completed
# Expert Position Outputs
Outcome # Expert Position Outcomes
# Expert Achievement Awards
Valorisation

The valorisation category consists of 3 subcategories for communication, engagement, and innovation activities along with associated generic indicators/metrics for researcher assessment.

Communication

This subcategory focuses on research communication via public writing and public speaking. There is flexibility in the definition of ‘publications’, which may include print and social media and ‘appearances’ which may be physical or digital and include talks, conferences, workshops, and recordings. There is also flexibility in the definition of ‘accessed’ which may include readership, attendees, views, and downloads, and ‘cited’ which may include mentions in print and social media.

Public Writing
Indicator Type Indicator
Process # Publications Being Drafted
Output # Publications Published
Outcome # Publications Accessed
# Publications Cited
Public Speaking
Indicator Type Indicator
Process # Appearances Planned
Output # Appearances Made
Outcome # Appearances Accessed
# Appearances Cited

Engagement

This subcategory focuses on on intersectoral and citizen engagement. There is flexibility in the definition of ‘intersectoral’ which may include academic, public, and private organisations and ‘collaborations’ which may include research collaboration, staff secondments and exchanges, volunteering and advocacy, and policy development. There is also flexibility in the definition of ‘citizen science’, which may include any research activities with citizens and encompass activities in projects with a citizen science dimension or full projects on citizen science. There is also flexibility in the selection of ‘intersectoral outputs’ and ‘intersectoral outcomes’ as well as ‘citizen science outputs’ and ‘citizen science outcomes’ which could be taken from relevant outputs and outcomes in the RAF.

Intersectoral Engagement
Indicator Type Indicator
Process # Intersectoral Engagements
Output # Intersectoral Outputs
Outcome # Intersectoral Outcomes
# Organisations Engaged
Citizen Engagement
Indicator Type Indicator
Process # Citizen Science Activities Ongoing
Output # Citizen Science Activities Completed
# Citizen Science Outputs
Outcome # Citizen Science Outcomes
# Citizen Scientists Engaged

Innovation

This subcategory focuses on the innovation of research through research exploitation and entrepreneurial spirit. There is flexibility in the definition of ‘being legalised’, including defining Intellectual Property Rights (IPR), patenting, and licensing of research outputs, which may include innovative products and services. There is also flexibility in the size of spin-offs and start-ups as well as the definition of ‘employees’ in terms of number of people or Full Time Equivalent (FTE).

Research Exploitation
Indicator Type Indicator
Process # Research Outputs Being Legalised
Output # Research Outputs with Defined IPR
# Research Outputs Patented
Outcome # Research Outputs with Licenses
Entrepreneurial Spirit
Indicator Type Indicator
Process # Spin-offs/Start-ups Being Created
Output # Spin-offs/Start-ups Created
Outcome # Spin-off/Start-up Employees
# Spin-off/Start-up Products
# Spin-off/Start-up Service

2. Open Science Researcher Assessment Framework: Indicators and Metrics

Research

The research category consists of 6 subcategories for proposals, methods, data, software, publications, and materials with Open Science indicators and metrics for researcher assessment.

Proposal Development

This focuses on proposals for research projects to an RPO or RFO which are openly available. There is flexibility in how the proposals are made ‘openly available’.

Indicator Type Indicator
Process # Developing Project Proposals Openly Available
Output # Submitted Project Proposals Openly Available
Outcome # Granted Project Proposals Openly Available
Methods Development

This subcategory focuses on methods to conduct research which are openly available. There is flexibility in how the method sets are actually made ‘openly available’ for use by the public.

Indicator Type Indicator
Process # Developing Method Sets Openly Available
Output # Finalised Method Sets Openly Available
Outcome # Openly Available Method Sets Implemented
# Openly Available Method Sets Accessed
# Openly Available Method Sets Cited

Data

This subcategory focuses on research data planning, management, and peer review, which are openly available. There is flexibility in the inclusion or exclusion of a focus on FAIR and how data management plans, data sets, and data peer reviews are made ‘openly available’ for public use.

Data Planning
Indicator Type Indicator
Process #(FAIR) Developing Data Management Plans Openly Available
Output # (FAIR) Finalised Data Management Plans Openly Available
Outcome # (FAIR) Implemented Data Management Plans Openly Available
Data Management
Indicator Type Indicator
Process # Developing (FAIR) Data Sets Openly Available
Output # Finalised (FAIR) Data Sets Openly Available
# Archived (FAIR) Data Sets Openly Available
Outcome # Openly Available (FAIR) Data Sets Accessed
# Openly Available (FAIR) Data Sets Cited
Data Review
Indicator Type Indicator
Process # Draft (FAIR) Data Set Peer Reviews Openly Available
Output # Submitted (FAIR) Data Set Peer Reviews Openly Available
Outcome # Accepted (FAIR) Data Set Peer Reviews Openly Available

Software

This subcategory focuses on research software development and peer review which are openly available. There is flexibility in software sets and peer reviews being ‘openly available’.
Software Development
Indicator Type Indicator
Process # Developing Software Sets Openly Available
Output # Finalised Software Sets Openly Available
# Archived Software Sets Openly Available
Outcome # Openly Available Software Sets Accessed
# Openly Available Software Sets Cited
Software Review
Indicator Type Indicator
Process # Draft Software Set Peer Reviews Openly Available
Output # Submitted Software Set Peer Reviews Openly Available
Outcome # Accepted Software Set Peer Reviews Openly Available

Publications

This subcategory focuses on research publications and peer reviews which are openly available. There is flexibility in how the publications and peer reviews are made ‘openly available’. There is also flexibility in the version of the publication which may be a preprint, Author Accepted Manuscript (AAM), or Version of Record (VoR) as well as the type of open access such as green or gold. There is
further flexibility in whether the publicat ions adhere to the principles of Plan S.
Publication Drafting
Indicator Type Indicator
Process # Draft Publications Openly Available
Output # Submitted Publications Openly Available
Outcome # Published Publications Openly Available
# Openly Available Publications Accessed
# Openly Available Publications Cited
Publication Review
Indicator Type Indicator
Process # Draft Publication Peer Reviews Openly Available
Output # Submitted Publication Peer Reviews Openly Available
Outcome # Accepted Publication Peer Reviews Openly Available
Materials Development

This subcategory focuses on research materials which are openly available. There is flexibility in how the material sets aremade’openly available’ depending on the type of materials.

Indicator Type Indicator
Process # Developing Material Sets Openly Available
Output # Finalised Material Sets Openly Available
Outcome #Implemented Material Sets Openly Available
# Openly Available Material Sets Accessed
# Openly Available Material Sets Cited
Education

The education category consists of 5 subcategories for courses, resources, teaching, supervision, and skills development with related Open Science indicators/metrics for researcher assessment.

Courses

This subcategory focuses on educational courses which are on Open Science or openly available. There is flexibility in the definition of courses on ‘Open Science’ which may focus on many or specific Open Science practices as well as how the courses are made ‘openly available’.
Course Development
Indicator Type Indicator
Process # Open Science Courses Being Developed
# Developing Courses Openly Available
Output # Open Science Courses Finalised
# Finalised Courses Openly Available
Outcome # Open Science Courses Implemented
# Implemented Courses Openly Available

Resources

This subcategory focuses on educational resources which are on Open Science or openly available. There is flexibility in the definition of resources on ‘Open Science’ which may focus on many or specific Open Science practices as well as how the resources are made ‘openly available’.

Resource Development
Indicator Type Indicator
Process # Open Science Resources Being Developed
#Developing Resources Openly Available
Output # Open Science Resources Finalised
# Finalised Resources Openly Available
Outcome # Open Science Resources Implemented
# Implemented Resources Openly Available
 

Teaching

This subcategory focuses on teaching students and courses which are on Open Science or openly available. There is flexibility in the definition of courses on ‘Open Science’ which may focus on many or specific Open Science practices as well as how the courses are ‘openly available’.
Student Teaching
Indicator Type Indicator
Process # Open Science Course Hours Assigned
Output # Open Science Course Hours Taught
Outcome # Students Passed in Open Science Courses
# Students Passed in Openly Available Course

Supervision

This subcategory focuses on supervising students in Open Science and making student theses openly available. There is flexibility in the definition of ‘Open Science’ which may include many or specific Open Science practices and how thestudent theses are ‘openly available’.

Student Supervision
Indicator Type Indicator
Process # Students Being Supervised in Open Science
Output # Students Supervised in Open Science
Outcome # Supervised Student Theses Openly Available
# Supervised Students in Open Science Graduate

Skills

This subcategory focuses on skills development in Open Science by researchers. There is flexibility in the definition of ‘courses’ and ‘certificates’ which may include variation in course scope and duration and ‘Open Science’ which may focus on many or specific Open Science practices. Courses and certificates may cover research, education, leadership, and valorisation skills.

Skills Development
Indicator Type Indicator
Process # Open Science Skills Courses Being Followed
Output # Open Science Skills Courses Completed
Outcome # Open Science Skills Certificates Obtained
Leadership

The leadership category consists of 4 subcategories for people, project, and organisational unit management and recognition with Open Science indicators/metrics for researcher assessment.

People

This subcategory focuses on supervising staff in Open Science. There is flexibility in the definition of ‘Open Science’ which may include many or specific Open Science practices. There is also flexibility in how these are ‘openly available’ and how projects are ‘involving Open Science’.
Staff Supervision
Indicator Type Indicator
Process # Staff Being Supervised in Open Science
Output # Staff Supervised in Open Science
Outcome # Supervised Staff Theses Openly Available
# Supervised Staff Projects involving Open Science

Projects

This subcategory focuses on managing projects involving Open Science. There is flexibility in the definition of ‘involving Open Science’ which may include variation in the scope and duration of projects and ‘Open Science’ which may include many or specific Open Science practices.

Project Management
Indicator Type Indicator
Process # Projects involving Open Science Being Managed
Output # Projects involving Open Science Completed
Outcome # Projects involving Open ScienceSuccessfully Evaluated

Organisation

This subcategory focuses on managing organisational units involving Open Science. There is flexibility in the definition of ‘Open Science’ which may include many or specific Open Science practices. There is also flexibility in the selection of ‘unit management outputs’ and ‘unit management outcomes’ involving Open Science, which could be taken from relevant outputs and outcomes already identified in the RAF and should be formally agreed within the organisation.

Unit Management
Indicator Type Indicator
Process # Unit Management Positions in Open Science Assigned
Output # Unit Management Positions in Open Science Completed
#Agreed Unit Management Outputs involving Open Science
Outcome # Agreed Unit Management Outcomes involving Open Science

Recognition

This subcategory focuses on the recognition of researchers through expert positions in Open Science. There is flexibility in the definition of ‘Open Science’ including many or specific practices and how ‘expert position outputs’ and ‘expert position outcomes’ are ‘openly available’.

Expert Positions
Indicator Type Indicator
Process # Expert Positions in Open Science Assigned
Output # Expert Positions in Open Science Completed
# Open Science Expert Position Outputs
# Expert Position Outputs Openly Available
Outcome # Expert Position Outcomes Openly Available
# Expert Achievement Awards for Open Science
Valorisation

The valorisation category consists of 3 subcategories for communication, engagement, and innovation along with associated Open Science indicators/metrics for researcher assessment

Communication

This subcategory focuses on research communication via public writing and public speaking, which is openly available. There is flexibility in how the publications and appearances are made ‘openly available’ and also in the definition of ‘Open Science’ including many or specific practices.

Public Writing
Indicator Type Indicator
Process # Draft Publications Openly Available
Output # Published Publications Openly Available
Outcome # Openly Available Publications Accessed
# Openly Available Publications Cited
Public Speaking
Indicator Type Indicator
Process # Appearances on Open Science Planned
Output # Appearances on Open Science Given
Outcome # Appearances on Open Science Accessed
# Appearances on Open Science Cited
# Appearances Openly Available

Engagement

This subcategory focuses on intersectoral and citizen engagement involving Open Science. There is flexibility in the definition of ‘Open Science’ including many or specific Open Science practices. There is also flexibility in the selection of ‘intersectoral outputs’ and ‘intersectoral outcomes’ as well as ‘citizen science outputs’ and ‘citizen science outcomes’ for Open Science, which could be taken from
relevant outputs and outcomes for Open Science already identified in the RAF.
Intersectoral Engagement
Indicator Type Indicator
Process # Intersectoral Engagements involving Open Science
Output # Intersectoral Outputs involving Open Science
Outcome # Intersectoral Outcomes involving Open Science
# Organisations Engaged for Open Science
Citizen Engagement
Indicator Type Indicator
Process # Citizen Science Activities involving Open Science Ongoing
Output # Citizen Science Activities involving Open Science Completed
# Citizen Science Outputs involving Open Science
Outcome # Citizen Science Outcomes involving Open Science
# Citizen Scientists Engaged in Open Science

Innovation

This subcategory focuses on the open innovation of research including research exploitation and entrepreneurial spirit involving Open Science as in Table 36. There is flexibility in the definition of ‘Open Science’ including many or specific Open Science practices and how research outputs are ‘openly available’. Spin-offs and start-ups may be focused on supporting or exploiting Open Science.
Research Exploitation
Indicator Type Indicator
Process # Openly Available Research Outputs Being Legalised
Output # Openly Available Research Outputs with Defined IPR
# Openly Available Research Outputs Patented
Outcome # Research Outputs with Open Licenses
Entrepreneurial Spirit
Indicator Type Indicator
Process # Open Science Spin-offs/Start-ups Being Created
Output # Open Science Spin-offs/Start-ups Created
Outcome # Open Science Spin-off/Start-up Employees
# Open Science Spin-off/Start-up Products
# Open Science Spin-off/Start-up Services
Research Assessment Framework (RAF) Purpose

The Research Assessment Framework (RAF) serves as a dynamic tool to reform the evaluation of researchers, encouraging and rewarding Open Science practices in project applications, career development, and progression at Research Performing Organisations (RPOs) and Research Funding Organisations (RFOs). RAF encompasses the entire spectrum of research, education, leadership, and valorisation activities. It is designed to be adaptable across countries, disciplines, and organisations, offering both quantitative and qualitative approaches to implementing indicators.

Privacy Preferences

When you visit our website, it may store information through your browser from specific services, usually in the form of cookies. Our Privacy Policy can be read here.

Here you can change your Privacy preferences. It is worth noting that blocking some types of cookies may impact your experience on our website and the services we are able to offer.

Click to enable/disable Google Analytics tracking code.
Click to enable/disable Google Fonts.
Click to enable/disable Google Maps.
Click to enable/disable video embeds.
Our website uses cookies, mainly from 3rd party services. Define your Privacy Preferences and/or agree to our use of cookies.