OPUS helps reform the assessment of research
Indicators/Metrics for Open Science/Researcher Assessment
OPUS has developed a set of indicators and metrics, and interventions for Open Science toward a system that incentivises and rewards researchers to take up practices of providing open access to research outputs, early and open sharing of research, participation in open peer-review, measures to ensure reproducibility of results, and involving all stakeholders in co-creation.
The RAF is designed to comprehensively cover the full spectrum of researcher activities and offer researchers the possibility to be assessed on all relevant activities in a given assessment period, rather than solely on the traditional metrics of the number of peer-reviewed publications and citations and the journal impact factor (JIF). The inclusion of Open Science metrics in a given RAF at an RPO or RFO ensures that Open Science activities are explicitly recognised in the assessment. It remains the prerogative of RPOs and RFOs to determine how exactly to reward researchers for Open Science.
1. Generic Researcher Assessment Framework: Indicators and Metrics
The RAF is structured around 4 main categories of activities, which aredivided into subcategories of these activities and consist of indicator groups of related indicators/metrics.
Proposal Development
This focuses on proposals for research projects to an RPO or RFO. The qualitative description of the proposal may include the associated funding call and roles, activities, and efforts of the researchers involved in the proposal development and in the to-be-granted project.
Indicator Type | Quantitative Metric |
Process | # of Project Proposals Being Developed |
Output | # of Project Proposals Submitted |
Outcome | # of Project Proposals Granted |
Methods Development
This subcategory focuses on methods to conduct research. There is flexibility in the definition of ‘methods, which may include research methodologies and protocols. There is also flexibility in how the method sets are ‘implemented’, such as by the researcher themselves or by others; how the method sets are ‘accessed’, which may include restricted forms of access; and how the method sets are ‘cited, which may be dependent on standard practices within a specific discipline.
Indicator Type | Indicator |
Process | #Method Sets Being Developed |
Output | # Method Sets Finalised |
Outcome | # Method Sets Implemented |
Outcome | # Method Sets Accessed
# Method Sets Cited |
Data Planning, Management, and Review
There is flexibility in the definition of ‘data management plan, which may be dependent on standard practices within a specific discipline. There is also flexibility in the inclusion or exclusion of a focus on adopting the Findable, Accessible, Interoperable, and Reusable (FAIR) principles for research data. There is further flexibility in how data sets are ‘archived’, which may be in a trusted repository with(out) long-term preservation; how data sets are ‘accessed’, including restricted forms of access; and how data sets are ‘cited’, which may depend on standard practices within a specific discipline. Data set peer reviews involve formal peer review requested by an academic venue or publisher.
Data Planning
Indicator Type | Indicator |
Process | #(FAIR) Data Management Plans Being Developed |
Output | # (FAIR) Data Management Plans Finalised |
Outcome | # (FAIR) Data Management Plans Implemented |
Data Management
Indicator Type | Indicator |
Process | # (FAIR) Data Sets Being Developed |
Output | # (FAIR) Data Sets Finalised # (FAIR) Data Sets Archived Outcome |
Outcome | # (FAIR) Data Sets Accessed # (FAIR) Data Sets Cited |
Data Review
Indicator Type | Indicator |
Process | # (FAIR) Data Set Peer Reviews Being Drafted |
Output | # (FAIR) Data Set Peer Reviews Submitted |
Outcome | # (FAIR) Data Set Peer Reviews Accepted |
Software
This subcategory focuses on research software development, and peer review. There is flexibility in the definition of software’, ‘which may include algorithms, code, and packages. There is also flexibility in how the software sets are ‘archived’, which may be in a trusted repository with(out) long-term preservation; how the software sets are ‘accessed’, including restricted forms of access; and how the software sets are ‘cited’, which may be dependent on standard practices within a specific discipline. Software set peer reviews may include non-editorial board and security check requests.
Software Development
Indicator Type | Indicator |
Process | # Software Sets Being Developed |
Output | # Software Sets Finalised # Software Sets Archived |
Outcome | # Software Sets Accessed # Software Sets Cited |
Software Review
Indicator Type | Indicator |
Process | # Software Set Peer Reviews Being Drafted |
Output | # Software Set Peer Reviews Submitted |
Outcome | # Software Set Peer Reviews Accepted |
Publications
This subcategory focuses on research publications and peer reviews. There is flexibility in the definition of ‘publications’, which may include books, chapters, and articles. There is also flexibility for the recognition of the JIF of the academic venue or publisher. There is further flexibility in how publications are ‘archived’, which may be in a trusted repository with(out) long-term preservation; how publications are ‘accessed’, including restricted forms of access; and how publications are ‘cited’, which may depend on standard practices within a discipline. Publication peer reviews always involve formal peer review, which is requested by an (editorial board of an) academic venue or publisher.
Publication Drafting
Indicator Type | Indicator |
Process | # Publications Being Drafted |
Output | # Publications Submitted |
Outcome | # Publications Published # Publications Accessed # Publications Cited |
Publication Review
Indicator Type | Indicator |
Process | # Publication Peer Reviews Being Drafted |
Output | # Publication Peer Reviews Submitted |
Outcome | # Publication Peer Reviews Accepted |
Materials Development
This subcategory focuses on research materials. There is flexibility in the definition of ‘materials, which may include artwork, tools, instruments, and hardware. There is also flexibility in how the material sets are ‘implemented’ and ‘accessed’, which may be dependent on the type of materials,and how the material sets are ‘cited, which may be dependent on standard practices in a discipline.
Indicator Type | Indicator |
Process | # Material Sets Being Developed |
Output | # Material Sets Finalised |
Outcome | # Material Sets Implemented # Material Sets Accessed # Material Sets Cited |
Courses
This subcategory focuses on educational courses. There is flexibility in the definition of ‘courses, which may include variation in the scope and duration of the courses. There is also flexibility in how the courses are ‘implemented’, such as by the researcher themselves or by other researchers.
Course Development
Indicator Type | Indicator |
Process | # Courses Being Developed |
Output | # Courses Finalised |
Outcome | # Courses Implemented |
Resources
This subcategory focuses on educational resources. There is flexibility in the definition of ‘resources, which may include articles, books, recordings, images, games, and digital tools. There is also flexibility in how the resources are ‘implemented’, ‘accessed’, and ‘cited’, which may dependon the type of resource and also on the standard practices for resources within a specific discipline.
Resource Development
Indicator Type | Indicator |
Process | # Resources Being Developed |
Output | # Resources Finalised |
Outcome | # Resources Implemented |
Outcome | # Resources Accessed # Resources Cited |
Teaching
This subcategory focuses on teaching students and courses. There is flexibility in the definition of ‘courses’, which may include variation in course scope and duration, as well as ‘students’, which may include bachelor, master, and, if applicable, doctoral students. There is also flexibility in the number of students ‘passed in courses’, which may be the number per course or total across courses.
Student Teaching
Indicator Type | Indicator |
Process | # Course Hours Assigned |
Output | # Courses Hours Taught |
Outcome | # Students Passed in Courses |
Supervision
This subcategory on supervising students. There is flexibility in the definition of’ supervision’, which may include mentoring students and supervision of student theses, as well as in the definition of’ students’, which may include bachelor, master, and, if applicable, doctoral students.
Student Supervision
Indicator Type | Indicator |
Process | # Students Being Supervised |
Output | # Students Supervised |
Outcome | # Supervised Student Theses # Supervised Students Graduated |
Skills
This subcategory focuses on skill development by researchers. There is flexibility in the definition of ‘courses’ and ‘certificates’, which may include variations in course scope and duration. The courses and certificates may cover any research, education, leadership, and valorisation skills.
Skills Development
Indicator Type | Indicator |
Process | # Skills Courses Being Followed |
Output | # Skills Courses Completed |
Outcome | # Skills Certificates Obtained |
People
This subcategory focuses on supervising staff. There is flexibility in the definition of ‘supervision’ which may include mentoring staff and supervision of (post)doctoral theses, as well as ‘staff’ which may include local and visiting (post)doctoral and senior researchers and other staff. There is also flexibility in the definition of ‘theses’, which may include (post)doctoral theses, as well as ‘projects’, which may include research and non-research projects managed by supervised staff.
Staff Supervision
Indicator Type | Indicator |
Process | # Staff Being Supervised |
Output | # Staff Supervised |
Outcome | # Supervised Staff Theses # Supervised Staff Projects |
Projects
This subcategory focuses on managing projects. There is flexibility in the definition of ‘projects’ which may be funded or non-funded but should involve official management responsibility.A project which is ‘successfully evaluated’ has been formally reviewed and successfully closed.
Project Management
Indicator Type | Indicator |
Process | # Projects Being Managed |
Output | # Projects Completed |
Outcome | # Projects Successfully Evaluated |
Organisation
This subcategory focuses on managing organisational units. There is flexibility in the definition of ‘units’ which may include a team, group, institute, faculty, or university. The ‘positions’ should involve official management responsibility. There is also flexibility in the selection of ‘unit management outputs’ and ‘unit management outcomes’ which could be taken from relevant outputs and outcomes already identified in the RAF and should be formally agreed within the organisation.
Unit Management
Indicator Type | Indicator |
Process | # Unit Management Positions Assigned |
Output | # Unit Management Positions Completed # Agreed Unit Management Outputs |
Outcome | # Agreed Unit Management Outcomes |
Recognition
This subcategory on the recognition of researchers through expert positions. There is flexibility in the definition of ‘expert positions’ including invited strategic, advisory, and honorary roles, which recognise researchers for their expertise and experience. There is also flexibility in the definition of ‘expert position outputs’ which may include meetings, presentations, and reports, as well as ‘expert position outcomes’, depending on the type of expert position. The ‘achievement awards’ may be for any recognised research, education, leadership, and valorisation contributions.
Expert Positions
Indicator Type | Indicator |
Process | # Expert Positions Assigned |
Output | # Expert Positions Completed # Expert Position Outputs |
Outcome | # Expert Position Outcomes # Expert Achievement Awards |
Communication
This subcategory focuses on research communication via public writing and public speaking. There is flexibility in the definition of ‘publications’, which may include print and social media and ‘appearances’ which may be physical or digital and include talks, conferences, workshops, and recordings. There is also flexibility in the definition of ‘accessed’ which may include readership, attendees, views, and downloads, and ‘cited’ which may include mentions in print and social media.
Public Writing
Indicator Type | Indicator |
Process | # Publications Being Drafted |
Output | # Publications Published |
Outcome | # Publications Accessed # Publications Cited |
Public Speaking
Indicator Type | Indicator |
Process | # Appearances Planned |
Output | # Appearances Made |
Outcome | # Appearances Accessed # Appearances Cited |
Engagement
This subcategory focuses on on intersectoral and citizen engagement. There is flexibility in the definition of ‘intersectoral’ which may include academic, public, and private organisations and ‘collaborations’ which may include research collaboration, staff secondments and exchanges, volunteering and advocacy, and policy development. There is also flexibility in the definition of ‘citizen science’, which may include any research activities with citizens and encompass activities in projects with a citizen science dimension or full projects on citizen science. There is also flexibility in the selection of ‘intersectoral outputs’ and ‘intersectoral outcomes’ as well as ‘citizen science outputs’ and ‘citizen science outcomes’ which could be taken from relevant outputs and outcomes in the RAF.
Intersectoral Engagement
Indicator Type | Indicator |
Process | # Intersectoral Engagements |
Output | # Intersectoral Outputs |
Outcome | # Intersectoral Outcomes # Organisations Engaged |
Citizen Engagement
Indicator Type | Indicator |
Process | # Citizen Science Activities Ongoing |
Output | # Citizen Science Activities Completed # Citizen Science Outputs |
Outcome | # Citizen Science Outcomes # Citizen Scientists Engaged |
Innovation
This subcategory focuses on the innovation of research through research exploitation and entrepreneurial spirit. There is flexibility in the definition of ‘being legalised’, including defining Intellectual Property Rights (IPR), patenting, and licensing of research outputs, which may include innovative products and services. There is also flexibility in the size of spin-offs and start-ups as well as the definition of ‘employees’ in terms of number of people or Full Time Equivalent (FTE).
Research Exploitation
Indicator Type | Indicator |
Process | # Research Outputs Being Legalised |
Output | # Research Outputs with Defined IPR # Research Outputs Patented |
Outcome | # Research Outputs with Licenses |
Entrepreneurial Spirit
Indicator Type | Indicator |
Process | # Spin-offs/Start-ups Being Created |
Output | # Spin-offs/Start-ups Created |
Outcome | # Spin-off/Start-up Employees # Spin-off/Start-up Products # Spin-off/Start-up Service |
2. Open Science Researcher Assessment Framework: Indicators and Metrics
Proposal Development
This focuses on proposals for research projects to an RPO or RFO which are openly available. There is flexibility in how the proposals are made ‘openly available’.
Indicator Type | Indicator |
Process | # Developing Project Proposals Openly Available |
Output | # Submitted Project Proposals Openly Available |
Outcome | # Granted Project Proposals Openly Available |
Methods Development
This subcategory focuses on methods to conduct research which are openly available. There is flexibility in how the method sets are actually made ‘openly available’ for use by the public.
Indicator Type | Indicator |
Process | # Developing Method Sets Openly Available |
Output | # Finalised Method Sets Openly Available |
Outcome | # Openly Available Method Sets Implemented # Openly Available Method Sets Accessed # Openly Available Method Sets Cited |
Data
This subcategory focuses on research data planning, management, and peer review, which are openly available. There is flexibility in the inclusion or exclusion of a focus on FAIR and how data management plans, data sets, and data peer reviews are made ‘openly available’ for public use.
Data Planning
Indicator Type | Indicator |
Process | #(FAIR) Developing Data Management Plans Openly Available |
Output | # (FAIR) Finalised Data Management Plans Openly Available |
Outcome | # (FAIR) Implemented Data Management Plans Openly Available |
Data Management
Indicator Type | Indicator |
Process | # Developing (FAIR) Data Sets Openly Available |
Output | # Finalised (FAIR) Data Sets Openly Available # Archived (FAIR) Data Sets Openly Available |
Outcome | # Openly Available (FAIR) Data Sets Accessed # Openly Available (FAIR) Data Sets Cited |
Data Review
Indicator Type | Indicator |
Process | # Draft (FAIR) Data Set Peer Reviews Openly Available |
Output | # Submitted (FAIR) Data Set Peer Reviews Openly Available |
Outcome | # Accepted (FAIR) Data Set Peer Reviews Openly Available |
Software
Software Development
Indicator Type | Indicator |
Process | # Developing Software Sets Openly Available |
Output | # Finalised Software Sets Openly Available # Archived Software Sets Openly Available |
Outcome | # Openly Available Software Sets Accessed # Openly Available Software Sets Cited |
Software Review
Indicator Type | Indicator |
Process | # Draft Software Set Peer Reviews Openly Available |
Output | # Submitted Software Set Peer Reviews Openly Available |
Outcome | # Accepted Software Set Peer Reviews Openly Available |
Publications
Publication Drafting
Indicator Type | Indicator |
Process | # Draft Publications Openly Available |
Output | # Submitted Publications Openly Available |
Outcome | # Published Publications Openly Available # Openly Available Publications Accessed # Openly Available Publications Cited |
Publication Review
Indicator Type | Indicator |
Process | # Draft Publication Peer Reviews Openly Available |
Output | # Submitted Publication Peer Reviews Openly Available |
Outcome | # Accepted Publication Peer Reviews Openly Available |
Materials Development
This subcategory focuses on research materials which are openly available. There is flexibility in how the material sets aremade’openly available’ depending on the type of materials.
Indicator Type | Indicator |
Process | # Developing Material Sets Openly Available |
Output | # Finalised Material Sets Openly Available |
Outcome | #Implemented Material Sets Openly Available # Openly Available Material Sets Accessed # Openly Available Material Sets Cited |
Courses
Course Development
Indicator Type | Indicator |
Process | # Open Science Courses Being Developed # Developing Courses Openly Available |
Output | # Open Science Courses Finalised # Finalised Courses Openly Available |
Outcome | # Open Science Courses Implemented # Implemented Courses Openly Available |
Resources
This subcategory focuses on educational resources which are on Open Science or openly available. There is flexibility in the definition of resources on ‘Open Science’ which may focus on many or specific Open Science practices as well as how the resources are made ‘openly available’.
Resource Development
Indicator Type | Indicator |
Process | # Open Science Resources Being Developed #Developing Resources Openly Available |
Output | # Open Science Resources Finalised # Finalised Resources Openly Available |
Outcome | # Open Science Resources Implemented # Implemented Resources Openly Available |
Teaching
Student Teaching
Indicator Type | Indicator |
Process | # Open Science Course Hours Assigned |
Output | # Open Science Course Hours Taught |
Outcome | # Students Passed in Open Science Courses # Students Passed in Openly Available Course |
Supervision
This subcategory focuses on supervising students in Open Science and making student theses openly available. There is flexibility in the definition of ‘Open Science’ which may include many or specific Open Science practices and how thestudent theses are ‘openly available’.
Student Supervision
Indicator Type | Indicator |
Process | # Students Being Supervised in Open Science |
Output | # Students Supervised in Open Science |
Outcome | # Supervised Student Theses Openly Available # Supervised Students in Open Science Graduate |
Skills
This subcategory focuses on skills development in Open Science by researchers. There is flexibility in the definition of ‘courses’ and ‘certificates’ which may include variation in course scope and duration and ‘Open Science’ which may focus on many or specific Open Science practices. Courses and certificates may cover research, education, leadership, and valorisation skills.
Skills Development
Indicator Type | Indicator |
Process | # Open Science Skills Courses Being Followed |
Output | # Open Science Skills Courses Completed |
Outcome | # Open Science Skills Certificates Obtained |
People
Staff Supervision
Indicator Type | Indicator |
Process | # Staff Being Supervised in Open Science |
Output | # Staff Supervised in Open Science |
Outcome | # Supervised Staff Theses Openly Available # Supervised Staff Projects involving Open Science |
Projects
This subcategory focuses on managing projects involving Open Science. There is flexibility in the definition of ‘involving Open Science’ which may include variation in the scope and duration of projects and ‘Open Science’ which may include many or specific Open Science practices.
Project Management
Indicator Type | Indicator |
Process | # Projects involving Open Science Being Managed |
Output | # Projects involving Open Science Completed |
Outcome | # Projects involving Open ScienceSuccessfully Evaluated |
Organisation
This subcategory focuses on managing organisational units involving Open Science. There is flexibility in the definition of ‘Open Science’ which may include many or specific Open Science practices. There is also flexibility in the selection of ‘unit management outputs’ and ‘unit management outcomes’ involving Open Science, which could be taken from relevant outputs and outcomes already identified in the RAF and should be formally agreed within the organisation.
Unit Management
Indicator Type | Indicator |
Process | # Unit Management Positions in Open Science Assigned |
Output | # Unit Management Positions in Open Science Completed #Agreed Unit Management Outputs involving Open Science |
Outcome | # Agreed Unit Management Outcomes involving Open Science |
Recognition
This subcategory focuses on the recognition of researchers through expert positions in Open Science. There is flexibility in the definition of ‘Open Science’ including many or specific practices and how ‘expert position outputs’ and ‘expert position outcomes’ are ‘openly available’.
Expert Positions
Indicator Type | Indicator |
Process | # Expert Positions in Open Science Assigned |
Output | # Expert Positions in Open Science Completed # Open Science Expert Position Outputs # Expert Position Outputs Openly Available |
Outcome | # Expert Position Outcomes Openly Available # Expert Achievement Awards for Open Science |
Communication
This subcategory focuses on research communication via public writing and public speaking, which is openly available. There is flexibility in how the publications and appearances are made ‘openly available’ and also in the definition of ‘Open Science’ including many or specific practices.
Public Writing
Indicator Type | Indicator |
Process | # Draft Publications Openly Available |
Output | # Published Publications Openly Available |
Outcome | # Openly Available Publications Accessed # Openly Available Publications Cited |
Public Speaking
Indicator Type | Indicator |
Process | # Appearances on Open Science Planned |
Output | # Appearances on Open Science Given |
Outcome | # Appearances on Open Science Accessed # Appearances on Open Science Cited # Appearances Openly Available |
Engagement
Intersectoral Engagement
Indicator Type | Indicator |
Process | # Intersectoral Engagements involving Open Science |
Output | # Intersectoral Outputs involving Open Science |
Outcome | # Intersectoral Outcomes involving Open Science # Organisations Engaged for Open Science |
Citizen Engagement
Indicator Type | Indicator |
Process | # Citizen Science Activities involving Open Science Ongoing |
Output | # Citizen Science Activities involving Open Science Completed # Citizen Science Outputs involving Open Science |
Outcome | # Citizen Science Outcomes involving Open Science # Citizen Scientists Engaged in Open Science |
Innovation
Research Exploitation
Indicator Type | Indicator |
Process | # Openly Available Research Outputs Being Legalised |
Output | # Openly Available Research Outputs with Defined IPR # Openly Available Research Outputs Patented |
Outcome | # Research Outputs with Open Licenses |
Entrepreneurial Spirit
Indicator Type | Indicator |
Process | # Open Science Spin-offs/Start-ups Being Created |
Output | # Open Science Spin-offs/Start-ups Created |
Outcome | # Open Science Spin-off/Start-up Employees # Open Science Spin-off/Start-up Products # Open Science Spin-off/Start-up Services |