ICOR Public Meeting: Harmonising Research Metrics and Indicators Across the Ecosystem

• December 4, 2024

On 13 November 2024, the Incentivizing Collaborative and Open Research (ICOR) initiative held its seventh public meeting, focusing on strategies for harmonising research metrics to communicate impact effectively across various contexts. The session underscored the intrinsic connection between responsible research assessment and open scholarship, emphasising the need for balanced and complementary indicators that underpin hiring, promotion, and funding decisions. See more here.

The meeting, introduced by Anna Hatch, Program Officer for Open Science Strategy at the Howard Hughes Medical Institute, featured presentations by three experts who shared insights into evolving approaches to research metrics and the broader goal of fostering equity and inclusivity in research assessment.

Research Metrics: Shared Challenges and Solutions
Kelly Cobey, University of Ottawa Heart Institute and Co-Chair of DORA

Kelly Cobey began by critiquing the traditional reliance on metrics such as journal impact factors and publication counts, which often prioritise quantity over quality. These metrics, she argued, discourage innovation and collaboration while undervaluing diverse contributions to research.

Representing the Declaration on Research Assessment (DORA), Kelly highlighted the organisation’s efforts to promote fairer evaluation systems. She introduced DORA’s guidelines for responsible use of quantitative indicators, addressing their limitations and potential biases. The discussion also featured TARA (Tool to Advance Research Assessment), a project supporting academic institutions in reforming career assessments through the development of new policies and practices.

Kelly concluded by emphasising the importance of international collaboration to refine and implement equitable research metrics, ensuring that contributions like patient engagement, diversity, and open science are recognised alongside traditional outputs.

The Rise of Structured CV Narratives
Karen Stroobants, CultureBase Consulting

Karen Stroobants, Vice Chair of the Coalition for Advancing Research Assessment (COARA), explored the role of qualitative evaluation in research assessment. She discussed the growing adoption of structured CV narratives, which shift the focus from quantitative metrics to personal accounts of research impact.

This approach allows researchers to present their work holistically, highlighting diverse contributions and reducing biases in hiring and promotion processes. Karen noted that many European funders and institutions are adopting narrative CVs to foster a more inclusive research culture.

While expressing concern about an overemphasis on metrics, Karen remained optimistic, envisioning a future where qualitative assessments encourage researchers to reflect on their work’s broader impact, ultimately driving positive changes in research practices.

Redefining Data Metrics
Iratxe Puebla, Make Data Count

Iratxe Puebla introduced Make Data Count, an initiative developing open metrics for assessing data usage. She emphasised the unique nature of research data, which is used and reused in ways that traditional metrics fail to capture.

Drawing from funder and institutional case studies, Iratxe stressed the importance of combining quantitative and qualitative approaches to data metrics, contextualising their use to ensure meaningful evaluations. She highlighted the need for iterative development of metrics, considering factors like data quality and how it is engaged with by the research community.

Her insights extended beyond data-specific metrics, offering valuable lessons for designing balanced and thoughtful indicators across all aspects of research assessment.

A Holistic Path Forward

The meeting reinforced the shared goal of creating equitable and effective research metrics. As the research community transitions towards open scholarship and responsible assessment, a balance between quantitative and qualitative approaches is essential. By avoiding overreliance on numerical metrics and contextualising their application, we can ensure that research assessments reflect the true impact and diversity of scholarly contributions.

Through collaboration and iterative refinement, ICOR and its partners aim to drive meaningful reform, fostering a research culture that values openness, transparency, and inclusivity.

Source: https://incentivizingopen.org/2024/12/harmonising-research-metrics-and-indicators-across-the-ecosystem-2/

Photo via  CC BY E Wood, Research Services, Waipapa Taumata Rau | The University of Auckland