Judge Open Science by its Outcomes, Not its Outputs

• June 18, 2024

Rethinking Open Science Monitoring: Beyond Counting Publications

In the wake of numerous policies aimed at fostering open science, a significant focus has emerged on monitoring its progress. One notable initiative is the French Open Science Monitoring Initiative, which compiles data from various countries and institutions and proposes general principles for monitoring open science. This initiative coincides with a recently launched Unesco consultation on the subject.

Current monitoring efforts primarily track the uptake of open-science policies and outputs, such as publications and data. However, this approach is analogous to understanding fungi by counting mushrooms: it overlooks the intricate network of activities and interactions beneath the surface that drive the visible outcomes.

Open science seeks to transform not only the practices of research but also its motivations and values. It extends far beyond publishing, encompassing public engagement and evaluation. Focusing solely on easily quantifiable metrics neglects these critical activities, which promote equity and share the benefits of research with society.

The Pitfalls of Narrow Monitoring

Limiting monitoring to quantifiable outputs can have unintended and detrimental consequences. For instance, an increase in open access publications might appear as progress, making more information freely accessible. However, if this openness relies on pay-to-publish models, it creates inequality. Researchers who cannot afford publication fees become less visible, and unscrupulous publishers may weaken review processes to boost profits.

These issues disproportionately affect researchers in marginalized scholarly communities, undermining the values of open science and weakening less affluent research systems. Moreover, this approach reinforces a publish-or-perish culture that prioritizes quantity over the thoughtful creation of knowledge intended for meaningful purposes. Lay readers should not have to navigate an overwhelming volume of research outputs.

Similar challenges arise in data sharing. There is limited evidence of meaningful reuse of datasets outside of specific disciplines like genomics. Issues of quality control are poorly understood, and long-term storage and curation remain problematic. Thus, counting datasets without considering their utility and impact offers little insight.

A Need for Contextual and Value-Driven Monitoring

Current monitoring efforts, while useful, are too narrow and risk overlooking important engagement activities and the broader benefits of open science. To fully realize the promises of open science, a more comprehensive approach that embraces context and values is essential.

Indicators need to specify their particular pathways to open science. Different forms of open access publishing, such as gold, diamond, and green open access, each have unique normative implications and should not be aggregated into a single measure.

Furthermore, monitoring should extend beyond the point of creation or engagement to consider their effects and outcomes. It is crucial to track who bears the costs and who benefits from open science policies.

Learning from History and Mapping Connections

History offers valuable lessons for monitoring processes. The OECD’s Frascati Manual, developed in 1963, focused on measuring inputs and outputs in science, technology, and innovation. However, this approach failed to capture the processes, drivers, and outcomes of innovation. In response, the OECD launched the Oslo Manual in 1992, which employed surveys to gather more nuanced data.

Similarly, open science monitoring should reveal the connections between researchers and stakeholders, along with behavioral changes and underlying motivations. Surveys are a primary method for capturing this type of information.

A pluralistic approach to monitoring, possibly based on surveys and narratives, can link practices to value-driven outcomes. This shift aligns with the movement in research assessment away from journal impact factors and citations towards narrative accounts of impact.

If open science represents a systemic transformation of the research system, including its values, then its monitoring strategies must match this ambition. Open science is about more than producing accessible and reproducible research; it aims to effect meaningful change in science. Monitoring should, therefore, track contributions towards collective benefits, integrity, and equity in science.

Authors

Ismael Rafols is the Unesco Chair on Diversity and Inclusion in Global Science at Leiden University, and Louise Bezuidenhout is a senior researcher at the Centre for Science and Technology Studies, Leiden University.

Photo via WikiData

Original article can be found here