Skip to main content

Beyond journal metrics: Why it’s time to embrace more meaningful methods of research assessment

  • 2025-09-17 10:45
  • 11:45
  • Room: 81/R-003B - Science Gateway Auditorium B
  • Speaker:
    • Fiona Hutton, , Fiona Hutton is eLife’s Head of Publishing. Originally a research scientist, in 2001, while working as a postdoctoral researcher at Columbia University, Fiona signed the open letter to scientific publishers that called for a freely accessible library of published research and scholarly discourse in medicine and the life sciences. She then transitioned to work in science publishing, initially as Editor of Trends in Biochemical Sciences (TiBS), before holding positions at Wiley and then Cambridge University Press, as Head of STM OA Publishing & Executive Publisher. During this time, Fiona conceptualised a range of open-access publishing titles and series including Cambridge Prisms and Research Directions – on the foundations of open research principles. With a passionate belief that publishers have to evolve to maintain relevance and purpose within the scientific community, Fiona joined eLife in April 2022, bringing her expertise to the organisation as it moved towards launching the eLife Model for publishing. Fiona is an active member of the community and also sits on the Board of Directors of OASPA and the Steering Group of DORA. , eLife, https://elifesciences.org/, UK

With the increasing popularity of preprints, there are ongoing discussions among the research community about the need to apply peer review to help readers navigate new findings. eLife adopted such an approach in 2023, when we launched our model for publishing. The outputs are Reviewed Preprints, which include the original preprint, public reviews and an eLife Assessment that conveys the significance of the findings and strength of evidence, allowing readers to judge the research based on its own merits rather than where it is published.

Due to our efforts to challenge the status quo in publishing, our indexing status in Web of Science changed last year, meaning eLife no longer receives an Impact Factor. This was followed by concerns that eLife papers would no longer count toward funding or career progression opportunities. We therefore spoke to funders and institutions globally to better understand their position, and reported that 95% of respondents still consider eLife papers when evaluating research contributions. Our conversations highlighted that there is less consideration for the Impact Factor than perceived by the research community, and signalled broad support for more open science practices – showing that it’s time to move away from journal metrics in favour of more transparent and meaningful methods of assessment.

In this session, we will talk more about eLife Assessments, our conversations with the community and why it’s time to embrace innovative approaches to research assessment that better serve science and scientists. We also invite further discussion and participation from the audience.

Designing What Matters: Co-Creating Open Science Dashboards

  • 2025-09-17
  • 2025-09-17 10:45
  • 10:45
  • Room: 81/R-003B - Science Gateway Auditorium B
  • Speaker:
    • Giulia Malaguarnera, , Giulia Malaguarnera is a researcher with a PhD in Neuropharmacology, serving as Outreach and Engagement Officer at OpenAIRE. In her role at OpenAIRE, she focuses on fostering stakeholder engagement and promoting Open Science practices, with particular attention to research assessment. She is actively involved in European Open Science Cloud (EOSC) initiatives and contributes to projects such as GraspOS, where she works on dataspace architecture and user engagement strategies.Giulia is also a member of the CoARA Working Group Towards Open Infrastructures for Responsible Research Assessment, dedicated to developing sustainable e-infrastructure policies. A former President of Eurodoc and member of the Marie Curie Alumni Association, she is committed to advancing transparent and inclusive research evaluation systems across Europe. , OpenAIRE, https://www.openaire.eu/, Italy
    • Ioanna Grypari, , Dr. Ioanna Grypari is a technical project manager at OpenAIRE and Athena Research Center (ARC) in Greece, with a strong background in Econometrics. She leads cross-functional teams in developing indicators and platforms for Open Science and Research & Innovation, leveraging data analytics, extensive databases, and AI workflows. Dr. Grypari manages the indicators team for OpenAIRE's monitoring services and spearheads the development of data-related products for EC projects with a focus on assessing the societal impact of research (e.g., Data4Impact, IntelComp). She is the coordinator of Horizon Europe Project PathOS, focusing on identifying and measuring the causal effects of Open Science. , OpenAIRE, Athena Research Center, Greece
    • Alessia Bardi, , Alessia Bardi is a researcher in computer science at the Institute of Information Science and Technologies of the Italian National Research Council with a PhD in Information Engineering. Her research activities focus on infrastructures for scholarly communication, Open Science, and Scholarly Knowledge Graphs. She he is actively working on the OpenAIRE infrastructure as member of the technical team and service manager of OpenAIRE CONNECT, closely collaborating with research initiatives and infrastructures to deliver customizable discovery and impact monitoring portals. , ISTI-CNR, https://www.isti.cnr.it/en/, Italy
    • Tereza Szybisty, , Tereza Szybisty is a dedicated Open Science advocate with experience in Open Science policy-making across various levels of scientific organizations. She holds a PhD in Management and has worked as an Open Science Specialist, Policy Officer, and trainer of early-career researchers.Tereza is the founder of the Stop Predatory Practices initiative, which raises awareness of unethical academic publishing. At OpenAIRE, she serves as the Research Project Manager for the EOSC Track project and leads communication, engagement, and training activities in the PathOS project. , OpenAIRE, https://www.openaire.eu/, Czechia
    • Zenia Xenou, , Zenia Xenou is an Engagement and Training Officer at OpenAIRE AMKE. She holds an MSc in Biomedical Engineering and a Bachelor's degree in Mechanical Engineering from the Cyprus University of Technology. With extensive experience as a researcher in multiple R&D projects, Zenia has driven product design and manufacture, and championed organizational development in several positions. Currently, she serves as the Service Manager for the OpenAIRE Researcher Profile, an innovative tool designed to empower researchers to showcase their contributions beyond publications and promote responsible research assessment. , OpenAIRE, https://www.openaire.eu/, Greece

Designing What Matters: Rethinking Open Science Monitoring Together
How should we measure and reward Open Science in ways that matter to researchers, institutions, and funders? In this 90-minute co-creation workshop, participants will build their own Open Science dashboard using real indicators from PathOS and GraspOS Horizon Projects. Working in groups, you’ll craft visual prototypes, debate what to measure and why, and explore how stories, not just scores, can drive meaningful change in research assessment. Join us to design smarter, fairer ways to track Open Science impact.

From Competition to Collaboration: A National Model for Sustainable and Interoperable Research Data Repositories

  • 2025-09-17 10:45
  • 11:15
  • Room: 81/R-003A - Science Gateway Auditorium A
  • Speaker:
    • Agnieszka Cybulska-Phelan, , Agnieszka Cybulska-Phelan - open research data specialist at Open Science Platform team, Interdisciplinary Centre for Mathematical and Computational Modelling (ICM), University of Warsaw. Carries out educational and training activities on RDM and OS. Responsible for the day-to-day operations of research data repositories. Graduate of the Data Steward postgraduate program at the University of Vienna. , University of Warsaw, https://en.uw.edu.pl/, Poland
    • , Jakub Szprot - Head of the Open Science Platform at the Interdisciplinary Centre for Mathematical and Computational Modelling (ICM), University of Warsaw. He oversees nationwide IT services that provide open access to scientific literature and data. He’s also responsible for policy advocacy and supervises various training and educational activities in open science and research data management. He has participated in numerous European and Polish projects concerning research infrastructures, open access, research data management, and digital humanities. He co-authored and edited reports on different aspects of open science. He has been a member of committees and expert groups of UNESCO, the European Commission, and the Polish Ministries of Science and Higher Education and Education and Science. As a DARIAH-PL National Coordinator, he developed digital research infrastructure for the arts and humanities on both the national and European levels. , University of Warsaw, https://en.uw.edu.pl/, Poland

This presentation explores a collaborative national network of research data repositories in Poland, where institutional, disciplinary, and generalist infrastructures coexist rather than compete. The model offers a practical example of how interoperability and central coordination can support researchers and institutions in selecting the most appropriate repository type—without duplicating efforts or fragmenting services.

Drawing on a growing ecosystem currently comprising four main repositories (including a generalist platform hosting over 50 institutional dataverses), the presentation demonstrates how this architecture helps address key challenges in the research data landscape:
- ensuring long-term sustainability (technical, operational, and financial),
- alleviating staffing shortages,
- and motivating researchers to share their data.

The model relies on shared infrastructure, harmonized metadata exchange, and cross-repository visibility, enabling researchers to deposit data in the repository best suited to their needs—while benefiting from aggregated institutional presence and increased discoverability.

By presenting this approach, we aim to share insights into a scalable, community-driven strategy that reduces overheads, strengthens national coordination, and enhances the openness of research data practices. By highlighting a collaborative infrastructure model, this presentation contributes to the broader conversation on how we can reimagine openness through shared responsibility, inclusive governance, and sustainable practices.

Open Science Infrastructures: The Case of Austrian RDM Policies

  • 2025-09-17 10:45
  • 10:45
  • Room: 81/R-003A - Science Gateway Auditorium A
  • Speaker:
    • Celine Wawruschka, , Celine Wawruschka is an Open Science policy advisor at Austrian Universities (uniko) and is in charge of the stakeholder platform Open Science Austria (OSA). In the field of open science, she previously set up the open access journal ‘Medieval Worlds’ at the Institute for Medieval Studies of the Austrian Academy of Sciences and was editor-in-chief for several years. Furthermore, she was responsible for the project development and project coordination of Citizen Science at the Vienna Adult Education Centres and in this context developed and managed the project ‘Climate-friendly everyday practices: a participatory science communication project for adolescents and young adults with migrant background’. , Open Science Austria (OSA), https://www.osa-openscienceaustria.at/en/

The initiative of the European Research Area (ERA) aims at creating a unified research and innovation space across Europe. One of the key objectives of ERA is to promote open access to research results and data, thus encouraging transparency, reproducibility, and wider dissemination of knowledge. In order to achieve an alignment with this ERA objective (and others), the European Research Area National Action Plan (ERA-NAP) supports the individual EU member states to contribute to a European Open Science infrastructure.

In this way, national measures for an open science infrastructure are promoted, which ultimately aim to create an open science infrastructure within a European framework. Open Science policies are the means of choice for the creation of such a structure at national level – because only they guarantee the joint endeavour to bundle the interests and needs of all stakeholders.

In my presentation, I outline the goals and challenges of working on such a national Open Science policy using the example of research data management in the Austrian Higher Education Area, which a working group of the Austrian University Conference has addressed, in line with the ERA-NAP. How do different types of universities define research data? What restrictions regarding openness do they wish to reserve for themselves - and for what reasons? Up to what point can a framework policy form a common ground – and at what point should individual standards be applied?

OSTrails: Connecting Tools and Communities for a Federated Open Science Ecosystem

  • 2025-09-17 10:45
  • 11:45
  • Room: 81/R-003A - Science Gateway Auditorium A
  • Speaker:
    • Elli Papadopoulou, , Elli is a digital librarian and research associate at the Institute for the Management of Information Systems of Athena Research Center. Her academic background is in Library Science and Information Systems (BSc) and in Public Policy and Public Management (MSc). Elli is active in Open Science from 2013 and for the past 7 years supports awareness, adoption and implementation of policies and best practices across the Greek academic and research communities serving as the OpenAIRE representative in Greece (NOAD-GR). Elli is also a liaison for Services & Technology Standing Committee in the OpenAIRE Executive Board. She enjoys collaborating with colleagues on Open and FAIR Research Data Management through RDA groups that she co-chairs, managing the ARGOS DMP service and co-chairing the EOSC-A FAIR Metrics and Digital Objects Task Force. In EOSC, she has contributed to policy, technical conceptualisation and implementation activities in different projects since the very beginning of EOSCpilot. This year, she embarked on a new journey serving as deputy coordinator for the INFRA-EOSC OSTrails project. Her research interests are focused on Responsible Research and Innovation, Open Science and FAIR Research Data Management, and AI. , Athena Research Center, https://www.athenarc.gr/en/home, Netherlands

The EU-funded OSTrails project is building a federated Open Science infrastructure by enabling researchers and institutions to discover, plan, track, and assess their work in transparent and interoperable ways. With 41 partners and 25 pilots—including cross-domain, national, and Horizon Europe testbeds—OSTrails is piloting the practical integration of over 80 interoperable tools and services. This diversity reflects the scale needed to support a truly federated Open Science ecosystem.

At the core of the project are two key enablers: a modular Interoperability Reference Architecture and a tool independent Plan-Track-Assess (PTA) framework. Together, these provide a shared foundation for aligning diverse research management services—from Data Management Planning platforms to Scientific Knowledge Graphs and FAIR assessment tools—across distributed infrastructures.

To ensure long-term impact, OSTrails has launched a comprehensive training and capacity-building programme, equipping research communities and service providers with the knowledge and tools to adopt and extend project outcomes. These efforts build on the co-designed pilots, which validate interoperability across varied institutional, disciplinary settings.

For researchers, OSTrails simplifies cross-institutional Research Data Management workflows and clarifies pathways to Open Science best practices. For service providers, it offers a practical model for aligning with EOSC while retaining domain-specific autonomy.

This presentation will showcase how OSTrails contributes to a federated EOSC ecosystem by delivering standards-driven solutions that prioritise interoperability and FAIRness as essential components of a sustainable and inclusive infrastructure, supporting both federation and the development of the Web of FAIR Data and Services.

Strange bedfellows, indeed: How the impact of the current political moment could accelerate an open science future

Open Science in Turbulent Times: Co-Opted or Catalyzed?

Amid political attacks on science in the US, Open Science finds itself championed for unexpected reasons. Is this a co-optation, or a catalyst for global change? Join a provocative panel exploring how shifting power dynamics, funding priorities, and ideologies are reshaping the future of international research, openness, and scientific integrity.

Valuing what matters: Developing new approaches to research assessment

  • 2025-09-17 10:45
  • 10:45
  • Room: 81/R-003B - Science Gateway Auditorium B
  • Speaker:
    • Mathijs Vleugel, , Mathijs Vleugel serves as the head of Helmholtz Open Science Office since September 2024. He supports the Helmholtz Association, its research centers, and the broader scientific community in shaping the cultural shift toward Open Science. Mathijs is a steering group member of the focus area "Digitality in Science" of the Alliance of German Science Organizations and currently serves as the coordinator of the CoARA National Chapter Germany. , Helmholtz Association, https://www.helmholtz.de/en/, Berlin

Advancing the quality and impact of science requires a fundamental reimagining of the practices and criteria used to evaluate researchers and their institutions. The Helmholtz Open Science Office supports the cultural shift toward Open Science within the Helmholtz Association, Germany's largest scientific organization. As a signatory of the Coalition on Advancing Research Assessment (CoARA) and coordinator of the CoARA National Chapter Germany, the Helmholtz Open Science Office is involved in several activities that aim to reform research assessment.

This contribution describes key initiatives that are currently taking place within the Helmholtz Association. Together, they aim to broaden the recognition of diverse research activities and output formats as valuable scholarly contributions, in alignment with the principles of research integrity and responsible research evaluation. We present the development of a "Quality Indicator for Data and Software Products”, which is designed to capture quality of research outputs beyond traditional text publications and to promote their visibility. In addition, we present the implementation of an award for sustainable research software and the work of a new Helmholtz Task Group on Research Assessment, which provides a platform for the exchange and development of modern and quality-oriented research assessment practices.

Our contribution situates these activities within broader initiatives in Germany, such as the focus area "Digitality in Science" of the Alliance of German Science Organizations, and describes how they interact with international initiatives like CoARA and the Barcelona Declaration on Open Research Information.

Who Watches the Watchers? Rethinking Open Science Monitoring

  • 2025-09-17 10:45
  • 11:15
  • Room: 81/R-003B - Science Gateway Auditorium B
  • Speaker:
    • Batool Almarzouq, , I'm an honorary research fellow at the University of Liverpool and a Research Project Manager for AI for Multiple Long Term Conditions: Research Support Facility (AIM RSF) at the Alan Turing Institute. AIM RSF is part of a £23 million investment by the NIHR in AI, aimed at connecting researchers across consortia to ensure long-term, real-world impact for multi-morbidity. In addition, I have the privilege of being a part of the Open Science expert group by the International Association of Universities (IAU), representing Arab countries. The IAU, initiated by UNESCO, has been a vital global institution since 1950. My involvement in the Expert Group on Open Science involves developing recommendations and sharing best practices to guide universities in their transition towards Open Science. I advocate for transforming cultural norms to facilitate the adoption of open research practices, tools, and ethos, while addressing the existing power dynamics and inequalities in knowledge production. I believes that Open science is fundamentally about  decolonisation by challenging the legacy of settler colonialism, which often marginalised Indigenous knowledge systems, and by promoting the integration and respect of these diverse perspectives in the broader scientific discourse I lead the Open Science community Saudi Arabia (OSCSA) which introduces and contextualises Open Science practices in Arabic-speaking countries

As the open science movement gains momentum, we face a critical paradox: the very frameworks designed to promote transparency and accessibility risk perpetuating the same power imbalances they aim to dismantle. This talk explores the uncomfortable truth about how current monitoring approaches in open science often mirror colonial knowledge production patterns, with the Global North continuing to set standards and metrics for the entire scientific community.
I examine how well-intentioned monitoring frameworks can inadvertently reinforce existing hierarchies in knowledge production. While we generate volumes of "open" research about equity and inclusion, many of these efforts remain tethered to institutions that paradoxically contribute to global inequities. The talk challenges us to move beyond superficial metrics and performative inclusion, advocating instead for a decolonial approach that centres on epistemic justice and acknowledges historical power imbalances in knowledge production.
I propose shifting our focus from quantifying outputs to supporting authentic dialogue about research practices and their broader societal implications. This perspective invites us to critically examine who holds the power to define and measure "openness" in science, and how we might reimagine monitoring frameworks to truly serve the global scientific community.