Context/Needs Based Statement

The standard open science/open source science guidance frameworks (FAIR, CARE, OSF, etc.) are largely administrative, taking place during the pre-award and post-closeout stages of a research project.  After decades of advances toward the adoption of OS principles, advancements in OS praxis remain scarce.  We propose the implementation of a uniquely metaresearch practice as a methodological standard for open science research: open science data review (OSDR).  

Summary

In theory, OSDR prescribes a detailed investigation into existing data(sets) as the primary commitment of an open science project.  The approach is aimed at strengthening the practice of open science as its own form of investigation, distinguishing it from scientific traditions and research efforts that are destined for closed access outputs.  To achieve its data analysis objective, the OSDR framework examines biases in existing datasets.  The two-fold aims are (1) to establish an OS-specific practice in the lab and (2) to encourage researchers to think more critically about the integrity, value, relevance, and strength of research data in its published form.

There are at four types of data review that can be undertaken as a precursor to new work or during the course of an open science research project:

1. Reproduction studies;

2. Replication studies; 

3. Robustness studies; and

4. Critical analysis.

Objectives

Centering OS projects on the data that underpins most-cited works and findings that are otherwise taken for granted opens a pathway for scientists to challenge the collection, organization, and treatment of research data, data artifacts, and metadata.  OSDR reproducibility, robustness, and replication studies establish the basis for further data collection, markup, and tagging.  

Critical analysis, in particular, invites a deeper investigation into data characteristics that are often relegated to footnotes, discussion sections, or not mentioned at all in primary literature–cherry picking, p-hacking, dark data management, negative data reporting, technical categorization & classification, and other types of wrangling that impact results.  However, all types of open science data review may reveal whether cited works suffer from undue bias, analytical weaknesses, or more serious integrity issues.  For example, reproduction can help to unpack assumptions and/or flaws regarding  instrumentation and data analysis design.  Replication can validate original results as a direct form of analysis, and further elucidate data collection and wrangling strategies.  Robustness challenges data integrity or relevance by testing it with novel analytical frameworks.  Critical analysis provides a meta pathway for OS investigators to scrutinize published data in any other way that does not involve experimentation.

Advancing openness, transparency, and reproducibility

On its face, open science data review promotes openness, transparency, and reproducibility.  It  further offers a modernized framework that delivers a more precise insight mechanism into open science work by focusing on the data of cited studies, rather than the analysis.  For example, when citing a work as evidence of prior findings, an investigator is further validating those findings as a whole, including indirect artifacts of that work–the selection of researchers on the team, the project design, the instruments used to measure and analyze, the collection and treatment of data, etc.  As an OS-specific method, OSDR applies one of the four types of data review as a standard method to either confirm or challenge prior findings, and to further justify new work.  

To be clear, the proposed standard is an open science methodology, not a data science methodology.  

Advancing metaresearch practices

The OSDR encourages interdisciplinary use of research data that may have been collected and tagged for narrow, objective-driven use by scrutinizing it for value beyond its original purpose.  In metaresearch terms, this means that the OSDR method offers new analysis of existing data in different contexts, which opens new pathways into peer review.  For example, OSDR invites investigators to explore the design, collection, markup, and analysis of published data and ask how it might be enriched, refined, and/or repurposed for interdisciplinary applications.  

In the short term, the challenge is to develop a robust set of test cases across a broad selection of disciplines.  It will take time and a significant effort from the global community of open science researchers to undertake the multi-disciplinary pilot project.  In the long term, the challenge is driving the widespread adoption of the framework as a new standard in praxis.  That will require considerable outreach, publication, discussion, and peer review before acceptance and implementation.  Nonetheless, this approach provides a concrete method for the open science community of practice to distinguish its work, which can then lead to downstream improvements in incentive. 

IGDORE USA will host a series of mini-studies over the next three years.  Each 6-12 month mini-study team will receive funding to apply one of the four types of data review to work they are already doing, or to prepare for work they plan to pursue.  

Each mini-study should be produce publishable results that report the challenges, benefits, and lessons learned from using this method as a core open science practice in the lab or field; or in preparation for new work. 

Apply

Complete the proposal form to join the project.