Abstract
In the last decade Open Science principles, such as Open Access, study preregistration, use of preprints, making available data and code, and open peer review, have been successfully advocated for and are being slowly adopted in many different research communities. In response to the COVID-19 pandemic many publishers and researchers have sped up their adoption of some of these Open Science practices, sometimes embracing them fully and sometimes partially or in a sub-optimal manner. In this article, we express concerns about the violation of some of the Open Science principles and its potential impact on the quality of research output. We provide evidence of the misuses of these principles at different stages of the scientific process. We call for a wider adoption of Open Science practices in the hope that this work will encourage a broader endorsement of Open Science principles and serve as a reminder that science should always be a rigorous process, reliable and transparent, especially in the context of a pandemic where research findings are being translated into practice even more rapidly. We provide all data and scripts at https://osf.io/renxy/.
Discussion
In addition to previous concerns and investigations of the disruption that the pandemic has caused for research [2], we have found strong evidence of how COVID-19 has impacted science and scientists on several levels. Firstly, we have highlighted the striking scientific waste due to issues in study designs or data analysis. Secondly, we have found evidence of the misuse of preprints in news reports which seem to refer to non-peer-reviewed manuscripts as reliable sources. Thirdly, we have found that the fast-tracking of peer-reviews on COVID-19 manuscripts, which was needed to give vital treatment directives to health authorities as quickly as possible, led to potentially suspicious peer-reviewing times often combined with editorial conflicts of interest and a lack of transparency of the reviewing process. Finally, we highlighted that the lack of raw-data sharing or third-party reviewing has led to the retraction of four major papers and had a direct impact on the study design and conduct of international trials.
The Open Science movement promotes more transparency and fairness in the access to scientific communication, the production of scientific knowledge and its communication and evaluation. Looking at the number of publishers removing their paywalls on COVID-19 related research, one might argue that the COVID-19 pandemic has been a catalyst in the adoption of Open Science principles. However, the aforementioned issues paint a more complicated story. The urgency of the situation has led to a partial Open Access policy but with a very opaque peer-review process coupled with a misuse of preprints and raw-data-sharing policies not being enforced. Full adoption of Open Science principles could, however, have saved precious research resources: open peer review would have helped in the detection of the editorial conflicts of interest and made it apparent whether manuscripts were thoroughly reviewed; adoption of registered reports would have strengthened study designs and data analysis plans; proper and monitored use of preprints would have helped the communication of early results between researchers; strengthening of the policies of raw-data sharing or reviewing could have prevented the Surgisphere scandal; and full Open Access might have accelerated the search for solutions to the pandemic both in medical and socio-economic contexts. In addition to this, statistics reviews could have helped to make studies and their results more robust and limit the impact of exaggeration or misinterpretation of results.
It remains, however, that these principles are not enough. The pandemic has highlighted other issues that Open Science cannot solve. For instance, the misuse of preprints by journalists probably stems from the fact that many journalists may not be trained to understand and navigate the complex academic publication system, and some journalist may be seeking sensationalist news headlines. The pandemic has also highlighted the already existing science-literacy issue. Finally, we cannot exclude that some of the misuses and abuses that we have highlighted are a direct result of the current metric-centered evaluation of research and researchers which has already been shown to lead to questionable research practices in the past and has been the subject of criticism from scientists for decades [36, 91, 92].
We, as scientific researchers attached to transparency and fairness in the production, communication, use and evaluation of scientific knowledge, hope that this manuscript successfully argues and promotes a faster adoption of all Open Science principles. We therefore call upon readers of this manuscript to co-sign it, should they agree with our recommendations, through the following link http://tiny.cc/cosigningpandemicopen.
I recently attended the UNESCO Open Talks Webinar “Open Science for Building Resilience in the Face of COVID-19”, which touched on many of the ideas from the pre-print above. The webinar recording is available on YouTube, and I've also written up a short summary which can be accessed here. The WHO representative made it clear that they were in favour of Open Science and that it has assisted them in their work.
More generally, I think that Open Science is relevant to EAs from two perspectives. Firstly, it has the potential to reduce problems with and increase benefits from scientific research, which could have positive benefits for society. More directly, EA research often summarizes academic research and EAs should benefit if that is both (legally) freely accessible and also done more transparently. Although a lot of EA research is effectively published open-access (e.g. forum/blog posts) it could be also interesting to consider what other open science ideas can be incorporated into EA research.