To open science, destigmatize critique

TabulaSynthase, the blog of Whitehead Institute, brings together ideas and perspectives from the Whitehead community and beyond. 

Here we feature a guest blog by Jessica Polka, a visiting scholar at Whitehead Institute, Director of ASAPbio, and President of Future of Research.

Utopian visions about how the web would change science are as old as the web itself. In 1989, the same year that Tim Berners-Lee wrote his famed CERN proposal, William Allen Wulf introduced the concept of a “collaboratory,” a “center without walls, in which the nation’s researchers can perform their research without regard to physical location, interacting with colleagues, accessing instrumentation, sharing data and computational resources, [and] accessing information in digital libraries.”

While Wulf’s vision is now entirely economically and technologically practical, it’s still at odds with cultural practices and incentives. The ease with which we now record, share, and access digital information awkwardly bumps up against more traditional professional norms of the scientific community - imagine social media embargoes or poster photography bans at society meetings. Despite the fact that biologists are actually quite liberal with sharing findings prior to their formal publication compared to other disciplines, we protect those disclosures with varying levels of privacy. Therefore, the only thing that enters into the publicly accessible literature is the final, published paper. In order to tell a good story, those final papers leave out a lot of valuable information contained in slides, preprint manuscripts, posters, single figures, code, data analysis pipelines, and even lab notebooks.

If we were comfortable with sharing our interim research products early, our science could more rapidly inform the research of others around the world. We could get better (and broader) feedback to improve our work. We could identify collaborators and reduce needless duplication of effort, or we could channel that duplication into robust and well-planned independent replications. Science as a whole would become more efficient.

Barriers to sharing

Given these benefits, why don’t we make interim research products available well in advance of formal publication? Certainly, the barriers are not technical, but rather social, and I’d argue they fall into three categories.

The first is that sharing early research results may run afoul of journal policies, and may prevent later “formal publication.” Fortunately, however, with the rise of preprints in the life sciences, attitudes about what constitutes “prior publication” are rapidly changing, and an increasing number of journals no longer consider sharing results online a barrier to consideration, especially within established preprint servers like bioRxiv and arXiv.

The second perceived barrier is related to the first: that the community will not adequately recognize contributions when they are shared in the grey literature. Having been divulged without sufficient visibility, the data could be vulnerable to being scooped by a competitor who publishes in a more conventional and visible venue, perhaps after gaining some advantage from seeing (and not properly acknowledging) the work. As the visibility of interim research products increases, though, it stands to reason that scooping will become less of a plausible concern. After all, fixing work into a public, permanent, and time-stamped record is the best possible defense from scooping.

The third barrier to sharing interim research products is more fundamental than either of the two above: discomfort with being wrong or showing errors to peers.

Moving beyond the version of record

Of course, we share preliminary data with one another specifically to identify and correct errors and omissions. We spitball with our labmates and show messy data at lab meetings. We brainstorm with our collaborators and put preliminary data into talks or posters. But these behaviors fit into fairly proscribed concentric circles of permanence and privacy, and the mutable documents they produce rarely make it into public view.

By contrast, we think of publicly-shared papers as static documents, and in the days when they were printed, bound, and mailed, they essentially had to be. Perhaps as a result, their correction or modification is stigmatized, and articles can be retracted for problems affecting one part of a larger work. In turn, this stigma contributes to discomfort with sites like PubPeer, where criticisms of journal articles are publicly aired. After all, because papers are fixed and “final” objects, it’s almost impossible for these critiques to have a positive outcome for the original author.

But it doesn’t have to be this way. Digital technologies provide ways to organize many different versions of documents, from tracking changes in Microsoft Word to merging pull requests in GitHub. We are already familiar with the kinds of tools that would enable us to modify works previously thought of as set in stone. But again, the barrier is not technological but social. This would require us to admit publicly that we have been wrong - or at least not entirely right.

Destigmatizing uncertainty, errors, and iterative improvement

Admitting mistakes (or even uncertainty) is difficult for people in general, scientists included. That said, there are certain communities (for example, open-source software development or Wikipedia) where public corrections and collaborative contributions are a natural part of the system. If we want to shift our culture toward this state, there are two concrete steps we can take.

First, we can advocate for the publication of (anonymous) peer review reports, as many journals (including EMBO, eLife, Nature Communications, and BMC titles) are already doing. Reviewers and authors of a paper are privy to an animated discussion; hiding this from general view creates an artificial illusion of silence. If we were to make visible this vast, existing corpus of discussion, we might change expectations about where this discussion belongs and who can participate.

More immediately, we can post preprints and invite feedback on them. The comment section of bioRxiv is moderated and is home to many helpful exchanges. For one exceptional example, see the conversation between Helder Maiato and David Pellman underneath the latter’s preprint. This type of dialog may happen in private, but bringing it into the public benefits the whole community. As Vincent Boudreau, a grad student in the field, tweeted, “seeing agreements and disagreements is way more valuable than reading any review.” And it has the bonus effect of normalizing civil and constructive public dialog.

We may not get all the way to working in “collaboratories,” but being more open about critique and feedback could make science more connected and efficient.