F1000 Research seems to delight in being surreal — publication before review, authority generated extrinsically instead of intrinsically, and “reviews” consisting of zero words of feedback.
It’s a topsy-turvy publishing model.
Yet, F1000 Research has been accepted into the PubMed index via PubMed Central. It has also since redefined itself as a mega-journal, after initially saying expressly it was not a journal. Its search for identity may be linked to its inability to assert its own authority.
How did F1000 Research get indexed in PubMed? Let’s look at what I recently obtained via the ongoing and expanded Freedom of Information Act (FOIA) request filed with the National Institutes of Health (NIH). The materials reveal how PMC changed its role relative to journals in order to accommodate the F1000 Research model. They also lay bare the contradictory mechanics operating within the F1000 Research model itself.
F1000 Research applied for inclusion in PMC in October 2012. Because F1000 Research is an open access (OA) journal, inclusion in PMC also grants it indexing in PubMed (but not MEDLINE — owing to the rampant confusion between PubMed and MEDLINE, however, this distinction is for specialists only). It was granted inclusion in December 2012, with some stipulations, as this email from the NLM to Rebecca Lawrence of F1000 on December 10, 2012, shows:
In the email responding to news of PMC’s approval of F1000 Research for inclusion and indexing in PubMed, Lawrence’s response probes gently to see if PMC would consider keeping the previous F1000 Research standard:
In response, PMC reiterates its standards:
Notice that now PMC is saying that their approval is for deposit in PMC, not indexing in PubMed. It’s a small point, but the two are so conflated that the elision is natural.
This “Approved with Reservations” status bears further scrutiny. What exactly is it? According to F1000 Research’s PMC application, submitted on October 3, 2012, by Rebecca Lawrence:
We’ve seen criteria for acceptance expand a good deal with things like “methodologically sound” or “scientifically sound” for some of the mega-journals. These standards eschew novelty or interest (I’m going to be writing my paper about coin flipping any day now, because it will likely be methodologically sound). But this new set of definitions gets more basic yet — no longer do the methods or the science need to be “sound” according to F1000 Research. In this new vision, the methods only need to amount to “science.” Any science. Boring. Methodologically flawed. Repetitive. Irrelevant. Just as long as it’s science.
It’s part of the upside-down world of F1000 Research that “Approved with Reservations” counts as an approval status. This is the opposite of what a similar status would be in a pre-publication review system. I can’t say that emphatically enough — it is the op-oh-sit! But F1000 Research describes it as a “positive referee status” in its application:
Another email from October 24, 2012, confirms this backwards point of view.
For journals using pre-publication peer review, the status of “Approved with Reservations” may not seem all that unusual. It’s a signal to an author that the paper is very good, but needs some fine-tuning or more analysis or rewording of key conclusions before it gains final acceptance. However, the paper is not accepted and certainly not published, and the status is not a “positive referee status” but merely a less-negative referee status. But it is still negative. It is a form of conditional rejection. Retaining the leverage of publication all but guarantees that the author will make the minor modifications the editors and reviewers require.
In its backwards model, F1000 Research has already published the paper, so “Approved with Reservations” is a toothless de facto approval. Yes, the reservations are included in the online review, but readers have to sort through them, and they are not often terribly detailed. In any event, the problem runs deeper.
Because the papers are published before they are reviewed, they can’t be unpublished. PMC tried to see what might happen in this case in October 2012, in a long list of questions:
The answer is a bit evasive and technically incorrect:
That URL (and even one corrected to add the www.) leads to a 404 error, so I don’t know how PMC could have checked for further details. But that’s not all that’s wrong with this response. First, an article retraction or withdrawal usually only changes the status of its entry in an index. As the NLM states:
NLM does not remove the citation for a retracted article, but updates the citation to indicate it has been retracted, and links the original citation to the citation for the published retraction notice.
Claiming that an article that’s retracted would no longer be indexed is incorrect.
Lawrence doesn’t answer the question directly, and it’s an important one. In the F1000 Research system, an article is published, then reviewed. The reviews can change the indexing status of an article, but only once. According to the agreement made by F1000 Research and PMC, an article can be indexed with two “Approved with Reservations” and one “Approved.” That article might also then be revised to deal with the reservations found in the two “Approved with Reservations,” despite the journal having no real leverage over authors at this point. Those revisions could lead to two “Not Approved” statuses on Version 2. What then? The paper has been published and indexed. It can’t be unpublished. And apparently, it can’t be unindexed. Does it have to be retracted then? And what if further revisions improve the paper so that Version 3 is exactly what the reviewers ordered? Is it unretracted?
As we’ll see below, there may never be a final paper in F1000 Research, only temporal versions.
This issue came up during the back and forth between F1000 Research and PMC, but the exchange is brief and unsatisfying. First, PMC’s question:
Lawrence’s response seems at first glance to make sense, but falls apart when you think about it:
There are strong assumptions in this scenario — that Version 2 doesn’t deviate in important ways from Version 1, such that the initial reviewers’ “ticks” are no longer valid; that Version 2 isn’t re-reviewed by the first reviewers as well, and doesn’t end up with two “?” and one “tick.” More significantly, the scenario is never considered in which an author receives two Approvals and is indexed then goes back in, modifies the paper significantly, carries the two Approvals forward into a version the reviewers would effectively never have reviewed as it deviated so much from the initial paper, and then sits back and lets biased information permeate the landscape. It would be difficult to detect the changes and impossible to know that the reviewers had not approved of the latest version. And don’t tell me this is far-fetched — we’ve seen a lot of misbehavior over the past few years among authors.
There are a lot of problems with the F1000 Research model.
Another problem was revealed when PMC forced F1000 Research to change their standards mid-stream. This has created problems for authors who initially met the F1000 Research standards for having their articles indexed. As a blog post on the F1000 Research site noted in February:
In 2012, we agreed with Scopus and others that an article would be indexed as long as it had obtained any two of our ‘Approved’ or ‘Approved with Reservations’ statuses. At the start of 2013, NLM agreed to index our content but they requested slightly more stringent criteria: either two ‘Approved’ statuses, or two ‘Approved with Reservations’ statuses plus one ‘Approved’ status.
Going forward, we decided to apply these more stringent criteria across all the indexers of our articles to make it simpler for everyone. What this does mean though is that there are a small number of articles published in 2012 that passed the old criteria but not the new criteria. It would be unreasonable to the authors to un-index them but of course those articles will not be indexed in PubMed as it stands, and most of those authors are working on new revisions as we speak.
Essentially, the editorial acceptance standard of F1000 Research was reset for the sake of indexing — not improved in a general way to match a standard peer review benchmark ala MEDLINE inclusion, but specifically dictated by PMC. So-called “open peer review” seems to have opened F1000 Research up to a detailed revision of its criteria.
Thanks to all of this, at F1000 Research, you can have an article that was published, indexed in all the places the journal was indexed at one point, and is now indexed in a subset of the places the journal is indexed once standards for acceptance changed.
More interestingly, PMC generated the standard of acceptance at F1000 Research. Instead of an indexing service being downstream from the journal when it comes to acceptance, PMC moved upstream. PMC’s specific standards became F1000 Research’s specific standards. That’s how upside-down things are at F1000 Research.
It took me a while to appreciate that F1000 Research has essentially outsourced its editorial authority — a major indexer has caused it to restate its standards of acceptance, making the indexer the new upper measure of validation. The effect shows itself in a new design on the F1000 Research articles landing page, which used to be just a running list, but which now sports a tabbed design with an “Indexed” tab and an “All Articles” tab. They might as well put “Accepted” and “Accepted and Rejected” on those tabs — the meaning would not change, but might become uncomfortably clear.
As I mentioned above, F1000 Research articles are apparently never set in stone, but elements like “published” and “indexed” and reviews are. But while they may not be set in stone by the publisher, they can suffer from neglect and indifference — something no paper in a pre-publication review system would suffer. Papers in traditional peer review systems either get an acceptance or rejection. There is no purgatory for papers to wander for eternity. But there is a purgatory at F1000 Research.
In my January 2013 post on F1000 Research, I noted that an article from November 2012 had not received any reviews. Checking back in April, the article still has not received any reviews. It is listed in the “All Articles” (or, “Accepted and Rejected”) tab. At what point (after six months?) will it be considered rejected? Or is it published but never accepted if it’s never reviewed and indexed (accepted)? In an email from November 2012, Lawrence responds to a question from PMC about the time period available for review in a way that can’t be reassuring to authors in this situation:
I would also take issue with her assertion that it’s a benefit to publish a paper that can be revised endlessly. The potential pitfalls with this were discussed above.
It’s also worth noting — mostly in case we ever pass this way again — that here is a government organization via PMC dictating a specific new standard around editorial acceptance for an applicant journal. At first, I hesitated to write this out, but the more I thought about it, the more odd it seemed. I couldn’t shake the feeling that this might be one of those “you saw it here first” moments. Instead of letting the thought pass, I guess it’s better to jot a note to the future that here we saw an OA publisher accept a new editorial standard from the US government.
F1000 Research seems to like to feel subversive. However, they have ceded their editorial authority to an index run by the US government. That’s not very subversive at all.
More importantly, the painful distortions to publication, peer review, and authority being perpetrated by F1000 Research makes pretzel logic seem straightforward. It’s no wonder that despite approval for indexing in PMC, no F1000 Research articles have appeared yet. Tagging this batch of illogic into a coherent system is a chore that NCBI and NLM technologists are finding quite perplexing. Even if they figure it out, whether published papers are ever reviewed, reviews pertain to the proper version, or indexed articles retain their indexed pedigree all remain major uncertainties.
I have no punchy ending to this post. F1000 Research just makes my head hurt.