Peer pressure; Academic publishing.

"An influential academic safeguard is distorted by status bias

WHEN, IN 1905, the then-unknown patent clerk Albert Einstein sent his revolutionary ideas on special relativity, the photoelectric effect, Brownian motion and a few other topics to the German journal Annalen der Physik, its editors were happy to publish them. Submissions were rare and therefore rarely rejected--unless the text was clearly bonkers.

Things are different now. Most top academic journals use a system of peer review, which asks independent experts in the same field to assess papers before they are accepted. Reviewers are meant to check the methods, analysis and conclusions and, crucially, whether the work meets the required standards for publication.

No scientist would claim that peer review is perfect. There are plenty of famous cases of ground-breaking papers being rejected after flawed advice from reviewers, while seldom a week goes by without one field or another rounding on a shoddy piece of work on social media and asking how on Earth it passed peer review. Many researchers describe the review process by borrowing Winston Churchill's quip about democracy: it's the worst system except for all the others.

A new study of the peer review process reveals a novel and depressing, if not totally surprising, fault. It indicates that a modern-day Albert Einstein, or any researcher with a good idea but without an already-stellar reputation, might struggle to get their foot in the door. Status bias means the name of the individual on the paper can matter as much as the findings when it comes to what gets published, suggests the study, which was released last week as a working paper on the SSRN repository.

Researchers have suspected for a long time that work from established senior figures often gets an easier ride in peer review and is more likely to be accepted and published. It is an example of the so-called Matthew effect of accumulated advantage, that eminent people get disproportionate credit for work--named after the biblical parable of the talents in the Gospel of Matthew, which states that "to everyone who has will more be given".

In the new study, researchers at the University of Innsbruck, in Austria, collaborated with Vernon Smith, an experimental economist at Chapman University, in California, and a winner of the Nobel Memorial Prize in Economic Sciences. Dr Smith had just completed a project with Sabiou Inoua, a colleague at Chapman University who at the time was a Ph D student. The duo had written a paper on financial and market data that was ready to submit to an academic journal.

The team from Innsbruck had a devious plan--use the name of either Dr Smith or Mr Inoua as the paper's author and send it to peer reviewers to see how they judged the quality of the work. Editors at the Journal of Behavioural and Experimental Finance, the journal to which the pair submitted their manuscript, were admirably ready to play along.

They first asked more than 3,300 potential reviewers if they would be willing to take the time to assess the manuscript, based on a short abstract emailed to them that listed one of the two authors' names, or omitted the names entirely. In this scenario, including Dr Smith's name saw the acceptance rate jump--almost 40% agreed to review when he was the author compared with closer to 30% when the author was Mr Inoua or not listed.

Those who agreed were sent a full manuscript to review, with the same pattern of names, and more than 500 reviewers submitted reports. When they thought it was Mr Inoua's work alone, 65% of reviewers voted to reject it. That is almost three times as many as the 23% of reviewers who rejected the same paper when it carried only Dr Smith's name.

But it was also a significantly higher rejection rate than the 48% who spurned the paper when it was completely anonymous. Not only did Dr Smith's eminence boost his numbers, but the newbie status of Mr Inoua counted against him.

Does the pernicious impact of status bias seep beyond the pages of this particular journal and this particular field? Juergen Huber, one of the Innsbruck team, is certain that it does. Every discipline from chemistry and physics to medicine and genetics has its own superstars, he says, while some results indicate that top institutions like Harvard University also get a status boost in peer review.

One option to deal with the bias is to remove all names from all manuscripts under review. But Dr Huber points out this is increasingly difficult with the rise of preprints and working papers published online before they are formally submitted to a journal. Any reviewer of an anonymous manuscript could simply search for its tell-tale online trail.

The story has an interesting coda. Mr Inoua and Dr Smith's bold willingness to test the limits of peer review has not come without cost. The Journal of Behavioural and Experimental Finance is yet to publish their paper. It is waiting for the duo to respond to the reviewers' comments--all 500 of them." [1]

 

·  ·  ·  1. "Peer pressure; Academic publishing." The Economist, 17 Sept. 2022, p. 69(US).