No fewer than three academic journals dismissed the economist George Akerlof's paper The Market for Lemons as "trivial" and "too generic" when it was submitted in the late 1960s. Almost four decades later it was regarded as a seminal text and its author thought worthy of the Nobel prize for economics.
Peer review, when an academic submits a scholarly work to the scrutiny of other experts in the field for publication in a journal or for a grant, for example, has always been an imperfect science. But lately it has had more, and fiercer, critics.
They say peer review is biased against innovation and originality. They argue that it costs too much - more than £196m a year was the estimate by Research Councils UK last year. And they say it takes up too much time now that more academics than ever are submitting papers and fewer claim they can afford the time to "peer review" them.
Today a report published by the British Academy - an academic club of 800 scholars elected for distinction in the humanities and social sciences - speaks up for peer review. The professors quote Joan Sieber, a psychologist at California State University, who has said: "One suspects that peer review is a bit like democracy - a bad system, but the best one possible."
Albert Weale, professor of government at the University of Essex and chair of the committee responsible for the report, describes peer review as "the essential backbone to knowledge and the crucial mechanism in maintaining its quality".
Robert Bennett, professor of geography at Cambridge University, says it is "an essential, if imperfect, practice for the humanities and social sciences".
The report's writers snap back at those who attack peer review. They back up their ripostes with the comments of journal editors, research councils, charities and funders, academics and postdoctoral students. To those who say peer review is biased against innovation and that journal editors "play safe" and are "friendly to their own work", the academy's response is that universities and research councils are awarding more grants for risky, avant-garde research projects than ever.
The report admits that "there may be scope for the government to consider ways in which it can encourage endowments ... within universities to support small grants for innovative, high-risk research".
But it warns: "It is important not to commit the fallacy of assuming that, because high quality will be innovative, the innovative is necessarily high quality ... other criteria include: accuracy, validity, replicability, reliability, substantively significant, authoritative and so on."
Banality gets acceptance
Marian Hobson, professor of French at Queen Mary, University of London, says: "If a journal editor gets everything right all the time, they are probably aiming for the middle, banally all-right work, which will be out of date in the blink of an eyelid. Really excellent work may sometimes take a while to be accepted."
To those who lambast peer reviewing for being too time consuming and costly, the professors have the following suggestion: give far more recognition to the unpaid, altruistic labour of those who do it and the system will be under less strain.
Hobson says: "If done properly, [peer review] entails bibliographical searches, checking of statements, repeated visits to the university library, not just to Google. Yet this kind of activity counts for nix, nothing, zilch in the research assessment exercise [in which every active researcher in every university in the UK is assessed by panels of other academics to receive grants for their research]."
The academy stops short of demanding peer reviewers be paid. It realises this would be impossible for all but the most wealthy journal publishers. Instead, the report recommends that the importance of peer reviewing should be better reflected in the research assessment exercise. "Those responsible for the management of universities and research institutes need to ensure that they ... encourage and reward peer review activity," it says.
This might stop some high calibre academics, already overburdened with work, from being put off peer reviewing, the professors say. It might also attract junior lecturers and even postdoctoral students. More reviewers would mean the system was under less strain. The strain is partly triggered by an increase of up to 62% in the number of academic papers submitted of in some fields in the past five years.
Here lies another problem, says the report. "As we conducted our review, we were struck ... by the extent to which there is little attention to training in peer review," it says. "Training is important, not just in itself, but because of the privileged position that peer reviewers enjoy.
"By virtue of reading a paper, reviewers can acquire access to original data sets, new empirical results or innovative conceptual work. In the business world, these would count as commercial secrets. In the academic world, the ethos is that reviewers are part of the gatekeeping system, the ultimate rationale of which is the fast and efficient dissemination of research findings.
"The integrity of the peer review system is therefore of great importance. One of the ways in which that integrity is maintained is through its dependence upon professional and unselfish motivations, and this in turn suggests the importance of training in the professional and ethical conventions of the practice."
The academy's report ends with a warning to the government: plans to overhaul the way research is assessed after next year will change peer review for the worse, especially in the humanities.
Metrics-based approach
After 2008, the quality of research - and hence the amount of funding that universities receive from the government - will be judged largely on the basis of statistics such as grant income and contracts. It is accepted in the sector that this "metrics-based" approach will work better for science and engineering than for arts and humanities research, which does not receive much income and where books take longer to have an impact.
Hobson says: "Metrics is helpful in giving a kind of overview measured in terms of items. A bit like a waistline measurement. It doesn't give much of an idea of whether they are slim or fat, unless they are at the extreme ends of the spectrum.
"Wittgenstein at his death had one book and one article published. Another book was on the way, but unfinished. "Heaven knows what would have happened to him in today's academia."