Do the RAE judges read all the research submitted? They couldn't if they tried

"Have you read them all?" Selina Scott's question to the Booker prize judge, Fay Weldon, in 1983, has entered literary lore and, doubtless, will live forever in anthologies of literary quotation.

The last RAE (literally the last, in its present form, one understands) departed from previous practice by demanding physical copies, not mere citation, of all submitted items. Lorries duly set out from universities in the four corners of the country, giving a new meaning to the term "heavy goods vehicles".

The maths of the operation looked daunting, verging on superhuman. Take my subject area, English. Some 110 departments, with, on average, 25 fulltime or equivalent staff, each required to turn out four samples of published work: say an average of 200 pages per colleague. Roughly half a million pages then, some 30,000 for each of the dozen-and-a-half members of the panel.

But, of course, some fields (20th-century literature, for example) generate a great deal more wordage than others (Old Norse, for example). And we're not talking Hello magazine.

A page of high-density scholarship, will take – how long? –five minutes on average? More depressing maths. One's very calculator aches at the thought.

The panellists are, of course, doing this part time, and still doing fulltime jobs in their own institutions. They are less than handsomely paid: less, as it happens, than Booker prize judges who have to contend with only 120 volumes (many, I can confirm, much easier to read than Hello magazine).

Scott's impertinent query is inescapable. Did they read them all? Put another way, did the assessors think, for a moment, that they were obliged to read all the scholarly deadweight trucked to them? Could they, even if they did think so, have done it?

I suspect the answer to all three questions is no. And I have some evidence, albeit superficial, to support that commonsensical suspicion.

One department I know (don't assume it's necessarily the one I'm closest to) had its administrative staff neatly package each item separately in a cellophane envelope, sealed firmly with bands of sellotape. When the material was returned, some two thirds of the samples were open (no sellotape). A third looked exactly as they had when they were sent out. I saw them with my own eyes.What to make of it? Not necessarily any dereliction of assessing duty. The following explanations suggest themselves:

The point of the exercise was not to read every last ounce of scholarship submitted, but to ensure that it actually existed, in published form.

Many of the items were already familiar to specialist members of the panel, all of whom keep up with their subject. They had already scrutinised the items; reviewed them, conceivably. There was no need to unpack them.

Some items were regarded, prima facie, as not worth wasting scarce time on.

A senior member of another department confided an unsettling experience. Two of his four items were returned to him – both, as it happened, hardback books (the hard currency of the RAE) – with yellow postits still attached. One read "output 1", the other "output 4".

This would seem to indicate a thumbs down for one (negligible) and a thumbs up (internationally meritorious) for the other. Both postits were so tenaciously attached that they could not be removed without defacing the covers. One book, alas, can never more grace the scholarly shelf, with its garish mark of Cain. The other may twinkle happily.

They were clearly looked at; however cursorily. But what strikes one is the Procrustean crudity of the grades. Scholarship in English is typically praised as "lively" and "stimulating", or dismissed as "dull". These are sense impressions as much as critical judgement.

The categories imposed by the RAE exercise are, necessarily one suspects, triage, not critical response. And is it likely that the same scholarly mind (a very good one, in my judgement) would veer so precipitately up and down the scale? The members of the panels whom I have known have been conscientious and strenuously impartial. But the machine they serve is crocked. RAE, with its stress on per-scholar norms, has generated over-production. Much more material than could be useful digested. A boa constrictor, one is told, can swallow a goat: it can't swallow an elephant.

Everyone will have their own ideas as to how the RAE machine should be re-engineered. Mine is that each department (in my subject area, and others like it) should be required to submit 10, or fewer if they wish, outstanding pieces of scholarship produced by staff in the assessment period. So what if not everyone is represented on the submission sample? They also serve.

Apart from ensuring that the total submission presented to the panel would be manageable, the system would internalise competition to "make the list". The contest would take place within the department, where it belongs, rather than between departments, where it becomes destructively gladiatorial. And it would focus on quality, not quantity.

Such an arrangement would parallel what is done with the Man Booker prize: publishing houses are restricted to two submitted novels, plus any of their authors previously shortlisted. Were these restrictions not in place, the big publishers – Cape, Faber, Random House, Viking Penguin – would swamp the field and win every time. As Oxbridge does the RAE.

Instead, if gossip is to be believed, the RAE is moving away from subjective critical judgement to "objective criteria". Which in English could, I suppose, mean how many times a scholar crops up in the Arts and Humanities Citation Index, or how much cash a department garners in grant money and fellowships, or how many academic prizes are won.

It may work in the sciences. It won't work in arts departments.