The REF

Posted by Jean Adams

Sometimes when you work at a university you get lured into thinking that the REF is the only thing that matters in the whole world.

REF = research excellence framework: “the new system for assessing the quality of research in UK higher education institutions”.

Ref gives Pablo Zabalete red card 

Once every 5 years or so the whole UK university system evaluates itself. Each department at each university submits information on its work since the last assessment exercise. Panels of experts make judgements on the scientific importance of all the work submitted and every department is ranked for excellence within its field. Then the central university funding agency uses the rankings to decide how much money each university gets. 

When I say they review our ‘work’, I primarily mean they review the papers we’ve published in scientific journals. Some other things are taken into account, but mostly it’s our journal papers that matter. Each academic submits four papers and the expert panel rates them as world leading, internationally excellent, internationally recognised, nationally recognised, or zero. There is no difference here between doing work that is not ‘nationally recognised’ and not having bothered at all.

Cue academics getting their knickers in a big twist about what ‘world leading’ means and how we can demonstrate that our publications are ‘world leading’. Not so long ago, we got totally obsessed by the impact factor. This is a rating given to every scientific journal that is simply the average number of times a paper published in the journal is referenced in another paper in a scientific journal. 

The assumption was that if your research was published in the sort of journal that people tend to reference a lot then it was likely to be pretty good research. Yes really. That really was how we decided if research was good or not. Not anything like will it save a life, or make the world a better place, or improve out understand of human nature. Just: is it published in the sort of journal that other people mention in their work? There are even rumours of journals explicitly manipulating the system by only publishing papers that referenced other papers in the same journal – presumably specifically to help increase the impact factor of the journal.

But for REF2014 we have a totally new and improved method of deciding how good your work is: the citation count. It ignores anything to do with the journal itself and is a simple count of the number of times someone else references your paper in another journal paper. Yes really (again). Even after thinking about it quite hard, and recognising that we might have got it wrong last time, this is the best a bunch of clever people can do for determining how well we’re all doing.

In my world, the only thing that matters is if someone else is writing about me. You are welcome to write: “Adams et al., (2009) were clearly talking nonsense”. But please do remember to write it.