Posted by Peter Tennant
You don't make a pig any fatter by weighting it. For years this has been my favourite animal-related moral, narrowly beating The Fox and the Grapes*. The pig observation is a classic remark on the value, or lack thereof, of examination as an educational device. You don't make a child any cleverer by testing it. Yet for every summer of my childhood (and much of my early adulthood), I was ritually weighed. Since I had a good memory, I rather enjoyed it. But much I believed that recounting the four chambers of a cow's stomach proved I was smart, it didn't; it just proved I could remember the four chambers of a cow's stomach.
.bmp)
This October, UK higher education takes part in its own pig weighting festival, the Research Excellence Framework (or REF). The premise is reasonable. The UK government has £1.76 billion that it wants to spend on research. Rather than dole it out equally, it's decided it's fairer (and more cost effective) to try and give the money to the places that do the most and the best research.
So far so fair. But how do you go about such a task? They could have done a complete audit. Each university presents the sum of all its research (in the form of books, publications etc) and the money is then given to those that have done the most. Except quality obviously matters more than quantity, so only those 'outputs' over a certain minimum standard will be deemed worthy of reward, with standard judged in the usual academic way, i.e. by groups of dusty professors. Presumably they were offered biscuits; professors will do anything for biscuits. Of course, they can't be expected to read everything (have you ever tried to read an academic paper? YAWN!), so instead universities only have to provide a selection of outputs for each researcher, and they don't have to include every member of staff either.
Sounds reasonable enough? Except you now have the formula for a right old mess. Especially once you add that as well as deciding how the money gets spent, the REF will also be used to judge reputation.
Poppleton University: Home to the fattest pigs outside the banking industry
Most people agree that academic league tables are a pretty useless way of comparing schools. Yet schools still do everything they can to maximise their standing. My school focused relentlessly on the C boundary (the grade considered a good pass in the English aged 16 examinations). If you were safely above it, you were on your own. If you were too far below it? Well, around 50 kids were expelled at aged 15 for 'behavioural problems'. The fact they had no hope of getting 5 subjects at C or above was presumably a convenient coincidence.
Similar shenanigans were apparent during the last audit of UK academic research. Some departments amusingly entered only a handful of their best performing staff to boost their showing. This time, star researchers are being parachuted in like premier league football stars and some places are even employing specialist REF gurus to help boost their submissions.
But isn't this all just harmless pig trading? Well, there's another more insidious problem. You can't measure something without changing it. In fact, you can't even observe something without changing it. In a remarkable experiment that not only forms the foundation of quantum physics, but also (and more importantly) forms my favourite non-animal related moral, it can be shown that the universe itself will mischievously change its behaviour when it's being observed. Fire a beam of electrons at two narrow slits and they will make a pattern as if each electron has miraculously gone through both slits at the same time. Set up monitors on each slit to watch this happening and... the pattern changes... to the boring 'one electron per slit' one that you would classically expect. Somehow the electrons know when they're being watched and behave differently. Like me if I'm practising a silly voice and then realise someone can hear me.
In trying to measure 'quality', the REF changes the research environment in a number of ways, the biggest being a dangerous shift towards short-termism. To be REFable (the quite unsavoury term used to mean 'worthy of submission') a member of staff usually needs (among other things) four high quality publications within the audit timeframe. Firstly, this means that a lot of staff are going to be branded as 'unworthy' despite producing work that is 'recognised internationally'**. Which isn't very nice. Or very motivating. If a group of dusty professors told me that my research wasn't any good, I'd instantly conclude they're all past it. So yes, I'd be just like the fox in the Fox and the Grapes. Only with a slightly less bushy tail. Secondly, focussing on 'four high quality publications' doesn't leave much time for any real scientific exploration, i.e. the kind that takes a lot of time and repeated effort to perfect. Like Darwin's On the Origin of Species. 20 years to write a single book? There's no way Darwin would have been REFable.
This shortermism is further exaggerated by the REF's attempt to measure 'impact'. The argument goes like this: since the public pay for academic research they deserve to see tangible benefits. Thanks to 'evidence based medicine' this isn't too hard for someone working in public health, just come up with some good evidence and eventually it'll filter through the system***. But for other subjects… Computing? Electricity? E=MC^2? List the world's greatest intellectual advances and most of them have one thing in common: it took years for any of them to have a measurable 'impact' on society. Then they changed the world.
So next time you set about trying to measure something, think carefully about the possible side effects. In other words, beware Goodhart's Law: 'When a measure becomes a target, it ceases to be a good measure'. Targets are blamed for many evils within the NHS. The REF seems pretty tame in comparison. It may cost millions, stress out thousands, and undermine the very heart of scientific endeavour, but there was no alternative, especially not one that was considerably cheaper and easier... Actually, scratch that last point... I don't know about you, but I fancy a bacon sandwich…
*The Fox and the Grapes is my favourite of Aesop's fables, and a classic illustration of cognitive dissonance. A hungry fox spies some grapes hanging from a tree. Several times he tries to reach them, but he can never quite jump high enough. Eventually, he gives up and concludes the grapes were probably sour anyway, thereby reducing the dissonance between his desire to eat them and his inability to obtain them.
**The REF will grade research into four categories. Unless the work is considered at least 'internationally excellent' (i.e. 3 star) it won't be considered worthy of funding.
***Well, that's the idea. Unfortunately, politicians don't always like 'evidence'. Especially when the evidence works against their financial interests.
Sounds reasonable enough? Except you now have the formula for a right old mess. Especially once you add that as well as deciding how the money gets spent, the REF will also be used to judge reputation.
Poppleton University: Home to the fattest pigs outside the banking industry
Most people agree that academic league tables are a pretty useless way of comparing schools. Yet schools still do everything they can to maximise their standing. My school focused relentlessly on the C boundary (the grade considered a good pass in the English aged 16 examinations). If you were safely above it, you were on your own. If you were too far below it? Well, around 50 kids were expelled at aged 15 for 'behavioural problems'. The fact they had no hope of getting 5 subjects at C or above was presumably a convenient coincidence.
Similar shenanigans were apparent during the last audit of UK academic research. Some departments amusingly entered only a handful of their best performing staff to boost their showing. This time, star researchers are being parachuted in like premier league football stars and some places are even employing specialist REF gurus to help boost their submissions.
But isn't this all just harmless pig trading? Well, there's another more insidious problem. You can't measure something without changing it. In fact, you can't even observe something without changing it. In a remarkable experiment that not only forms the foundation of quantum physics, but also (and more importantly) forms my favourite non-animal related moral, it can be shown that the universe itself will mischievously change its behaviour when it's being observed. Fire a beam of electrons at two narrow slits and they will make a pattern as if each electron has miraculously gone through both slits at the same time. Set up monitors on each slit to watch this happening and... the pattern changes... to the boring 'one electron per slit' one that you would classically expect. Somehow the electrons know when they're being watched and behave differently. Like me if I'm practising a silly voice and then realise someone can hear me.
In trying to measure 'quality', the REF changes the research environment in a number of ways, the biggest being a dangerous shift towards short-termism. To be REFable (the quite unsavoury term used to mean 'worthy of submission') a member of staff usually needs (among other things) four high quality publications within the audit timeframe. Firstly, this means that a lot of staff are going to be branded as 'unworthy' despite producing work that is 'recognised internationally'**. Which isn't very nice. Or very motivating. If a group of dusty professors told me that my research wasn't any good, I'd instantly conclude they're all past it. So yes, I'd be just like the fox in the Fox and the Grapes. Only with a slightly less bushy tail. Secondly, focussing on 'four high quality publications' doesn't leave much time for any real scientific exploration, i.e. the kind that takes a lot of time and repeated effort to perfect. Like Darwin's On the Origin of Species. 20 years to write a single book? There's no way Darwin would have been REFable.
This shortermism is further exaggerated by the REF's attempt to measure 'impact'. The argument goes like this: since the public pay for academic research they deserve to see tangible benefits. Thanks to 'evidence based medicine' this isn't too hard for someone working in public health, just come up with some good evidence and eventually it'll filter through the system***. But for other subjects… Computing? Electricity? E=MC^2? List the world's greatest intellectual advances and most of them have one thing in common: it took years for any of them to have a measurable 'impact' on society. Then they changed the world.
So next time you set about trying to measure something, think carefully about the possible side effects. In other words, beware Goodhart's Law: 'When a measure becomes a target, it ceases to be a good measure'. Targets are blamed for many evils within the NHS. The REF seems pretty tame in comparison. It may cost millions, stress out thousands, and undermine the very heart of scientific endeavour, but there was no alternative, especially not one that was considerably cheaper and easier... Actually, scratch that last point... I don't know about you, but I fancy a bacon sandwich…
*The Fox and the Grapes is my favourite of Aesop's fables, and a classic illustration of cognitive dissonance. A hungry fox spies some grapes hanging from a tree. Several times he tries to reach them, but he can never quite jump high enough. Eventually, he gives up and concludes the grapes were probably sour anyway, thereby reducing the dissonance between his desire to eat them and his inability to obtain them.
**The REF will grade research into four categories. Unless the work is considered at least 'internationally excellent' (i.e. 3 star) it won't be considered worthy of funding.
***Well, that's the idea. Unfortunately, politicians don't always like 'evidence'. Especially when the evidence works against their financial interests.
Another great blog post Peter! What a well written perspective on the REF, often when another standardized "bar" is raised we are all too keen to just try to jump it without asking the bigger questions (though I know rules are rules…) Thanks for sharing this and hope it can inspire some dialogue amongst the Ref-able-which is not me-but I am *sure* the experience is more sour than sweet ;)
ReplyDeleteLot's to agree with here, but not the bogus argument about Darwin. "There's no way Darwin would have been REFable." Heard people make it before. Darwin was very productive, he published loads while waiting to publish Origin. By the standards of his day he'd been very REFable.
ReplyDeleteOooops, that's my credentials as a historian ruined. Interesting that other people have made the same bogus argument, I've never heard it said. Poor old Darwin.
Delete