What fairy tales and pinball machines can tell us about using research

Posted by Peter van der Graaf, AskFuse Research Manager / Fuse Knowledge Exchange Broker, Teesside University

Once upon a time... the UK Knowledge Mobilisation Forum 2019 took place in Newcastle at Seven Stories, the National Centre for Children’s Books. The Forum brings together practitioners, researchers, students, administrators and public representatives who are engaged in the art and science of sharing knowledge and ensuring that it can be used. This year I was part of the organising group and we deliberately chose Seven Stories as location for the Forum with its focus on stories, which are an important mechanism for exchanging knowledge.

5 points awarded to Dan Wolstenholme of House Gryffindor 

Knowledge mobilisation wizards are also visiting #UKKMb2019 @fuse_online pic.twitter.com/010WtfEwh1

— UK KMb Forum (@UKKMbF) March 20, 2019

The Centre, spread out over seven floors, provided plenty of exercise to get to the different rooms for interactive poster sessions, knowledge fayres, fishbowls and workshops. The Harry Potter themed conference room also allowed me (not pictured right) to dress up as a Ravenclaw student while serving coffee. But the real highlight of the conference for me were the stories being shared by the key note speakers, Ishbel Smith from Heart in Mouth, who reflected on her practical experiences, and Andree le May and John Gabbay from the University of Southampton, who provided an in-depth research perspective. I will relate two of their stories: the first involves a house made of sweets, the second is about a pinball machine.

The power of stories is often underestimated in research but has gained some traction in academia, particularly for evaluating new interventions and because they make great impact case studies. At the same time, it has proven notoriously difficult to capture any impact in a story: we don’t really know what happens to research findings once they leave academia and trying to trace impact is like Hansel and Gretel clinging onto tiny bread crumbs in a large wood.

This brings me to the first story. Ishbel Smith used the story of Hansel and Gretel to highlight that the siblings missed the blindingly obvious (a house made of sweets is too good to be true and likely to be a trap), because they were too focused on the breadcrumbs and did not see the wider picture or context in which they were walking (a deep dark wood inhabited by a hungry witch). As knowledge mobilisers it is vital to understand where we are in a given situation to be able to unlock the relevant knowledge in that context.

Ishbel reiterated that contextual knowledge is just as important as content. Understanding the context in which research evidence is used and, perhaps more importantly, what happens to that evidence in a practice or policymaking context, is vital for mobilising it. To clarify this, Andree Le May and John Gabbay, told a story about how research evidence is transformed by using the analogy of a pinball machine.

In this second story, research evidence gets batted around across various groups of people within organisations as they interact with it. In each interaction, the evidence is slightly changed: people put their own spin on it and adjust it slightly for their own needs. The evidence literally receives a battering but in this process the evidence is made fit for a particular context and socially reconstructed. Like the pinball getting batted around inside the machine, with every contact the evidence changes shape. Perhaps not noticeably at first but you end up with something quite different and unrecognisable from the research evidence that you put into the dissemination process at the start. This means that, if you don’t understand as a researcher that your research findings will be transformed when it is being used by practitioners and policy makers, then you will never be able to find and follow it for your impact case studies.

Andree and John persuasively pointed out that the craft of knowledge mobilisation is not only using the right skills to get evidence into practice, but also to be able to be part of the story of how evidence is used. This requires not only technical skills (which are mostly studied by implementation science) but also the use of soft skills, such as the striking the right tone and style, being able to get your message across and contextualise knowledge; for example, do we know what the right problem and the right solution is for the context in which we are trying to mobilise knowledge?

Perhaps the most important skills we can develop for mobilising knowledge is how to enable learning: what did work and didn’t work in this context? How can we help others to apply research evidence into their own context? Stories provide a powerful tool for this: not to highlight what we have achieved as research institutions in the next REF submission, but to create a space to reflect on our experiences of using research evidence in different practice and policy context. To make these stories impactful, they have to be told by the people who used the research evidence. And we as researchers have to be willing to listen and be able to reflect on them with the evidence users.

If at first you don’t succeed, try, try, try and try again…

Posted by Suzanne Moffatt, Reader in Social Gerontology, Institute of Health & Society, Newcastle University

Ways to Wellness Link Workers provide support to people with
 long-term health conditions who are referred by their GP

Publishing a study protocol is always gratifying (even though it counts for nothing in the Research Excellence Framework!). Publication of this particular protocol evaluating the impact of social prescribing on health and wellbeing caused me to reflect on the 58 months that had elapsed between that initial invitation to develop an evaluation and getting the study protocol into print. Why on earth had it taken so long and what had we done in those months? Well, we weren’t twiddling our research thumbs, but we did get quite a few knock backs in our efforts to secure “the big grant”, and I thought it worth sharing with the research community as multiple failures before success is often untold.

We started with an outline bid to the National Institute of Health Research (NIHR) Public Health Research Programme in 2015 which was shortlisted, but not successful at the full proposal stage. Fortunately an application to the Cabinet Office was successful that same year, which enabled us to complete a qualitative study exploring the impact of social prescribing for service users and perspectives of ‘link workers’ delivering the intervention (work currently accepted for publication). In 2015 (busy year!) applications to the School for Public Health Research (SPHR) Public Health Practice Evaluation Scheme (PHPES) and School for Primary Care were both unsuccessful. But in 2016 we achieved success with a ‘cut down’ version of our SPHR PHPES application (capitalising on end of year finances, I believe) that allowed us to pilot questionnaire data collection (paper rejected and currently being prepared for re-submission). 2017 also saw us obtain a relatively small grant from Newcastle University’s Institutes for Ageing and Social Renewal, allowing us to follow up 24 out of our original qualitative sample of 30 service users, findings of which have recently been published.

Ukulele group, Throckley Community Centre, Newcastle upon Tyne

Despite our lack of success so far in obtaining the “big” grant, we had contributed a valuable body of work on the impact of social prescribing and into the bargain employed some excellent researchers – Mel Steer, Kirsty Laing and Jo Wildman. Yet, in spite of the apparently unstoppable popularity of social prescribing as a way of addressing long term health problems and tackling inequalities, a systematic review demonstrated that a robust evidence base about impact and cost effectiveness is lacking. When the 2017 NIHR Public Health Programme call for research on Community Groups and Health Promotion[1] (16/122) hit my inbox, it dawned on me that another application (our seventh in case you had lost count) fitted the bill. I felt a mixture of dread and exhilaration. Definitely the last chance saloon.

So, why was this bid successful? In essence, because it was more clearly focused on a specific condition (type 2 diabetes), our substantial body of underpinning research, the mixing of quasi-experimental and ethnographic methods and a really excellent team of researchers. So now all we have to do is deliver. But remember, if at first you don’t succeed, sometimes persistence pays off.

For more information about our evaluation of Ways to Wellness Social Prescribing, visit the project website.

Applications to the SPHR Public Health Practice Evaluation Scheme (PHPES) are now being accepted with funded projects expected to start from January 2020.


After 10 years’ success, as of 18 July 2025, this initial project by Ways to Wellness is closed following a withdrawal in funding. However, the learning from it provided the foundation upon which Ways to Wellness was able to build its varied current prototypes. Find out more about the work it continues to do at www.waystowellness.org.uk/projects



Reference:

1. This work is funded by the National Institute of Health Research, Public Health Research Programme, Community Groups and Health Promotion (grant no. 16/122/33). The research was informed by a NIHR School for Public Health Research (SPHR) funded project (project reference: SPHR-FUS-PES-WTW).

The views expressed are those of the author and not necessarily those of the NHS, the NIHR or the Department of Health and Social Care.

52 weeks in public health research, part 48

Posted by Amelia Lake and Mark Welford

From Amelia Lake: A sign in an Italian café in Shildon where the energy drink team had a meeting and were about to conduct focus groups. The team Steven Crossley (@akindoftrouble), Shelina Visram (@ShelinaVisram), Mandy Cheetham & me took the opportunity to look around the local area and sample local (Italian) cuisine!

From Amelia Lake: With a two-and-a-half year old the arrival of the festive season means the arrival of advent calendars. So far we've avoided chocolate ones. Long may that avoidance last!

From Mark Welford: Kaffee und Kuchen (coffee and cake) event for the Health and Social Care Institute to celebrate completing the move into our new quarters. The idea was to bring everyone together following the upheaval of the move from Parkside West - I wonder if a Kaffee und Frucht theme would have been as popular?! 

From Mark Welford: Peter van der Graaf getting creative with art supplies borrowed from his children. As an institute we have been challenged to take a selfie and come up with a three word statement that sums up our message to stakeholders for the School of Health and Social Care winter conference. The conference will focus on how we communicate with, and present ourselves to, our key stakeholders.

Our three words: 'Is this REFable?' The Research Excellence Framework (REF) is the system for assessing the quality of research in UK higher education institutions - it's all about research with impact!

------------------
A reminder from the Fuse blog group:

Each Thursday of 2014 we’ll try and post around four pictures on the Fuse blog that capture our weeks in public health research, from the awe-inspiring to the everyday and mundane. Given that more of the latter than the former exists in most of our lives, we foresee problems compiling 208 images worth posting on our own. So this is going to have to be a group project. Send an image (or images) with a sentence or two describing what aspect of your week in public health research they sum up and we’ll post them as soon as we can. You don’t have to send four together – we can mix and match images from different people in the same week.

Normal rules apply: images you made yourself are best; if you use someone else’s image please check you’re allowed to first; if anyone’s identifiable in an image, make sure they’re happy for it to be posted; nothing rude; nothing that breaks research confidentiality etc.

Email your posts to m.welford@tees.ac.uk or contact any member of the Fuse blog group.

Aim high, or just get it out there and move on?

Posted by Jean Adams

So shall we talk impact factors?

Every peer-reviewed journal that has been around for a few years has an impact factor. It’s a number (calculated to three decimal places!) that reflects the average number of times papers published in the journal are cited by other papers in peer-reviewed journals.

In an unthinking world, there is an assumption that the higher the impact factor of a particular journal, the ‘better’ the journal is. From there, it’s easy to start to think that the papers published in journals with higher impact factors describe ‘better’ research and that the people who write papers published in journals with higher impact factors are ‘better’ researchers.

It's just much easier to judge who's 'best' in some areas than others

Of course you can see that such a simple metric is never going to be a particularly accurate reflection of the quality of either the research or the people writing the research. Lots of people have written lots of things about how bad a job impact factors do of assessing quality. They’re even pretty poor at predicting the number of citations any particular paper will get. And who said that citations are a good marker of how good a paper (or the research it describes) is anyway?

I know all of this. I understand it. Yet I still care about impact factors. I still want to submit my papers to journals with high impact factors and it frustrates me when my papers get rejected by the ‘best’ journals and I have to do the long, boring job of trying the next ‘best’ journal, then the next ‘best’, and then the next ‘best’. Which is not to say that every paper I’ve ever published has been rejected by The Lancet (certainly none have been published there). But that I do want the best possible for each of my little paperlings.

Lately I’ve been doing quite a lot thinking about this, wondering why I am so obsessed by playing the impact factor game that I know to be such a highly suspect marker of skill – as much ludo as chess. I guess some of it is vanity – if that’s what people are going to think makes me ‘good’, then I’ll do it because I definitely want them to think I’m ‘good’ (although, why do I care what those people who place such value in such a suspect number care?). Some of it is pure competitiveness – if it’s hard to do that, then I’m going to do it.

But I guess right at the very bottom of it all is the REF. The big ol’, not quite sure that we think that’s a particularly good marker of quality either, REF. I have to have four papers that are ‘good enough’ for the REF or to be submitted as an individual; I definitely want to be submitted (see above re vanity and competitiveness, but let’s also add in it’s sort of my job); and in the absence of other markers of quality, we seem to be relying pretty heavily on impact factors to decide what papers are ‘good enough’. Not entirely. But quite a bit.

Whilst many of the people I work with are similarly fairly obsessed by the impact factor game. Not all of them are. Some people, who I have lots of respect for, reckon it’s a bit fat waste of time (and morale) to try your paper in a high impact factor journal, get it bounced, reformat it for the next one, get it bounced blah blah blah. They laugh and say ‘oh just get it out and move on’. Maybe it isn’t a coincidence that these tend to be the sort of people who have loads of ‘good enough for the REF’ papers anyway. But I am starting to wonder if what I need to be doing is not constantly ‘aiming high’, but developing a better sense of what papers are worth ‘aiming high’ for.

How to establish a reputational yardstick

Posted by Heather Yoeli

As anyone who has been around any university research department recently will probably have realised, the general atmosphere within academic circles is, erm, interesting. The deadline for submissions towards the next REF is swiftly approaching, and preparations are escalating towards a level of intensity last seen amongst those who trusted Harold Camping’s prediction that the world would end on May 21 2011. I do hope that none of my colleagues will take offence at such an analogy. But the 2014 REF does remind me of Harold Camping in that its rules, like the reasoning behind his prophecies, appear to the novice eye as rather convoluted.

Harold Camping predicts the end of the world

Basically, the REF’s main website explains itself as follows;

The REF will be undertaken by the four UK higher education funding bodies. The exercise will be managed by the REF team based at HEFCE and overseen by the REF Steering Group, consisting of representatives of the four funding bodies.

The primary purpose of the REF is to produce assessment outcomes for each submission made by institutions:

· The funding bodies intend to use the assessment outcomes to inform the selective allocation of their research funding to HEIs, with effect from 2015-16.

· The assessment provides accountability for public investment in research and produces evidence of the benefits of this investment.

· The assessment outcomes provide benchmarking information and establish reputational yardsticks.

The REF is a process of expert review.

Is it just me, or is that not entirely clear? I mean, I can sort of surmise that it’s something to do with telling universities how good their research is and might involve some cash prizes, but I’m not sure that the phrase ‘establish reputational yardsticks’ is one I’d use in everyday conversation. In the public health context, I think a reputational yardstick probably has something to do with the apparent REF buzzword of impact – that is, making our research relevant beyond academia and beneficial to the wellbeing of society and doing much of what we’re doing anyway with what we call knowledge exchange – but still, it’s a rather clumsy phrase. I would hate to trip over a reputational yardstick in the corridor.

My worry for the 2014 REF is that, as much is it will have achieved a great deal in encouraging universities and their researcher staff to work hard and to do their very best, its so-called impact may be limited by the way in which few people who are not themselves university researchers really understand enough about the REF itself. In my own completely unqualified opinion, the HEFCE might want to consider how to give the next REF a reputational yardstick as something just a little less impenetrably confusing.

The Pig Observation

Posted by Peter Tennant

You don't make a pig any fatter by weighting it. For years this has been my favourite animal-related moral, narrowly beating The Fox and the Grapes*. The pig observation is a classic remark on the value, or lack thereof, of examination as an educational device. You don't make a child any cleverer by testing it. Yet for every summer of my childhood (and much of my early adulthood), I was ritually weighed. Since I had a good memory, I rather enjoyed it. But much I believed that recounting the four chambers of a cow's stomach proved I was smart, it didn't; it just proved I could remember the four chambers of a cow's stomach.

This October, UK higher education takes part in its own pig weighting festival, the Research Excellence Framework (or REF). The premise is reasonable. The UK government has £1.76 billion that it wants to spend on research. Rather than dole it out equally, it's decided it's fairer (and more cost effective) to try and give the money to the places that do the most and the best research.

So far so fair. But how do you go about such a task? They could have done a complete audit. Each university presents the sum of all its research (in the form of books, publications etc) and the money is then given to those that have done the most. Except quality obviously matters more than quantity, so only those 'outputs' over a certain minimum standard will be deemed worthy of reward, with standard judged in the usual academic way, i.e. by groups of dusty professors. Presumably they were offered biscuits; professors will do anything for biscuits. Of course, they can't be expected to read everything (have you ever tried to read an academic paper? YAWN!), so instead universities only have to provide a selection of outputs for each researcher, and they don't have to include every member of staff either.

Sounds reasonable enough? Except you now have the formula for a right old mess. Especially once you add that as well as deciding how the money gets spent, the REF will also be used to judge reputation.

Poppleton University: Home to the fattest pigs outside the banking industry

Most people agree that academic league tables are a pretty useless way of comparing schools. Yet schools still do everything they can to maximise their standing. My school focused relentlessly on the C boundary (the grade considered a good pass in the English aged 16 examinations). If you were safely above it, you were on your own. If you were too far below it? Well, around 50 kids were expelled at aged 15 for 'behavioural problems'. The fact they had no hope of getting 5 subjects at C or above was presumably a convenient coincidence.

Similar shenanigans were apparent during the last audit of UK academic research. Some departments amusingly entered only a handful of their best performing staff to boost their showing. This time, star researchers are being parachuted in like premier league football stars and some places are even employing specialist REF gurus to help boost their submissions.

But isn't this all just harmless pig trading? Well, there's another more insidious problem. You can't measure something without changing it. In fact, you can't even observe something without changing it. In a remarkable experiment that not only forms the foundation of quantum physics, but also (and more importantly) forms my favourite non-animal related moral, it can be shown that the universe itself will mischievously change its behaviour when it's being observed. Fire a beam of electrons at two narrow slits and they will make a pattern as if each electron has miraculously gone through both slits at the same time. Set up monitors on each slit to watch this happening and... the pattern changes... to the boring 'one electron per slit' one that you would classically expect. Somehow the electrons know when they're being watched and behave differently. Like me if I'm practising a silly voice and then realise someone can hear me.

In trying to measure 'quality', the REF changes the research environment in a number of ways, the biggest being a dangerous shift towards short-termism. To be REFable (the quite unsavoury term used to mean 'worthy of submission') a member of staff usually needs (among other things) four high quality publications within the audit timeframe. Firstly, this means that a lot of staff are going to be branded as 'unworthy' despite producing work that is 'recognised internationally'**. Which isn't very nice. Or very motivating. If a group of dusty professors told me that my research wasn't any good, I'd instantly conclude they're all past it. So yes, I'd be just like the fox in the Fox and the Grapes. Only with a slightly less bushy tail. Secondly, focussing on 'four high quality publications' doesn't leave much time for any real scientific exploration, i.e. the kind that takes a lot of time and repeated effort to perfect. Like Darwin's On the Origin of Species. 20 years to write a single book? There's no way Darwin would have been REFable.

This shortermism is further exaggerated by the REF's attempt to measure 'impact'. The argument goes like this: since the public pay for academic research they deserve to see tangible benefits. Thanks to 'evidence based medicine' this isn't too hard for someone working in public health, just come up with some good evidence and eventually it'll filter through the system***. But for other subjects… Computing? Electricity? E=MC^2? List the world's greatest intellectual advances and most of them have one thing in common: it took years for any of them to have a measurable 'impact' on society. Then they changed the world.

So next time you set about trying to measure something, think carefully about the possible side effects. In other words, beware Goodhart's Law: 'When a measure becomes a target, it ceases to be a good measure'. Targets are blamed for many evils within the NHS. The REF seems pretty tame in comparison. It may cost millions, stress out thousands, and undermine the very heart of scientific endeavour, but there was no alternative, especially not one that was considerably cheaper and easier... Actually, scratch that last point... I don't know about you, but I fancy a bacon sandwich…



*The Fox and the Grapes is my favourite of Aesop's fables, and a classic illustration of cognitive dissonance. A hungry fox spies some grapes hanging from a tree. Several times he tries to reach them, but he can never quite jump high enough. Eventually, he gives up and concludes the grapes were probably sour anyway, thereby reducing the dissonance between his desire to eat them and his inability to obtain them.

**The REF will grade research into four categories. Unless the work is considered at least 'internationally excellent' (i.e. 3 star) it won't be considered worthy of funding.

***Well, that's the idea. Unfortunately, politicians don't always like 'evidence'. Especially when the evidence works against their financial interests.