Thursday, 13 September 2012

On evidence

Posted by Simon Howard

In my first week at medical school, one of the professors warned that most of what we were to be taught was factually wrong. It was an arresting statement, but it may have been true: Studies have shown that textbooks and experts frequently lag behind evidence, sometimes recommending “treatments” that are actually known to be harmful.

Do Primary Care Trusts do the same? PCTs, like the one I work in, currently commission the majority of NHS services provided to patients in their catchment areas (though not for much longer). Sometimes, academics get frustrated with PCTs for seemingly doing things that either have little evidence, or appear to contradict it altogether. Given that evidence is the bedrock of public health, and given the potential for decisions to affect whole populations, this might seem worrying.

In defence of PCTs, a lot of evidence based work does happen. Most major pieces of work include a review of academic literature at an early stage, and follow the findings. The annual Joint Strategic Needs Assessment and regular detailed Health Needs Assessments also take into account published literature and local and national data in a fairly systematic way.

But there are lots of barriers to following the evidence. Books and books could be written on this topic, from the applicability of evidence in the real-world to deciding if research is really relevant to a particular population. But I’m no expert, and I’m not going to try and describe anything technical, complicated, or even remotely clever. These are just a few examples of practical barriers to following the letter of the academic evidence in public health.

One huge barrier is – as with most things in life – money. In a world of ever-tightening budgets, an academic’s seemingly reasonable intervention can be unaffordable. As an extreme example, research by the FAA and CAA suggests that three or four lives would be saved in an average aircraft fire if all passengers were provided with smoke hoods. However, the vanishing rarity of in-flight fires, the enormous cost of supplying and maintaining smoke hoods, and the cost of the fuel required to propel them around the world, all make this proposal financially unjustifiable.

Not all examples are quite so clear-cut. Sometimes, instead of choosing not to do something, PCTs try to cherry-pick the best bits of interventions in a way that is almost certainly infuriating to the academics who pioneered them, and possibly less effective in practice. But, sometimes, doing something is better than doing nothing.

Often, there can be a big lag between publication of evidence and its implementation. One reason is the complex contractual nature of commissioning: it’s often difficult to make small changes to services that have already been commissioned. The constant pressure to reduce costs incentivises longer contracts which spread the financial risk, but which also increase the evidence-practice lag. I’m sure it’s deeply frustrating to be an academic shouting “there’s a better way to do this” while services continue unchanged.

There’s also a political element to public health. Decisions to cut services that are no longer supported by evidence are particularly tricky. In England and Northern Ireland, the evidence that cervical screening in women under the age of 25 causes more harm than good has led to a withdrawal of the service in this age group. The clear evidence, combined with clear recommendations from the World Health Organisation and National Screening Committee hasn’t stopped this becoming a topic for political debate and petition, and hasn’t (yet) changed policy in Wales or Scotland. It seems likely that this political element will play a bigger part in decision making as public health moves to the overtly politicised world of local authorities.

To me personally, the most frustrating barrier to following the evidence is an inability to access it. It continues to baffle me that the NHS doesn’t have anything like the level of straightforward desktop access to literature that university colleagues have. In the 21st century, it seems crazy that I sometimes have to ask the BMA to take a paper journal off a physical shelf, scan it in, and email it to me as the only practical cost-effective way to access a paper that’s of general interest, rather than something specific to any individual project.

I think a latent awareness of what’s going on in academia is important in public health. It might not matter so much when someone’s doing a big literature review prior to introducing a new service, but it can help with horizon-scanning, and with those little every day decisions that aren’t worthy of a trawl though the literature, and with planning for the future. This is something we can all play a part in: public health professionals probably need to broaden their awareness of the academic things going on around them, and academics probably need to shout louder about the latest developments in their fields. As an associate member, I’m probably biased, but I think FUSE is great at helping both groups.


  1. Simon - thanks very much for your post, which I enjoyed reading and with which I agree. Evidence is vital to safe, effective and efficient health services and public health interventions, but we cannot afford to let the best become the enemy of the good. As academics we need to understand the (limited) role that research evidence plays in decision making. For their part, practitioners and policy makers need to understand the constraints of generating and synthesising evidence in a timely fashion. However, I absolutely agree that we need to work together to remove whatever barriers currently exist to accessing evidence. Fuse, in common with many other research groups and research funders, should adopt a policy of publishing only in open access journals. MRC, which is handling our renewal application, has made it clear that funds for open access publishing will be freely available over the next 5 years and we will avail ourselves of this as a matter of policy. I hope that Fuse can also work with the NHS (and local government) in the North East to help break down barriers to accessing research, by engaging in negotiations with our five host universities. Let's get a group together to work on this as a part of our translational strategy.

  2. Hi Martin, thanks for your kind comments. I agree with all your points!

    I think open access publication is an important step, though of course shifting fees to researchers rather than readers has its own problems, particularly in terms of publication bias and tricky conflicts of interest for the publishers. Having said that, I think PLoS ONE's approach (and increasing popularity) in particular is proving that non-traditional publication models can work. I think the evolution of scientific publication will be one of the most fascinating things to observe (and, who knows, maybe influence) over the course of my career. In 40 years time, I suspect the landscape will look very different.

    For now, I think it's a great idea to get a group together to work on breaking down barriers to accessing research in the North East. I'm on study leave this week, but perhaps we can arrange to think about this together some time soon?