Wednesday 11 July 2012

How to get the evidence message across

Guest post by Katie Cole

The mantra of “but there’s no evidence for it!” is one I’ve said or thought many times, both in my work, discussions with family and friends, or when shouting at the BBC Today programme.


But as an early-career academic, I’m increasingly aware there is a complex web of considerations when trying to translate evidence into policy, and that there are times when chanting our mantra may do more harm than good.

I recently attended a Royal College of Physicians/Alma Mata seminar on alcohol advocacy. At one point, a panel member suggested that social norms interventions to address excessive alcohol consumption on university campuses “sounded very promising” and policy-makers were considering it. I’ve looked into US research into these interventions: a national evaluation concluded that they are ineffective in reducing alcohol consumption. Whilst I could have made this point, I felt it was more complex than that. Don’t we need to test the policy in the UK drinking context to make a more robust contribution to the debate? Shouldn’t we seek to support policy-makers to integrate evaluations into pilots, or to finance full-scale trials?

Another challenge I’ve had was during a placement at a Primary Care Trust. I was involved in the Individual Funding Request process, where the PCT considers funding treatments and procedures not normally available on the NHS. I worked up a number of cases, looked at the evidence base and presented the case to a panel of clinicians and non-clinicians. In most cases, the evidence base was of poor quality: finding a case series for the exact condition and treatment in question represented a minor professional achievement. Usually, the case series found that, lo and behold, most cases improved, which often sparked disproportionate optimism that we had a justification for funding the treatment. In contrast, when I found a randomised controlled trial with only modest results, the panel were more inclined to propose not funding the treatment. Here I was challenged to explain the difference between the strength of the evidence base, and the strength of the effect size; whilst at the same time, acknowledging the difficulty of decision-making against a poor evidence base.

A final challenge has been in developing The Lancet UK Policy Matters website, which includes short summaries of the evidence underpinning a range of UK health-related policy changes. In developing the format of the summaries, we had to be very clear to authors that statements purporting the intended benefit of the policy should not be included in the ‘evidence’ sections of the summary – this was reserved for peer-reviewed research or evaluations. Our experience in guiding authors highlighted to us how meticulous we as professionals need to be in the choice of language we use when drawing on our scientific expertise.

Above all other lessons, these experiences have taught me that advocating for evidence in policy making is challenging, complicated and requires skill. It demands an understanding of the evidence itself – its strengths and limitations – but also of the policy making process. Whilst these issues can be difficult to reconcile, the above experiences have only strengthened my drive to communicate effectively with all actors in the policy making process.

Katie Cole co-founded The Lancet UK Policy Matters website with Rob Aldridge and Louise Hurst.

9 comments:

  1. I used to work for a Northern PCT. I tried to use evidence-based practice. I was told to stop by my manager. She said not to do so showed a lack of political awareness that was an important element of the public health role. This really meant simply do what the Director of Public Health says without question.

    I complained to chief exec and the chair but they simply ignored me, breaking a host of Department of Health regulations in the process.

    If anyone else, has had similar experiences I'd be interested in hearing about it on this blog.

    ReplyDelete
    Replies
    1. Anonymous - this is a really interesting (depressing, sadly not that surprising) story and we would love to hear more about what you think the issues were in terms of creating an evidence-based 'culture'. Perhaps you might consider writing a blog post on this topic? Nothing defamatory - more constructively exploring barriers and lessons learnt. We could publish anonymously if you would prefer. If you're interested, get in touch with me the editor - jean.adams@ncl.ac.uk

      Delete
    2. Thank you for your comment.

      I think your experience resonates with points in my post - there are times to speak out as an evidence advocate and times to show political awareness to increase your overall effectiveness of improving population health.

      For example, if the board wants to fund an intervention that doesn't have a great evidence base but is politically popular, making a stand may isolate you from the subsequent decisions about funding other more effective interventions at the same time.

      It's a tricky call to make which demands experience and insight, and depends very much on the individual situation, so I can't comment on yours in particular. I hope you find a satisfactory resolution though.

      Delete
  2. just as a clinician horns his skill at the bedside, a public health practitioner has to horn his at the edge of politics. evidence is paramount but for it to see the light of day, it must weather and coexist with political rhetoric. there are no hard rules on the line between these two and one needs to individualize every situation.

    ReplyDelete
  3. What do we mean by 'politics'?

    It can mean of course going along with the political policy of which ever party is in power.

    It can also mean totally ignoring the same policy e.g. around the importance of evidence. This could be simply because one individual in an organisation e.g. a DPH, has a personal style that is autocratic. The politics of fear.

    I think an issue that needs putting on the table is how to actually create organisations that do empower staff to do a good job. I know the rhetoric says this happens already but does it? My experience says 'no' but I'd like to see proper research into what the overall picture is and what we can do about it.

    Similarly when you carry out evaluations, how do you highlight that many of the problems were interpersonal or due to the culture of the organisation that you work for? This is evidence too, vital evidence but difficult to obtain.

    ReplyDelete
    Replies
    1. 'Politics' refers to who gets what, why, when and how.

      It can refer to
      * How national political agendas are taken up or not by public sector bodies
      * Party politics in a local authority context
      * Community politics i.e. how the public sector relate to the public
      * Interagency politics
      * Intra-agency politics between different departments
      * Interpersonal politics within a department i.e. colleague to colleague or boss to staff

      There may be similarities in all of these 'politics' but I think it is useful to distinguish which is meant when people talk about politics in a public health context. Then we can look at the practical and ethical considerations and what to do about them. The response may focus on skilling up individuals or ensuring that systems work so that some behaviours are not tolerated e.g. bullying at work.

      Delete
  4. The problems of accounting for personality or culture at individual and organisational or societal levels respectively are largely ignored in most evaluation research. We have recently questioned this in relation to an apparently highly successful community based intervention for weight loss. Our instinct is that the apparent success may have most to do with the personalities of the highly enthusiastic and committed intervention staff, rather than the intervention design (which is actually not that well founded!). Similarly, we can probably all think of an inspirational leader who really made a difference in an organisation, despite all the usual resistance to change. These individual exceptions to the rule should be researchable. We ought to be able to identify the characteristics of effective change-makers (for example using the Taxonomy of Behaviour Change Techniques and Theory Domains Framework). The bigger challenge is to make these characteristics transferable and hence enable generalisability of effective interventions. That would be a real advance in translation.

    ReplyDelete
    Replies
    1. The importance of the personality of the person doing the intervention is an issue that is discussed in psychotherapy too of course.

      Delete
    2. Another area that may be worth looking at is what negative qualities in leaders go against using evidence-based practice e.g. being very self centred, controlling or suspicious of other people and acting on whim. Being very ideological may also get in the way.

      Delete