Showing posts with label KOliver. Show all posts
Showing posts with label KOliver. Show all posts

Tuesday, 16 July 2013

Systematic reviewing: soporific or scholarship?

Guest post by Kathryn Oliver

For several years, I was employed by a systematic review facility which provided review evidence for various government departments. We did full systematic reviews (about 18 months), rapid reviews (about 8 month), systematic maps, papers of all of the above... I read a lot of papers and learned a lot. One important lesson was about the difference between a good quality research study, and a good quality paper. We did our quality appraisal on the reportage of a study – not the study itself. If a trial didn’t mention randomisation, we assumed it wasn’t. If it said it was randomised, but didn’t say how, it got a few more points; randomisation methods reported (tossing a coin, birthdays, researcher selection (!!!), computer-generated) got them a few more. In short, I learnt a lot about social and clinical research methods, and how to write them up....


But I started to worry that all this reviewing was having a pernicious effect on me. Apart from the effects of occasional tedium, I wondered – was it making me a bad scholar? The reviews we did were high quality and published in good journals; there was consensus that we were doing a good job. But I still worry about missed opportunities. Without criticising my excellent training or support from systematic review colleagues, I present some personal reflections on what I could have done better:

1. Not ignore everything about the papers except for the methods and findings

Since my reviewing days, I’ve gone on to do other research jobs, a PhD and a post doc – so I’ve now grappled with writing research papers myself. I now appreciate the thought and craft that goes into introductions, discussions, conclusions, implications; all loaded with high-quality academic thought and really, the parts you want other people to take notice of. But did I as a systematic reviewer? No I did not. I went straight to the methods (“another rubbish paper then”) and then to the findings (“they haven’t even controlled for income! losers”) and ignored the rest. All that thought, all those carefully constructed implications for other researchers – out the window.

2. Take the implications further

Most systematic reviews (if not most papers these days) have a section called “Policy and Practice Recommendations” or similar. A systematic review may include, say 100 papers – that’s a lot of recommendations. Now I wonder, why didn’t we - or research funders, for whom it’s probably even more important – collate these into a “state of the field” list of recommendations on which there was consensus or disagreement?

3. Collate the conclusions

“More research is needed” was probably the conclusion of most papers I read. To be fair, some did accurately describe precise research questions – and these again could (should?) have been collated into list of outstanding questions which could feed into the agenda setting for research funders, used in priority-setting with patients and the public, or just posted somewhere to inform future grant applications.

4. Draw on more types of information

Despite advances in systematic reviewing methods to develop ways to synthesise evidence from non-trial evidence, there is still a huge emphasis on empirical research data – for entirely understandable reasons. I still wonder if there are opportunities to draw on other type of research output though: commentaries and conceptual pieces, unpublished datasets (especially to answer questions raised by the review), policy pieces to give context and feed into recommendations...

I still do systematic reviews, and I still don’t do all these things, partly because of the time, lack of funding, and so on and so forth. I do try and write a commentary about the review now though, reflecting on the ‘state of the field’ which is a useful exercise for me, if no one else. These days, I try and think of a systematic review as a way to really learn about what is going on in a body of literature, and an opportunity to engage with the current debates. I think the trick is giving yourself the time to think, and not just plough through that huge pile of full-text retrieved....

Thursday, 20 September 2012

Making the connection

Guest post by Kathryn Oliver

About 2 years ago, I had one of those ‘Eureka’ moments that totally changed my life. Genuinely. It was right up there with finding out about Oyster cards, or washing machines, or something.

At the time, I was a PhD student in my first year, working on a fairly standard project about developing health indicators. As a project, it was fine – about the use of evidence by policy makers, one of my main interests, and I was getting lots of experience in survey design. But for years, I’d been kicking round ideas in my head about the importance of personal relations. Didn’t they really explain nearly all human behaviour? Weren’t peer effects important for the spread of obesity or smoking? Wasn’t social capital important for mental health?

I’d been living on my own in London for a year or two and had found myself pondering the role of human relationships more and more. Of course, I had friends and relations, but I also liked being known by the man in the newsagents and the end of the road, and saying 'hi' to the neighbours. Did they count, I wondered? Would these relationships be enough to protect me from isolation, or going ballistic on the tube?

Imagine my delight when, attending a Social Network Analysis seminar day, run by the Mitchell Centre at the University of Manchester, I discovered an entire body of research – methods, philosophy, approaches – which looked at connections between individuals using formal statistical methods. Finding out that other people had had similar ideas to me, and had developed dedicated research methods to investigating these ideas was probably one of the best research moments I’ve ever had.

Unlike traditional statistics, network analysis does not treat individuals (whether bridges, policy makers, or swingers) as independent. Instead, any ties between actors are identified, described quantitatively and/or qualitatively and mapped. The statistics used are based on graph theory, but you don’t have to understand it to admire the elegance and usefulness of network analysis. Depending on the relationship collected, people’s attitudes, behaviours, health outcomes and more can be predicted.

For me, this is really the missing element from a lot of public health research. It can be used to identify good targets for research, or opinion leaders in secondary schools, so more targeted messages can be produced and sent out. It allows us to understand, describe, and analyse the social context within which individuals live. And, of course, make beautiful pictures. 

Example of Social Network Analysis diagram.
People have used network analysis to study all kinds of things – it’s very popular in the business world to identify ‘future leaders’ or ‘people who make things happen within my business’. Researchers have compared US senators voting patterns to cows who lick one another.

My PhD changed quite a lot after this seminar. I ended up using a combination of social network analysis and ethnography to study where public health policy makers found evidence, who the main sources of evidence were and how evidence was incorporated in the policy process. For years, academics in my field have been talking about the importance of interpersonal knowledge translation and how policy makers prefer to get their info from real people. Now I’ve been able to add my own tiny part of the story, come up with new research ideas on the basis of my findings, and learn a niche method (always useful).

My boyfriend still calls them snog webs though.