Systematically unsystematic: confessions of a novice reviewer

Posted by Amy O'Donnell

Two years ago, I somehow found myself agreeing – nay proposing – that I should do a systematic review as part of my PhD.

For those unfamiliar with the term (*searches Wikipedia*) a systematic review involves a literature search focused on a clearly formulated question that aims to identify, appraise and synthesise all high quality research evidence relevant to that question.

As far as I was concerned, the findings would usefully inform the other elements of my research, the process would enable me to learn valuable new skills, and the material might even be publishable. What was not to like?

Well for starters there was finding out exactly how to do it (the starting point turns out to be a 281 page guide that drops vaguely menacing terms like ‘stochastic’, ‘bootstrapping’ and a whole bunch of ‘a priori’s). Then there were the hours of screen time trawling through journal abstracts, thousands of which were irrelevant to my research. Throughout, I was keenly aware that the path from conception to realisation was littered with the exhausted shells of well-intentioned PhD researchers.

Fast forward to today, and I am only now (almost) in a position to start writing that elusive final report. So I ask myself – as indeed my supervisors have asked on numerous occasions – just what went wrong?

Well aside from the dithering, the various displacement activities (another colour-coded spreadsheet anyone?) and a general lack of get-up-and-go, I think it all boils down to one major problem.

I didn’t really know what I was looking for.

Sorry, where am I supposed to be going?

Don’t get me wrong, I had a vague idea. But it turns out that a vague idea just won’t cut the mustard when it comes to navigating your way through the twenty-one million plus records available via PubMed. Without a clear question to be answered or hypothesis to be tested, you are looking for the proverbial needle in a haystack. I didn’t know what literature to keep in, and what to throw out. I couldn’t articulate exactly what the review would contribute to my research. And I simply hadn’t thought carefully enough about how I would synthesise the results at the end of the whole process.

I confess it took the shame of failing my second year assessment to wake me up to this sorry state of affairs. But after several months of repeated trips back to the drawing board, and countless discussions with supervisors and other colleagues, I think I’ve finally got it. It’s not perfect, it’s going to be hellish to write up, and I’m annoyed with myself because I know I could have (should have) got here far more quickly and far less painfully. But it’s giving me some relevant and useable material, and I’ve learned a lot. Which I guess is the point.

As poet Edward Hodnett (sort of) wrote “If you do not ask the right questions, you don’t have a hope in hell of getting the right answers”.