In my previous post I shared the Keynote presentation I created for the module on systematic reviews I am doing at the IOE. In it I proposed a metaphor for systematic reviews which described them in terms of a photomosaic. I quite like this metaphor and want to explore it a bit further.
A photomosaic is an image which is created by using thousands of smaller images.
Here is an example.
[images source here]
Although there is sophisticated software that can create these mosaics automatically, some are painstakingly created by hand. It is the hand created ones that I think serve as a useful metaphor for systematic reviews.
Locating potentially relevant studies: To conduct a systematic review the authors must first search for literature that is relevant to the question that they aim to address. This is equivalent to the creator of the Mona Lisa image above locating the source images that he* will combine to make the final image. The artist has decided that he will use fine art images in his creation, so his predefined search protocol specifies any picture that is of fine art. The images actually used will be refined later, but at this point he wants to cast as wide a net as possible so as not to miss any potentially useful images.
Appraising potentially relevant studies: Once potentially relevant studies have been located, the authors of a systematic review must appraise each one based on pre-defined inclusion criteria, for example the way participants were allocated to comparison groups in experimental studies. In my metaphor, the artist must examine each image to decide whether it fits his inclusion criteria. For example, he has a pre-defined colour pallet, so only images which conform to that colour pallet will be included.
Synthesising the results of relevant studies: Once the studies have been appraised, and accepted or rejected for inclusion in a systematic review, the authors must synthesise their results. That is to say, they either aggregate the results of mainly quantitive studies (for example by using meta-analysis) to help judge the cumulative findings of all of the relevant research. Alternatively they configure the results of mainly qualitative research in order to organise the findings into a coherent summary of the evidence. My metaphor lends itself best to configurative systematic reviews (because when you add up lots of pieces of fine art you don’t get a Mona Lisa!). The artist has carefully configured the images by colour, tone, shape and form so that systematic synthesis in this way creates a new image.
The report/final artwork: In systematic reviews the studies that meet inclusion criteria each tell their own story. These stories can be very similar (for example two studies conclude that an intervention improves outcomes) or they can be contradictory (for example, of two studies one concludes that an intervention improves outcomes and the other concludes that an intervention does not improve outcomes). In a similar way, the images making up the Mona Lisa sometimes have similar ‘stories’ to tell (for example a number of the constituent images are portraits of people), and some have differing stories to tell (for example, as well as portraits, some of the constituent images are landscapes). Only when they have been systematically combined does a coherent picture emerge where inconsistencies have been ‘smoothed out’ by the accumulation of information. However, it is not totally smooth. There are jagged edges and places where the rendering is imperfect. Just as when new studies have been conducted, their findings can be incorporated into existing systematic reviews to assess whether the additional knowledge changes to the cumulative findings, a newly discovered piece of fine art might be incorporated into the Mona Lisa, potentially clarifying the resulting image.
Like all metaphors, this one breaks down in places. The main problem is that the artist who makes the photomosaic has an end in mind when he embarks on making it. He knows he wants to end up with a Mona Lisa and so sets his inclusion criteria to only incorporate images that will help to produce one. In this way it might be better compared to a non-systematic review, one where only research that fits the authors’ prejudices are selected. In my view, agnosticism about the possible conclusions of a systematic review is the only defensible position to take when embarking on the process for the first time. If you know the answer you ‘want’ before you start then you cannot be regarded as agnostic and your prejudices might introduce bias to your findings.
Another area where the metaphor begins to break down, as I have touched on already, is that it is insufficient to describe aggregative reviews. You don’t get a Mona Lisa by adding up a load of pictures of something else. However, you do sometimes get surprising results. For example, a review of the effectiveness of Drivers’ Ed programmes for young drivers concluded that they are associated with increased car crash rate. This is where agnosticism in the face of unclear evidence is important.
However, despite some of the problems with this metaphor, I like the idea that it involves locating potentially appropriate source material (studies/images) that may tell very different ‘stories’, appraising the source material for appropriateness for inclusion (methodology/colour), and systematic aggregation or configuration of the source material (meta-analysis/careful arrangement based on colour and form) to produce a outcome which reflects the totality of the source material.
*I have no idea of the gender of the artist who created the Mona Lisa image. I don’t like the ‘he or she’, ‘s/he’ and ‘his or her’ constructions in writing, and using the plural does not work well in this situation. On occasions like this I toss a coin to decide the gender I will use. It came up male for this post.