The view from the top of the hierarchy of evidence

About five years ago I began doing meta-analyses. (If, as they say, you lose a tooth for every meta-analysis you conduct, I am now gumming my way through my food.) I was inspired by their growing role as the premier source of evidence in the health and behavioural sciences. Yes, I knew, individual studies are low-powered, depend on very specific methodological assumptions, and are often badly done; but I was impressed by the argument that if we systematically combine each of these imperfect little beams of light into one big one, we are sure to see clearly and discover The Truth. Meta-analysis was how I proposed to counter my mid-life epistemological crisis.

I was therefore depressed to read a paper by John Ionnidis, he of ‘Why most published research findings are false’ fame, on how the world is being rapidly filled up with redundant, mass produced, and often flawed meta-analyses. It is, he argues, the same old story of too much output, produced too fast, with too little thought and too many author degrees of freedom, and often publication biases and flagrant conflicts of interest to boot. Well, it’s the same old story but now at the meta-level.

Just because Ionnidis’ article said this didn’t mean I believed it of course. Perhaps it’s true in some dubious research areas where there are pharmaceutical interests, I thought, but the bits of science I care about are protected from the mass production of misleading meta-analyses because, among other reasons, the stakes are so low.

However, I have been somewhat dismayed in preparing a recent grant application on post-traumatic stress disorder (PTSD) and telomere length. The length of telomeres (DNA-protein caps on the ends of chromosomes) is a marker of ageing, and there is an argument out there (though the evidence is weaker than you might imagine, at least for adulthood) that stress accelerates telomere shortening. And having PTSD is certainly a form of stress. So: do people suffering from PTSD have shorter telomeres?

It seems that they do. There are three relevant meta-analyses all coming to the same conclusion. One of those was done by Gillian Pepper in my research group. It was very general, and only a small subset of the studies it covered were about PTSD in particular, but it did find that PTSD was associated with shorter telomere length. As I wanted some confidence about the size of the difference, I looked closely at the other two, more specialist, meta-analyses.

A meta-analysis specifically on PTSD (by Li et al) included five primary studies, and concluded that PTSD was reported with shorter telomere length by -0.19 (95% confidence interval -0.27 to -0.10). All good; but then I thought: 0.19 what? It would be normal in meta-analyses to report standardised mean differences; that is, differences between groups expressed in terms of the variability in the total sample of that particular study. But when I looked closely, this particular meta-analysis had expressed its differences absolutely, in units of the T/S ratio, the measure of relative telomere length generally used in epidemiology. The problem with this, however, is that the very first thing you ever learn about the T/S ratio is that it is not comparable across studies. A person with a T/S ratio of 1 from one particular lab might have a T/S ratio of 1.5 0r 0.75 from another lab. The T/S ratio tells you about the relative telomere lengths of several samples run in the same assay on the same PCR machine with the same control gene at the same time, but it does not mean anything that transfers across studies like ‘1 kilo’, ‘1 metre’ or ‘400 base pairs’ do.

If you don’t use standardized mean differences, integrating multiple T/S ratio studies to obtain an overall estimate of how much shorter the telomeres of PTSD sufferers are is a bit like taking one study that finds men are 6 inches taller than women, and another study that finds men are 15 centimetres taller than women, and concluding that the truth is that men are taller than women by 10.5. And the problems did not stop there: for two of the five primary studies, standard errors from the original papers had been coded as standard deviations in the meta-analysis, resulting in the effect sizes being overstated by nearly an order of magnitude. The sad thing about this state of affairs is that anyone who habitually and directly worked with T/S data would be able to tell you instantly that you can’t compare absolute T/S across studies, and that a standard deviation of 0.01 for T/S in a population study simply couldn’t be a thing. You get a larger standard deviation than that when you run the very same sample multiple times, let alone samples from different people. Division of labour in science is a beautiful thing, of course, and efficient, but having the data looked over by someone who actually does primary research using this technique would very quickly pick up nonsensical patterns.

I hoped the second meta-analysis (by Darrow et al.) would save me, and in lots of ways it was indeed much better. For PTSD, it included the same five studies as the first, and sensibly used standardized mean differences rather than just differences. However, even here I found an anomaly. The authors reported that PTSD was associated with a much bigger difference in telomere length than other psychological disorders were. This naturally piqued my interest, so I looked at the forest plot for the PTSD studies. Here it is:

Excerpt from figure 2 of meta-analysis by Darrow et al.

You can see that most of the five studies find PTSD patients have shorter telomeres than controls by maybe half a standard deviation or less. Then there is one (Jergovic 2014) that apparently reports an almost five-sigma difference in telomere length between PTSD sufferers and controls. Five sigma! That’s the level of evidence that you get when you find the Higgs boson! It would mean that PTSD suffers had telomeres something like 3500 base pairs shorter than controls. It is simply inconceivable given everything we know about telomeres–given everything, indeed, we know about whole-organism biology, epidemiology and life. There really are not any five-sigma effects.

Of course, I looked it up, and the five-sigma effect is not one. This meta-analysis too had mis-recorded standard errors as standard deviations for this study. Correcting this, the forest plot should look like this:

Forest plot of the PTSD data from the meta-analysis by Darrow et al., with the ‘standard deviations’ corrected to standard errors in the study by Jergovic 2014.

Still an association overall, but the study by Jergovic 2014 is absolutely in line with the other four studies in finding the difference to be small. Overall, PTSD is no more strongly associated with telomere length than any other psychiatric disorder is. (To be clear, there are consistent cross-sectional associations between telomere length and psychatric disorders, though we have argued that the interpretation of these might not be what you think it is). What I find interesting is that no-one, author or peer-reviewer, looked at the forest plot and said, ‘Hmm…five sigma. That’s fairly unlikely. Maybe I need to look into it further’. It took me all of ten minutes to do this.

I don’t write this post to be smug. This was a major piece of work well done by great researchers. It probably took them many months of hard labour. I am completely sure that my own meta-analyses contain errors of this kind, probably at the same frequency, if not a higher one. I merely write to reflect the fact that, in science, the main battle is not against nature, but against our own epistemic limitations; and our main problem is not insufficient quantity of research, but insufficient quality control. We are hampered by many things: our confirmation biases, our acceptance of things we want to believe without really scrutinizing the evidence carefully enough (if the five-sigma had been in the other direction, you can be sure the researchers would have weeded it out), our desire to get the damned paper finished, the end of our funding, and the professional silos that we live in. And, as Ionnidis argued, vagaries in meta-analyses constitute a particular epistemic hazard, given the prestige and authority accorded to meta-analytic conclusions, sitting as they are supposed to do atop the hierarchy of evidence.

These two meta-analyses are of a relatively simple area, and cover the same 5 primary studies, and though they come reassuringly to the same qualitative conclusion, I still have no clear sense of how much shorter the telomeres of people with PTSD are than those of other people. The effect sizes found in the five primary studies as reported by Darrow et al. and by Li et al. are no better correlated than chance. So the two meta-analyses of the same five studies don’t even agree which study it was found the largest effect:

Two published meta-analyses of the same five studies show no better than chance agreement in their views of what the relative effect sizes were. Even allowing for the fact that they measure the effects on different scales, you might at least hope the rank order would be the same.

I hoped that meta-analysis would lift us above the epistemic haze, and perhaps it still will. But let’s not be too sanguine: as well as averaging out human error and researcher degrees of freedom, it is going to introduce a whole extra layer. What next? Meta-meta-analysis, of course. And after that…..?

Blue/Orange to play Durham

We are delighted to announce that Blue/Orange will play in Durham on Tuesday March 21st 2017, at 19:30pm, at the Empty Shop HQ in Framwellgate Bridge, DH1 4SJ. Tickets are available from here.

wes
Wes is in no condition to sell fruit

This simple space is going to be wonderful for the piece. I like productions where I can carry the set down with me on the train. And don’t forget, of course, performances in Newcastle on the Friday and Saturday of the same week, at Northern Stage.

And this little birdie got none……

We’ve just published a new paper on the effects of early-life adversity in starlings. We are particularly interested in how early adversity affects the shortening of telomeres. Telomeres are the protective DNA caps on the ends of our (and their) chromosomes, whose length is often used as a marker of biological age. We have found previously that nestlings who are smaller than their brood mates lose telomeres faster in the first few weeks of life. A bad start ages you.

However, we didn’t know what it is about being at a disadvantage in the brood that accelerates telomere loss. Is it that you don’t get so much to eat? Or is it the stress of having to struggle and beg more to hold your own when those around you are bigger and stronger? Or a bit of both? We decided to test this in a hand-rearing experiment. Here, we would play the parents and thus decide what each bird experienced in its formative days.

andrews-clareMany collaborators contributed to this study.
Clare Andrews did much of the hard work

We took four siblings from each of eight wild broods. From each family, one sibling was fed all it wanted nine times a day; a second was fed nine times a day but only 70% of what the first sibling got; the third was fed all it wanted nine times a day but had to do an additional 18 minutes a day of begging; and the fourth, who had it toughest of all, received 70% of what the third sibling did, and also did the extra begging.

Telomeres shortened rapidly in early life, but they shortened differentially according to early experience
Telomeres shortened rapidly in early life, but they shortened differentially according to early experience

The birds all survived – in fact these adversities are well within the natural range of what a wild starling might experience. Though they all fledged into normal adult birds, we found a difference in their telomeres (in red blood cells) when they were two months old. The more adversity we had given them, the greater the magnitude of their telomere shortening over their early life. Everyone’s telomeres get shorter in this period anyway, since a lot of cell division is going on, but the birds with more adversity showed more shortening. It seems that both the amount you get to eat, and the amount you have to struggle for it, both affect the pace of your cellular ageing, and do so additively (that is, if you have both adversities, it’s worse than having either one alone). This is important, since we know in both mammals and birds that conditions experienced in early life can affect survival. Cellular ageing might point to a mechanism by which this could occur.

DSC_0083starlings

We also made some other curious observations. The birds showed some differences in adult inflammation according to their developmental histories, but some combinations of adversity increased adult inflammation, whilst others reduced it. Birds that had had little food, but not had to beg for it, went on to become relatively obese as juveniles (which is interesting given the links between childhood stress and obesity in humans), but birds that had had little food and had to beg for it remained lean. Thus, it looks like early-life experience matters for your biology as an adult, but it matters in complex ways, not as simple as saying ‘more early-life adversity = more adult problems’.

The behavioural constellation of deprivation

We are used to the idea that the poor behave in a certain way- living for the day, devil may care, fatalistic, impulsive, enjoying life while they can- whilst the rich are more future-oriented, self-controlled and cautious. Just read the novels of Zola, for example, for vivid descriptions the appeal of present consumption over savings amongst those with the poorest lot in society.

zola

There’s actually a lot of evidence that, statistically, this is more than just a myth. People of lower socioeconomic status-in Britain for example-do tend to discount the future more heavily, are less health-conscious and future oriented. This can lead to a particular discourse about poverty: that people’s poverty is the consequence of their impulsive behaviours, and hence that poverty is in some sense the poor’s fault or failings that leads to their being poor (there would be no other good reason to be poor, right?).

In a ngpew paper coming out in the journal Behavioral and Brain Sciences, Gillian presents a different analysis of this phenomenon. What if the relative present-orientation that often goes with poverty were not a failing or weakness (or some kind of primordial character trait), but a sensible response to certain kinds of structural conditions?

Imagine a world where you felt that regardless of what efforts you made, you would be likely to be killed or lose everything at a relatively young age. What would you do? Would it make you all the more careful to eat your kale, have your testicular cancer screening and often check the pressure in your car tyres? Probably not. You would probably quite sensibly conclude that there was not much payoff to doing those things since something else would probably get you long before the benefits of those tedious efforts could be realised. You’d try to enjoy the life you could have while you had it. Thus, your present-orientedness, would be an appropriate response to the conditions under which you had to live.

That, in essence, is Gillian’s argument. People of lower socioeconomic position in contemporary societies tend to be exposed to worlds where they are in less of a position to realise the returns on future-oriented investments, because more uncontrollable bad shocks happen to them than happen to the rich (and, with growing inequality, a more conditional benefits system, and increased economic precariousness, this is tending to become even more true). If you accept that this is true, then much of their average behaviour, ex hypothesi, makes sense. It’s not a moral failing; you would respond that way too.

Of course, it’s not quite that simple, as a perusal of the paper will reveal. One of the big themes Gillian is interested in in the paper is the idea that the consequences of poverty become progressively embedded via feedback processes. If you start out down one lifestyle path – perhaps for a small but comprehensible reason – the alternative track becomes further and further away and the chance of reaching it, or the point of trying, less and less. This kind of embedding can even become transgenerational, as the behavioural strategies of one generation determine the starting input given to the next. But the big take-home message of the paper is that structural disadvantage is in the explanatory driving seat for the behaviour of the poor. The kinds of interventions, for example for health inequalities, that are of the greatest importance are those that address the structural disadvantages that make poor people likely (whatever they do) to have positive futures they can control and rely on.

bcd1

Behavioral and Brain Sciences is a peer commentary journal, so we are bating our breaths to see what colleagues in the social as well as biological sciences make of it all. One of the issues in this area is that so many disciplines have something to say about social inequality that it is easy to end up (a) relabelling ideas that in fact already exist in other disciplines; or (b) having ideas that are actually a bit different from the received ideas, but are mis-recognised as being something rather different from what you intend them to be. Let’s hope we have steered between that particular Scylla and Charibdis.

 

 

Hitting the Wall & Blue/Orange

 

longimage_hittingthewallI am very excited about our imminent production of Matthew Warburton’s Hitting the Wall at Northern Stage on November 30th.

In 2012, Wayne Soutter, a middle-aged father of two, attempted to swim the as-yet unconquered sea-channel between the Mull of Kintyre and Ireland. Hitting The Wall is a theatrical recreation of that extraordinary endeavour. Cold seas. Strong winds. Treacherous tides and 50 foot jellyfish. What could possibly go right?

Based on blogs and interviews with Wayne and Paul (the boat captain on the attempt), Hitting The Wall asks why we choose to do things that might better be left undone.

It’s going to be fun evening, with a chance to discuss the mad endeavour (actually, both mad endeavours, the swim and play about the swim) with the creative team afterwards.

You can buy tickets from here.

I am also equally ecstatic to announce that Straw Bear’s next production after that will be Joe Penhall’s extraordinary Blue/Orange, on March 24th and 25th 2017. Blue/Orange is about many things, most especially schizophrenia, race and Welsh rarebit, and was described (when it originally appeared in 2000) as the finest new play in the English language for a generation. We are presenting it in association with Brain Awareness Week. The cast has been finalised. More updates soon.

When methods meet

x160915165623_57372-jpg-pagespeed-ic-ahw2nzeu4iThe Scottish Graduate School of Social Science has made some interesting short films about the different methods available to social scientists, and in particular, whether they can fruitfully be brought together. In one of the films, the ethnographer Sam Hillyard and I discussed classic ethnography and experiments; can they be brought together, how are they different, and in what ways are they alike?

You can access the film and an associated worksheet here.