Month: March 2024

A guest blog in which Gillian Pepper states the obvious…..

Some time ago now, I was chatting with Daniel over lunch. I told him that Richard Brown and I were continuing to find evidence in support of a theoretical model that Daniel published over a decade ago. Daniel surprised me with his response. He declared that the conclusions of his model (which I will explain in moment) were so obvious that it would be surprising if they weren’t true. He had a point. And yet, we continue to act as if the obvious weren’t obvious. Perhaps, Daniel and I agreed, our conclusions would need to be repeated numerous times and to many audiences before they can perforate collective consciousness. As a starting point, Daniel invited me to write this guest blog.

The model of the “obvious”

Though Daniel’s original model contains various details and assumptions, the key points are as follows:

1. We are all exposed to health risks which, no matter what we do, will reduce our life expectancies. That is, there are risks beyond our behavioural control. For example, without refusing to ever leave our homes, we could never entirely eliminate our risk of death due to transport accidents. Daniel originally referred to this as extrinsic mortality risk, borrowing a construct from evolutionary biological models of senescence. We now call it uncontrollable mortality risk.
2. Some people are exposed to greater overall risk than others, and some are less able to mitigate the risks they face. That is, there are inequalities in exposure to risk. Depending upon where in the world you happen to live, and what resources you have available to improve your safety, there are myriad uncontrollable risks that might affect you. If you’re unlucky, war, violence, natural disasters, or extremes of weather might be hazards you face on a regular basis. Or perhaps the risks you face might be less obvious issues, such as mould and damp in your home, a polluted neighbourhood, or flammable cladding on your building. Whilst these issues may seem controllable to a relatively affluent person, they can still be classified as uncontrollable for those who can’t afford to move to a better neighbourhood, or to make the necessary repairs to their housing.
3. Uncontrollable risks reduce the future benefits of healthy behaviour. If there’s a non-zero chance that we will be struck down by an uncontrollable force before reaching an age at which the consequences of our lifestyle choices will be felt, then the temptation to indulge in short-term rewarding but long-term damaging behaviours, such as alcohol consumption will be greater. Especially when there is some benefit of that indulgence in the present (e.g. improved social bonding).
4. There is also a trade-off: time, money, and effort spent on health cannot be spent on other things that matter to us. Daniel’s model examines varying strengths of trade-off but, in general, the idea is that efforts spent on taking care of our health conflict to some extent with other things that might be important to us. Anyone who has experienced sleep deprivation due to caring responsibilities or eaten unhealthy convenience food due to time pressures at work will readily understand such trade-offs.
5. Consequently, exposure to uncontrollable risk should reduce our motivation towards healthy behaviour because it would mean investing efforts in health, instead of other priorities when, regardless of our efforts we might not live to see the long-term payoffs of taking better care of ourselves. This, I believe, is an unconscious driving force behind health motivation. One of a number of reasons (there will, of course, be other drivers too) that it can feel so difficult to do those things which we know would in some sense be better for our health.
6. Finally, the model suggests there will be a compound effect of extrinsic risk and health behaviour. An important implication of this is that people who, by no fault of their own, can do little to control the risks they face, will be less motivated to take care of their health (mitigate the risks they can control) than those of us who are lucky enough to feel safe and in control of our lives. And this will make the gulf in their achieved life expectancy even wider than it would have been for structural reasons. Social disparities in health behaviour can thus be seen as a downstream consequence of structural inequalities, rather than whim or ignorance, as some might assume.

To summarise the general idea: if you believed that, despite best efforts, you might die young due to war or natural disaster, would you worry much about whether you were eating enough fruits and vegetables? Probably not. And that was Daniel’s point. It would be rather surprising if people living in environments laden with threat were keen to quit smoking and forgo junk food. Nonetheless, we’ve dedicated a fair bit of time to testing this model.

We first tested the model by devising a measure of perceived uncontrollable mortality risk and assessing its relationship with self-reported health behaviour. When that study uncovered surprisingly large associations between perceived uncontrollable risk and health behaviour, we sought evidence of a causal relationship. We ran experiments designed to alter people’s levels of perceived control and measure their subsequent food choices. These found that people who were primed to feel that their personal risk levels were largely controllable were more likely to choose fruit than chocolate as a reward for taking part in the study. Richard Brown and I collected data during the COVID-19 pandemic to assess whether perceptions of uncontrollable risk had increased, and whether this was related to health behaviours in the UK (relatedly, we worked with Calvin Isch and colleagues to look at perceptions of uncontrollable risk in the USA). We found that perceived uncontrollable mortality risk had increased due to the pandemic and that it was associated with greater odds of smoking and lower odds of meeting Government guidelines on diet and exercise. More recently, Richard and I have published a replication and mini meta-analysis on the topic.

So, why all this effort to look for an association which would be puzzling if not present? Well, the answer is that the idea has some important implications. One of these implications is something I like to call the double dividend of safety.

The double dividend of safety

The idea of the double dividend of safety is simply that, if we make people safer by reducing those risks which they can’t avoid for themselves, we can expect that they will become more motivated to take care of their own health. So, we get the primary benefit of the initial improvement in safety, and the additional, secondary benefit of improved health from better health behaviour. That’s two benefits. A double dividend. If you think you’ve heard of the double dividend concept before, it may well be because you’ve encountered it in the context of environmental taxes. In this context, “double dividend” refers to the idea that environmental taxes should not only reduce pollution (the first dividend), but also reduce overall tax system costs if the revenue generated is used to displace other taxes that slow economic growth (the second dividend).

Understanding the double dividend of safety (rather than environmental tax) is important for numerous reasons. Among them, the fact that public health goals are often approached in silos. Behaviour-change programmes tend to operate in isolation, with practitioners rarely able to address the wider problems affecting those whom they seek to serve. This is not news, of course. Healthcare leaders have pointed out the need to break down this siloed approach. However, the double dividend of safety gives us another reason to call for joined-up thinking.

The concept could also be used to “sell” safety. You might think this unnecessary. Isn’t the importance of safety another one of those things that should be blindingly obvious? However, in a recent conversation with a Campaigns Manager at a global safety charity, I was surprised to learn that it can be difficult to persuade those in power that safety is important. “Safety isn’t sexy”, he said. This came as a surprise to me, but perhaps it shouldn’t have. Those who have the power to make change for others, on average, probably don’t have much experience of being unsafe. As Daniel mentioned in a recent blog on inequality, when the ruling classes have so little contact with what the majority experience it becomes difficult for them to make decisions that work for the public good. Yet, it remains true that public health funds are spent on giving the general public information and tools (usually in the form of websites and apps) in attempts to improve health behaviour. For example, the UK Government’s Better Health Campaign, which purportedly cost £10m. Such efforts make it clear that there is a desire to improve health behaviour.

What if, we were to instead shift our focus to making people safer? The double dividend of safety suggests that they would automatically be more motivated to take care of their health: a double win. Whilst this might initially seem like the harder (and probably more expensive) path to take, I’m willing to bet that it would also be the more gainful one in the long run.

Lately, I’ve been coming across a lot of proposed study designs that were like mansions. There I was, appreciating the well proportioned main research questions and the generosity of the outcome measures, when a little door got opened up in the panelling, and it became evident there were whole wings beyond the part where I came in; wings with other measures, sometimes in wildly different styles, and objectives of their own, and additional treatments, and turrets and gargoyles and intervening variables. The wings were somehow connected to the hall I had entered by, making for one big, rambling complex. Yet, they were somehow separable, in that you could live for years in one without needing to go into the others. Indeed, you could imagine them being parcelled off into entirely separate flats.

Your study should not be like a mansion. It should be more like a single room than a mansion. Your study should follow the principles of Bauhaus or Japanese minimalism. Clutter should be removed until rock bottom simplicity has been achieved; then the design should be decluttered all over again. The ambition should be repeatedly refined and made narrower. There should ideally be a single objective. Outcomes should be measured with the best available measure, and no others.  Control variables should be designed out of existence where possible.  Mediators and moderators – do you need them? Why? You haven’t answered the first question yet. The analysis strategy should have the aching simplicity of Arvo Part’s Spiegel Im Spiegel. Anything that can be put off to another future study should be, leaving this one as clear and austere as humanly possible.

I am aware that I always made my studies too complicated in the past, and I see the desire to do so almost without exception in the younger researchers I work with. I am wondering where the desire to over-complicate things comes from.

Part of it, I am sure, comes from the feeling that there is a potential upside to having more measures, and no cost. You’ve got the people there anyway, why not give them that extra personality questionnaire? Or stick in that extra measure of time perspective, or locus of control, or intolerance of uncertainty? The extra burden on them is small; and surely, if you have a superset of the things you first thought of, then you can find out all the things you first thought of, and maybe some more things as well.

We were taught to think this way by the twin miracles of multiple regression and the factorial experimental design. The first miracle meant, we thought, that we could put more predictors in our statistical model without undermining our ability to make estimates of the effects of the ones we already have. In fact, things might even get better. Our r2 value would only go up with more ‘control’ variables, and our estimates would become more precise because we had soaked up more of the extraneous variance.

The second miracle meant, in an experimental study, that we could cross-factor an additional treatment with the first, without affecting our ability to see the effects of the existing one. Let’s do the thing we planned, but have half the participants do it in an inflatable paddling pool, or wearing noise-cancelling headsets. Our ability to detect the original effect will still be there when you average across this treatment. And we will know about the effects on our outcome of being in a paddling pool, to boot!

The truth is, though, that nothing comes for free. Cross-factoring another experimental treatment can make it difficult to say anything very generalizable about the effects of the original treatment. We wanted to know whether, in the world, caffeine improves memory performance, and we discover that whether it helps or hinders depends on whether you are standing in a paddling pool or not. But, in life, in the real world conditions where one might use caffeine to boost memory, one has not, as a rule, been asked to stand in a paddling pool. What then is the take home message?

As for the miracle of multiple regression, this is even more problematic. The idea that including some extra variable X2 in your regression leaves you still able to estimate the effects of X1 on Y in an unbiased way holds only in a subset of the possible cases, namely when X2 has an effect on Y but is not affected by X1, Y or any of their unmeasured consequences.  It is very hard to be sure that these conditions apply to your study.  This fact is not widely appreciated, with the consequence that whole swathes of social and behavioural sciences include far too many variables in their regressions, including many that they should not (see here and here; I am looking at you sociology, and you, epidemiology). Your thing does not become more true if you have controlled for more other things; it usually becomes more obscure. In fact, if you see it in a complex analysis with lots of additional covariates (especially if you see it only then), this increases the chances that it is in fact a statistical artifact (here for a case study).

Another exacerbating factor is psychology’s obsession with identifying mediators. It’s all very well to show how to change some outcome, but what’s the mechanism by which your intervention works? Does it work by changing self-esteem, or locus of control, or stress? Again, we were taught we could answer mechanism questions at no cost to the integrity of our study by throwing in some potential mediating variables, and running a path analysis (where you run your regression model first without and then with the inclusion of the potential mediator, and compare results). But, again, with the exception of some special cases, doing this is bad. Not only  does adding a mediator often lead to overestimation of the degree of mediation, it actually imperils your estimation of the thing you cared about in the first place, the average causal effect. There is a whole slew of papers on this topic (here, here and here), and they all come to the same conclusions. Don’t clutter your study with mediators in the first instance; they will probably confuse the picture. Identify your causal effect properly and simply. Answering further questions about mechanism will be hard and will probably require new studies – maybe whole careers – designated for just that. (Similar comments apply to moderators.)

What underlies the impulse to over-complicate, at root, is, fear of being found insufficient. If I have only one predictor/manipulation and one outcome, how will I be judged? Can I still get published? Does if look tooo simple? What if the result is null? This is what I hate most about science’s artificial-scarcity-based, ‘significance’-biased, career-credentialing publication system. People feel they need a publishable unit, in a ‘good’ journal, which means they have to have a shiny result. They feel like they can increase their chances of getting one by putting more things in the study. This imperative trumps actual epistemic virtue.

So, complexity creeps in as a kind of bet-hedging against insecurity. Let’s add in that explicit measure of the outcome variable, as well as the implicit one. In fact, there are a couple of different explicit scales available: let’s have them both! That gives us lots of possibilities: the explicit measures might both work, but not the implicit one; or one of the explicit measures might look better than the other. There might even be an interaction: the treatment might affect the implicit measure  in participants who score low on the explicit measures – wouldn’t that be cool? (Answer: No). Even if the intervention does not work we might get a different paper validating the different available measures against one another. But the problem is that you can’t make a study which is at the same time an excellent validation study of some different measures of a construct, and also a test of a causal theory in that domain. It looks like a capacious mansion, but it’s just a draughty old house none of whose wings is really suitable to live in.

If you put in more objectives, more measures, and more possible statistical models, you are more likely to get a statistically significant result, by hook or by crook. This does not make the study better. We are drowning in statistically significant results: every paper in psychology (and there are a lot of papers) contains many of them. It’s not clear what they all mean, given the amount of theoretical wiggle room and multiple testing that went into their construction. Their profusion leads to a chaotic overfitting of the world with rococo ‘theories’ whose epistemic lessons are unclear.  We need fewer new significant results, and more simple and clear answers (even descriptive ones) to more straightforward questions. Your study could be the first step.

Perhaps the main unappreciated virtue of simpler studies, though, is that they make the researcher’s life more pleasant and manageable. (Relatedly, an often overlooked benefit of open science is that it makes doing science so much more enjoyable for the researcher.) When you double the number of variables in a study, you increase the possible analyses you might conceivably run by at least a factor of eight, and perhaps more. Don’t tell me you will have the strength of character to not run them all, or that, having discovered one of those analyses gets a cute little significance star, you will not fret about how to reframe the study around it. You will spend months trying out all the different analyses and not be able to make your mind up. This will be stressful. You will dither between the many possible framings of the study you could now write. Your partner will forget what you look like. Your friends’ children will no longer be toddlers and will have PhDs and children of their own. Under socialism, data analysis will be simpler than seems even imaginable under the existing forces and relations of production. Until then, consider voluntary downsizing of your mansion.

Note. Some studies have a lot of measures by design. I am talking about ‘general purpose’ panel and cohort studies like NHANES, Understanding Society, the SOEP, and the UK National Child Development Study. Rather than being designed to answer a specific question, these were envisaged as a resource for a whole family of questions, and their datasets are used by many different researchers. They have thousands of variables. They have been brilliant resources for the human sciences. On the other hand, using them is full of epistemic hazard. Given the profusion of variables and possible analyses, and the large sample sizes, you have to think about what null-hypothesis significance testing could possibly mean, and maybe try a different approach. You should create a Ulyssean pact before you enter their territories, for example through pre-registering a limit set of analyses even though the data already exist, and pre-specifying smallest meaningful association strengths, rather than null hypotheses. Even in these studies, the designers are conscious of trying not to have too many alternate measures of the same thing. Still, it remains the case that a lot of what I say in this post does not really apply to the designers of those projects. Your study should not be like a mansion, unless it actually is a mansion.