Measurement

I sat in a meeting, around five years ago, that changed my views about research entirely. Of course, I’m a creative, so my relationship with research has always been love-hate, but I’d always reluctantly assumed that data must be right. The meeting was a briefing for our biggest research company (one of the world’s biggest). We were discussing the questions we should ask people to obtain the information we wanted about their likely behaviour. Getting invited to meetings like this is one of the joys of being an in-house creative. It struck me very strongly that we were not just deciding the inputs but the outcomes. When I challenged this, the quite reasonable response was, “well, we need to define what we want to know if we want useful information”.

Way back in 1997, the anthropologist Dame Marilyn Strathern wrote a paper in which she summarised an obscure piece of economic theory written by Charles Goodhart into a more general rule: “When a measure becomes a target, it ceases to be a good measure”. This, unfairly I think, is named after the economist rather than after Dame Strathern, as Goodhart’s Law. 

It means, for instance, that if you have, say, a target to reduce child poverty, it then becomes impossible to measure child poverty without the target in mind. The target will define the way you measure and track your subject, even if there is no conscious attempt to influence or change the data. It “defines what we want to know”. It is the reason leading questions are not allowed in British law. “Can you walk ten steps?” asks a respondent to comply with a parameter, whereas “How far can you walk?” sets no expectation.

It may be objected that this problem with set targets might tend to undermine the science on which we depend to make decisions. In fact, it does not undermine science at all, but is strongly supported by it. In fact science can help us to understand when and how measurement might be useful. It is one of the key findings in Quantum physics that, as soon as you measure movement of a particle, the motion ceases to be measurable.

The reason for this surprising outcome in physics is that the subject of measurement is very small, and therefore difficult to see. The effect of small particles’ motion can be detected, but the motion of an individual particle is very difficult to trace. When it is, it is radically different to the general motion which was detected, and that motion is said to ‘collapse’. Without going into the complexities of this, the conclusion is well established and accepted: the act of measurement is not objective but is an intervention, in very much the same way as the setting of a target is an intervention. The reason it comes about is that physicists are attempting to deal with particles so small they cannot easily see them. Unlike other scientists, Quantum physicists have found it necessary to investigate a field where information cannot be demonstrated. The experiment which detects the gernal movement of particles is therefore based on understanding the probability of their whereabouts by observing the behaviour of a large number. When specificity is attempted the effects of the laws of probability naturally disappear. 

When I first read about this, I was strongly reminded of my impression at the research briefing. Our experience when investigating the unknown is perhaps unsurprisingly exactly the same as that of the scientists. Our researchers attempt to collect data to give guidance on what people will probably do in the future, and as soon as we attempt to measure this to be certain we create a moment ‘now’ which displaces that future action to the past. This is because probable action collapses as soon as we ask the question about it, however we frame that question. In fact, we can say that the uselessness of our activity is supported by science. 

As a footnote, we might observe that, in law, this does not matter. The law is only concerned with the past. No judge would presume to make a decision based on an assessment of possible future capacity. It would seem absurd.

Where does this leave us? I think it must frame our expectations, just as our expectations frame our research. Data can be good at giving us a picture of the past. ‘Real time’ data inches close to the present. It clearly does not, however, make a leap to the future, which is why overuse of real time intervention is destructive of trust. It is a transparently crude attempt to manipulate a decision about what to do next which people naturally resist. A leap to the future is possible, however. To do it, you need creativity and imagination, because you need to understand and empathise. It is a scientific truth that while data might inform this, only human beings can do it in a way human beings like.