Publish it all, I say.

Tom at Idiolect ruminates on this piece of research on awareness of being stared at. We’re talking ‘6th Sense’ here, people. Hairs bristling on the back of necks. Mind to mind communication. Bruce Willis, meandering with his best haunted expression (do you see what I did there? Perversely fitting, no?)

He’s generally unhappy about the fact that this was published, given that the authors do not present any possible ways of connecting the findings with our understanding of how reality works. I’m more optimistic, and think that, absent outright falsification or methodological error, it’s a good thing it was published, small effect or no.

It calls to mind the Maharishi effect, which I was vaguely aware of for some time but got a thorough schooling in through Robert P Abelson's excellent “Statistics as Rhetorical Argument”. This is my favorite book I’ve read this year, and I’m heading for a re-reading sooner rather than later. Abelson recounts this remarkable finding, wherein
The number of people performing transcendental meditation correlates with reduction in car accidents, lower crime rates, and incidence of fire. He takes up this example a number of times, in which the major study₁ got complex but significant results and appeared to follow its own methodology to the letter.

Of course, a result like this is going to be controversial, and rightly so: there are huge implications for science as well as national policies (mandatory meditation anyone?), so its damn important that this be right. As the research was published, every Tom Dick and Hermione with an inquiring mind was free to pore over what was done, and see if anything was wrong.

Sure enough, an explanation was found, and you will be unsurprised to hear it did not require the rejection of materialism, nor a consequent global rise in guru-republics. The problem lay with the non-random distribution of number of meditators across weeks used to match against the meditation periods. Higher numbers of meditators were available at holidays, weekends and the like and these calendar effects were trackable and produced an alternative explanation for the correlations.

It’s a good thing the Maharishi effect research was published, even though it turned out to be wrong. Why? Because it wasn’t obviously wrong: the confound they uncovered was not considered by the journal, and so presumably was not standard practice in the field of (hmm- psychological epidemiology?). It was a problem that had not been clearly classified as a problem.

When deeply unusual and, lets say it, unwelcome (from the point of view of the scientific community) research comes along, it galvanizes people to focus ever-harder on methodology, in a mass-game of spot the confound. Once the confound is spotted, it raises the quality of future published work, as it shouldn’t get by without recognizing and overcoming the issue. It has a hint of working with paradoxes in philosophy, but with an overtly empirical bent: “This can’t possibly mean what it says it means – so which of the inferences [or methodological steps] are unsound?”

Of course, if a confound really can't be spotted, and the result proves to be replicable, then this is an equally important reason to consider it. It might suggest that we need to be prepared for a paradigm shift in that scientific domain, as current theory simply cannot account for the genuine empirical findings. Or, perhaps more worryingly for the scientist, we might need to consider that our tools are not right for the job: either current scientific method or the human capacity to undertake and appraise it simply can't cope with everything we throw at it. Both would be radical results, but that doesn’t make them taboo. I don’t know of anything out there that seems likely to be taking us down that road though.

1 Orme-Johnson, D.W., Alexander, C.N., Davies, J.L., Chandler, H.M. & Larimore, W.E. (1988). International peace project in the middle east: The effects of Maharishi technology of the unified field. Journal of Conflict Resolution, 32, 776-812

UPDATE: More here in the comments thread, and a followup post further up the page.