Against Open Mindedness

Lots of people believe in psychic powers, but there has never been any convincing evidence for their existence.  Though there are many anecdotes attesting to their existence (below I will say something about why we ought not to be impressed by these stories), there has never been any genuine evidence in their favour. That is, until now. The Journal of Personality and Social Psychology, the most influential journal in social psychology, is about to publish a study that presents evidence for the existence of psychic phenomena.

The paper is by Cornell’s Daryl Bem. Essentially, Bem used standard methods in social psychology but backwards. For instance, he used a standard method for enhancing memory, but used it backwards. The standard (and commonsensical) method is to give people training on a word list. Then they are presented with a longer list of words, including the trained set. Finally, they are tested for recall of the longer list. Unsurprisingly, they are more likely to recall words they practiced than  those they did not. Running the method backwards, Bem gave his subjects a test of recall of a long list of words he presented them with, and only subsequently did he drill them on a subset of those words. The finding was that subjects recalled words on the training set better than those absent from it. In other words, their performance seems to have been enhanced by training they had not yet had. Similarly, Bem reversed the standard methodology using priming. Priming – by presenting a stimulus so briefly that it not consciously perceived – has an effect on behavior. It makes people quicker to respond to stimuli semantically related to the prime, for instance. Bem gave his subjects the prime after testing for facilitation of response. Once again, he found that the experimental manipulation (the manipulation which had not yet occurred) apparently had an effect on behavior.

What are we to make of this? Of course, we should take the results seriously. That is, though there is no plausible mechanism for the effects that Bem claims to have demonstrated (his discussion of quantum entanglement notwithstanding), we must remain open to the possibility that he has shown the need for us to identify such a mechanism. However, vindicating Bem’s results would require such a major upheaval in our scientific worldview, we ought to remain highly sceptical. We should not be open-minded; not, at least, if open-mindedness is (or entails) giving serious consideration to the possibility that what Bem has demonstrated is the existence of some kind of psychic phenomenon. Taking these results seriously should motivate us to find explanations of them that do not require such a revolution in thought. Only once we have failed, repeatedly, to find such explanations, should we move to being open-minded.

Science is a body of relatively consistent and mutually supporting claims. It is a web of findings and methods. They are empirically validated by their ability to generate predictions, not to mention their applications in technology. We should therefore be reluctant to introduce any revolutionary changes in our system of beliefs. The more central a belief is, the higher the standards we ought to apply: we need really convincing evidence to establish a claim that would ramify throughout our conceptual scheme. Some people might be worried by the apparent dogmatism of this view, thinking it mires science in the past and prevents exciting new breakthroughs. As the great philosopher of science Thomas Kuhn argued, just the opposite is true: we can only generate truly novel results – as opposed to mish-mash of vague conjectures – if we have very precise expectations concerning the results we ought to get. The commitment to Kuhn called a paradigm, which includes a commitment to an existing set of claims and a commitment to thinking that anomalous findings can be explained away  is, as he said, very effective at generating and ultimately at explaining anomalies. As Kuhn wrote, “novelty ordinarily emerges only for the man who, knowing with precision what he should expect, is able to recognize that something has gone wrong. Anomaly appears only against the background provided by the paradigm. The more precise and far-reaching that paradigm is, the more sensitive an indicator it provides of anomaly and hence of an occasion for paradigm change”.

We know a great deal about how the world works. We also know a great deal about how the human mind works, and this knowledge explains why people, even smart people, believe in the paranormal. We know that people are over-impressed by anecdote due to a raft of psychological biases inherent in the human mind: the confirmation bias, which leads us to seek confirming evidence for a theory rather than evidence against it, memory biases, which lead us to overvalue times in which predictions have turned out to be correct and ignore those in which they have failed, recency and salience biases, and so on. Given what we know about the human mind and about the world, we should not be overly disturbed by Bem’s results. We ought to confidently expect that attempts at replication will fail, and that plausible explanations for his work will emerge. Indeed, there has already been a failed attempt at replication.

Why did Bem get the results he did? I can only speculate. One possibility is that his findings are entirely by chance. This is unlikely, given the statistical tests he used, but the unlikely occurs now and again (remember, if the chances against someone doing something are a million to one, 60 people will do it in the United kingdom alone). Another possibility is that Bem’s paper is best read as a reductio of the methods he used: given that they generate results like these, we ought to use stricter tests for significance (some people have been arguing for a long time that the tests he used, standard though they are, are unreliable; they have taken this paper as further evidence). In the meantime, the only message I am taking away from this study is that we should always seek replication before we accept a scientific finding. Close minded? Yes, and proud of it.

  • Facebook
  • Twitter
  • Reddit

16 Responses to Against Open Mindedness

  • You’re right this paper deserves special scepticism; but the lesson I hope the field learns from this (admittedly provocative) paper is that everything deserves this level of scrutiny, including (and perhaps especially) the papers you agree with or depend on in your own work. We’ll see, I guess.

  • Neil Levy seems to be arguing for several cases of varying strength. His concluding recommendation, that “we should always seek replication before accepting a scientific finding”, is very weak indeed; anything weaker would violate standard scientific practice in any field. Then he provides more common-sense, with “we need really convincing evidence to establish a claim that would ramify throughout our conceptual scheme”. Then he gives a methodological defence of Kuhn’s over-arching paradigms, as the one way to identify real anomalies.

    All this is quite reasonable in its own terms, but there are the implications of his rhetoric and silences. He does not mention that Kuhn was trying to explain cases where the paradigm is overthrown in a Scientific Revolution; that is how real progress occurs, in his theory. It would seem that Bern’s experiment was carefully designed to test a crucial feature of the accepted paradigm, very much in the Kuhnian methodology. But the possibility of a Kuhnian scientific revolution in consciousness seems to be absent from Neil Levy’s perspective.

    One might invoke the spirit of the other great philosopher of science, Karl Popper, and ask Neil Levy whether there are any circumstances under which he would accept the reality of psychic phenomena. If so, then are still in the world of vigorous scientific debate, of the sort well characterised by Lakatos. If not, we are somewhere else.

  • Neil says:

    Jerome, you’re quite right in saying that these are views of different strength. I accept the stronger one and therefore also the weaker. Is there any evidence that would convince me of the existence of psychic powers? Yes, but extraordinary claims require extraordinary evidence. One quibble: at least at the time of the book from which I quoted, Kuhn seems to reject the idea that changes in paradigms constitute progress. For him progress occurs within a paradigm, not across paradigms.

  • Simon Rippon says:

    Neil – thanks for linking to Wegenmakers et. al. in your post – but their first point shows that the methods Bem used were not at all “standard” as you describe them – or that if they have become standard in some areas of social psychology, then they certainly ought not to be!

    This is because Bem confuses exploratory study (where you construct the hypothesis after examining the data) and confirmatory study (where you construct the hypothesis before running the experiment). To see the difference, consider the hypothesis that if I look out the window, the next five cars I see will be red. That’s ex ante very unlikely. If I run an experiment to test this hypothesis, it would be very unlikely to produce a positive result.
    Here’s an alternative, exploratory experiment: Record the characteristics of the next five cars I see, and see after the fact whether they all have something in common (e.g. all red, or all white, or all Ford, etc.) and only then construct my hypothesis. If they have a distinctive characteristic in common such as their colour, then my “hypothesis” will be that the five cars have that characteristic in common. The particular characteristic they share in common (e.g. that they are all white) would have had a very low ex ante probability of occurring by chance. But that’s not the same as the probability of my ex post being able to find some hypothesis or other that is now “confirmed” by the data I’ve collected. The ex ante probability of five randomly selected cars sharing some some distinctive feature or other in common is much higher than that of their sharing the colour red in common. But Bem ignores this point, and runs his statistical significance calculations based on the lower probability, not the higher one he should be using!

    I highly recommend Ben Goldacre’s book _Bad Science_ to general readers interested in getting a very non-technical overview of basic statistical methods and the importance of confusions of this kind in generating spurious (pseudo)scientific “results”.

  • John Carter says:

    oh come on…psychic powers ? Next step will be Jedi Mind Tricks… No really, I just hate this misinformation that’s going on in the world. People should stop believing everything they see…especially those “studies show that…” with no specific evidence

  • Neil says:

    Simon, it is a complete no no in all areas of science not to frame a hypothesis prior to testing. Exploratory tests are fine for framing hypotheses, but then there must be independent tests which hold them fixed. I am being charitable to Bem: there is no evidence that he did not frame his hypotheses prior to (at least most) of the experiments. And it is increasingly common to supply reviewers with supplementary materials, so he may have asserted explicitly that he did not use an exploratory strategy. I agree that if my charity is not warranted, then we need not look any further for explanations of these results.

  • Bem’s advocacy of what Wagenmakers call the Bem Exploration Method ‘write…the article that makes the most sense now that you have seen the results’ (Bem, 2003, pp. 171-172) could be defended as inference to best explanation, so I’m not sure that that piece of methodology is as intrinsically flawed as Neil is. Rather, we might question whether abductive inference has been properly used. It’s certainly an ingenious experiment and an astonishing result, so astonishing that Hume’s reasoning clearly applies: prophecies are miracles and *by definition* miracles are far less likely than anything in the common course of our experience; error and self-deceit are in the common course of our experience; so error or self-deceit far more likely and therefore better explain the result than pre-cognition.

  • I cannot resist re-telling a joke I heard long ago, delivered by (very surprisingly) a distinguished Soviet historian of science. It seems that a rationalist was out walking with his good friend a Jesuit, and they were arguing about miracles. Becoming exasperated, the Jesuit struck a large rock with his walking-stick, and water came gushing out. “Isn’t that a miracle?” he exclaimed. The rationalist replied, “An interesting accident…”. “An accident!” shouted the Jesuit, and struck the rock again, whereupon the water came gushing forth. “An interesting coincidence,” commented the rationalist. “Coincidence!” screamed the Jesuit, “I’ll show you!”. Whereupon he struck the rock again and again, whap, whap, whap, and every time the water gushed out. “Ah, now I understand”, said the rationalist, “Every time you strike the rock the water comes out. So here we have a case of constant conjunction of cause and effect, and by Hume’s criterion that is a scientific generalisation, and hence it is no miracle!”

    Of course I agree with everything Nick and Neil have said! Just having a bit of fun. In case the weekend gets boring, there is the other story in the current New Scientist, about the experiment ‘proving’ that people can see auras.

  • Peter Wicks says:

    All this reminds me of debates we’ve been having about homeopathy on this blog. There also, we have a hypothesis (that homeopathic remedies are effective for reasons other than placebo or behavioural ones) that appears to be fundamentally incompatible with current scientific theory. In that case a much stronger version of “closed-mindedness” than Neil’s was suggested, namely that experiments to test the “impossible” hypothesis should not be conducted in the first place, and if they are they should not be considered.

    Can such position be defended, and on what basis should we decide whether or not it can? From a utilitarian perspective it would seem that it can, to some extent. Not only do extraordinary claims require extraordinary evidence to be believed; they require pretty good evidence to be considered at all. The more so given that attention, rather than information, is the scarcer commodity today.

  • Peter Wicks’s principle reminds me of what I have called the ‘Semmelweis Syndrome’. This is a statement by the head of his hospital unit:

    “How dare you accuse our medical students of killing women in childbirth! I would never entertain such a monstrous accusation without full experimental proof, and I would never permit the experimental investigation of such a monstrous accusation”.

    It would also justify those legendary scientists who refused to look through Galileo’s telescope on the grounds that his claims would destabilise science and the whole religious-social order.

    Perhaps we should have a competition on who can most narrowly define the limits of permissible research, while leaving in something other than the utterly banal.

  • Sorry, I forgot this one.

    Since the climate change skeptics are putting the whole future of mankind at risk, all philosophers should agree that pragmaticallyl ‘the science is settled’ and ‘the debate is over’, and support a ban on any further questioning of Carbon-Based Anthropogenic Global Warming.

  • Falafulu Fisi says:

    I can’t believe that even this discussion is going on. Psi is nonsense because it violates causality and that’s physics. You cannot fuse the future together with the past & present or otherwise, material entity can be everywhere at once and exist for all time. Simply put it folks, that a person is not confined to a specific point in space & time, but he/she is smeared into the future/past/present in all points in space and all time. This violates causality, pure and simple. And don’t try to invoke quantum mechanics here, because it’s got nothing to do with it. Even the interpretation of quantum mechanics is wrong philosophically.

    Forget the method being used by Dr Bem whether it was correct or not and blah, blah, blah. One can used the same statistical hypothesis testing to see if God exist or not and the result could probably say that there is a statistical significance that God does exist? Statistical hypothesis testing is not physics. The advancement of modern physics over the last 100 years wasn’t aided by statistical hypothesis testing, such as prediction of the existence of quarks and other subatomic particles. One starts from first principles and then derive a theory that guides that person to search for the unobservable predicted by that theory. Example, quarks was predicted by particle physics standard model even before the first indirect experimental evidence confirmed that they do exist. Psi on the other hand was just dreamed up. Not based on any physical theory at all. It is similar to the claim of homeopathy.

    Physics Nobel laureate, late Richard Feynman had addressed this type of organized research which then become science. He labeled those type as Cargo Cult Science. This is exactly what Dr Bem was doing. It appeared scientific (exactly as posters above are saying) but beneath it is nothing more than nonsense. Move on folks, there is nothing here to be discussed about psi. It is nonsense and therefore, don’t waste time is trying to scrutinize the statistics whether it was appropriate or not. It is completely bollo*ks.

  • Falafulu Fisi says:

    I had some exchange with Andrew Wilson, who is the first poster/commenter on this thread at the very top at his blog, but it seems that he has not allowed further posts to a same blog thread on the work of Dr. Daryl Bem. That blog article is shown below:

    Andrew Wilson said…
    “Actually, for all it’s sins, the psi study is pretty scientific.”

    My reply to Andrew…

    “Andrew, you’re starting to talk like a psuedo-science advocate as Feynman addressed in his Caltech speech in 1974 (“Cargo Cult Science” –

    Now, I am amazed that you call yourself a scientist with that sort of comment you made above. Where do you draw the line of what is being regarded as science and what is not science? Psi is not scientific. It is not grounded in any physical theory or anything like that? Which branch of physics that psi theory falls in? None whatsoever. It is arbitrary and has no basis in reality (either observational or theoretical framework). You must throw out arbitrary assertions or otherwise, one would have to test an infinite set of hypotheses that one can just dreamed up.

    We know in such scenario, there is no limit to what can be tested and hypothesised. That’s why we built hadron colliders and things like that because the theory guides us to search for such possibility since theory says that such phenomena in nature exists, i.e., the equations and their solutions tell us some hints about those predicted unobservables. Physical laws are very well hidden. We cannot run an infinite sets of experiments to detect hadrons just because we think such particle or force x, y, z do exist. There must be some foundations to start with.

    Psi research has no foundations at all. It is similar to homeopathy. No foundation in reality to start with. They’re arbitrary and must be dismissed as metaphysics teaches us.
    It is a waste of time to prove psi. Metaphysics says, that the onus to prove are those that make the claim, i.e., Daryl Bem, but not us (the opponents or skeptics) as you stated. So, far Dr Bem hasn’t proved his claim. His statistical correlation is not proof. Psychologists don’t develop theory of nature or physical reality. You use statistical hypothesis testing, which is descriptive in nature and unreliable. The laws of the physical world were not invented via statistical hypothesis testing.

    It is time that you Andrew start saying that psi research is nonsense and non-scientific and stop pretending that is otherwise or trying to defend that it is scientific.”

  • Falafulu Fisi says:

    Oops, the correct link to Feynman’s talk on “Cargo Cult Science” is shown below:

  • For the record, and for what it’s worth, I haven’t disallowed further commenting; I’m always up for a good argument but it seems there was an error coming up when you tried to post. Sorry, no idea what the cause was.

    I’m not sure I want to invade this perfectly nice blog with this argument, but I’m not arguing psi is scientific. I just think Bem worked pretty hard to apply the scientific *method* to a controversial topic and that makes it good enough to put out there. Do I believe a word of it? No, it’s about psi which is ridiculous. But like it or not he got a bunch of statistically reliable results in line with his hypothesis and we need to show how he got them (my money’s on the Wagenmaker account; we psychologists have been doing stats badly for a long time, sadly). But establishing that counts as progress, and I’m basically ok with that.

  • Neil says:

    Wow, Falafulu, you’re even more close-minded than me! I once heard a statistician describe his profession as “the guardians of the scientific method”. I think that’s an exaggeration, but not by much. If we can establish a reliable and verifiable relationship between two entities – which is what the debate is about – then we simply have to adjust our ontology to capture it. We can’t allow the fact that we can’t explain the relationship to keep us from recognizing the relationship. The fact that psi is inexplicable given everything we know means we ought to demand various good evidence for its existence, and I am confident that the evidence will not be forthcoming. But it is perfectly scientific to investigate the relationship ahead of being able to explain it.


Subscribe Via Email