As many readers of this blog may know, in the last few years a “replication crisis” has caused intense soul searching in psychology – and particularly in social psychology. This crisis was sparked when several widely cited findings in psychology subsequently failed to replicate when tested by independent researchers (for some background see Earp and Trafimow’s paper here). This is – of course – a substantial problem because it severely limits the confidence we can have in psychological findings. For social psychological research this problem may be even graver, given that many of the topics studied – e.g. attitudes, prejudice, political voting – have direct implications for policy making. If we cannot have confidence in our findings, social psychology is in very hot water indeed.
To take an example from my own research field of social/moral psychology, in a highly influential study published in Science, Zhong and Liljenquist (2006) reported evidence of a “Macbeth Effect” whereby a threat to people’s moral purity leads them to seek, literally, to cleanse themselves. Seeking to expand upon these findings and explore whether they would be observed in different domains, my colleagues Brian Earp, Elizabeth Madva, and Kiley Hamlin attempted to replicate and extend this work. To their surprise, they were unable to replicate the original results. They then turned to me, and together we conducted a series of direct replications of Study 2 from Z&L’s seminal report. We used Z&L’s original materials and methods, investigated samples that were more representative of the general population, investigated samples from different countries and cultures, and substantially increased the power of our statistical tests. Despite our multiple good-faith efforts, however, we were unable to detect a “Macbeth Effect” in any of our experiments. We then tried to publish these findings, facing much difficulty in the process. Put simply, replication efforts are often not viewed favorably in the publication process. Eventually, however, we were able to publish our findings in the journal Basic and Applied Social Psychology (see here for an open access version of the paper).
This experience caused Brian Earp and I to reflect upon the replication crisis in psychology. In a new opinion piece, we suggest that this “crisis” can be interpreted as a disciplinary social dilemma, with the problem facing early-career researchers being especially acute. Social dilemmas – situations in which collective interests are at odds with private interests – are an enduring feature of the modern world, and have two fundamental characteristics: first, that each individual receives a higher payoff for defecting from what is in the collective interest (e.g., using all of the available resources for one’s own advantage) than for cooperating, regardless of what other individuals do; and second, that all individuals are better off if they all cooperate than if they all defect. We interpret the crisis through the lens of a social dilemma, arguing that while it is in everyone’s interest that high-quality, direct replications of key studies in the field are conducted (so that it is possible for the scientific community to know what degree of confidence to place in previous findings from the literature), it is not typically in any particular researcher’s interest to spend her time conducting such replications. This is for a number of reasons: (1) such replications may be time-consuming; (2) they are likely to take energy and resources directly away from other projects that reflect one’s own original thinking; (3) they are generally harder to publish; (4) even if they are published, they are likely to be seen as “bricklaying” exercises, rather than as major contributions to the field; (5) they (accordingly) bring less recognition and reward, including grant money, to their authors—and so on.
Brian and I propose a new a structural solution to this collective problem:
“as a condition of receiving their PhD from any accredited institution, graduate students in psychology should be required to conduct, write up, and submit for publication a high-quality replication attempt of at least one key finding from the literature, focusing on the area of their doctoral research.” (p.1).
As we note in the paper, before any formal implementation at a university level, the basic soundness of the idea would have to have been established—which means subjecting it to critical scrutiny. And so, in both the paper and on this blog, we present our idea in a public forum, and invite discussion and constructive feedback from our colleagues. Could PhD students resolve the tragedy of the academic commons?
Everett, J.A.C.**, & Earp, B.D.** (2015). A Tragedy of the (Academic) Commons: Interpreting the Replication Crisis in Psychology as a Social Dilemma for Early-Career Researchers. Frontiers in Psychology.
I really like this idea, so I’m going to criticize the one problem that stood out to me and wasn’t addressed in the opinion piece.
It’s not clear to me that shifting the burden of replication onto graduate students is a better outcome than the status quo, and I definitely don’t think it’s the optimal allocation. It’s unfair that early career researchers pay most of the costs of replication while the big fish are free to do whatever they please, but shifting the costs onto graduate students would not “eliminate any personal disadvantages that might accrue to individual students”, and would exacerbate them in some circumstances.
In a system where the supply and demand for PhDs in academia is so unbalanced (and only growing more so), it’s unlikely that hiring committees will accept less productivity from applicants who have spent a few months of their limited time in a PhD program on a replication attempt. Even more of the best jobs will go to grad students who can afford to work longer and longer hours and spend more resources on their work to make up for the difference – in other words, those who come from rich programs that pay generous stipends, those from large and well-funded labs that can afford to run studies and hire RAs to help, and those without obligations to spouses/children or health problems.
What’s the ideal allocation of the “burden of replication”? If anything, labs with more resources and more senior PIs should probably do the most replication, because they don’t face the costs in hiring/promotion that young researchers do, and they often have cheaper have access to larger samples and the manpower to conduct replications quickly.
I still think this is a great policy on balance, because most of those grad students weren’t going to get a long-term job in academia anyway, and what they learn can be valuable in whatever work they do (especially if it’s something data- or science-related in industry).
Ultimately, the success of a policy like this is going to depend on how hiring committees value replication relative to original research.
Like Nick, I think this is a really important issue that needs to be addressed, but I’m also a little worried that this ends up disproportionately burdening grad students (esp. those from more cash-strapped schools, as well as those in the ‘transition year’ who will be outshone by those in the year before them who didn’t have the replication requirement). The fundamental problem, as I see it, are the 5 disincentives Earp and Everett mention from publishing replications. The grad student solution dodges these and essentially takes advantage of a cheap, easily-pressured labor pool to do the low-prestige but necessary work – somewhat exploitative. If there’s enough will among those running grad schools to take serious collective action on the crisis, why not take advantage of that capital to address the 5 causes more directly, in a way that is more equitable (and may be more effective)? 1 and 2 are maybe intractable, but 3-5 are social phenomena. Grad school directors should take active steps to reward replication – being more willing to recommend them for publication when they review articles, taking them as seriously as more original articles when hiring, etc. That will increase the rewards for grad students and established researchers alike to engage in more replication studies.
I guess the advantage of the grad student approach is that it’s A) easier to implement and B) more concrete than improving disciplinary norms. But (A) only highlights the unfairness of the solution…it’s easier to implement because grad students are more vulnerable and easier to manipulate/coerce than one’s peers. Maybe some realpolitik is at work here though, where this the best/most realistic solution and it’s worth taking advantage of students to improve the field, which is a little depressing. As for (B), well, as Earp and Everett points out, this is a social phenomenon. It’s natural that we should look for a social solution, one that alters disciplinary norms, rather than a curricular one. And there may be ways to make altering 3-5 more concrete, like institutional policies about weighting of replication publications for hiring/tenure or journals instructing referees to take them more seriously.
Comments are closed.