This book review is a translated and slightly adjusted version of one that is currently in press for “De Psycholoog”. The review was inspired by the recent Dutch translation De 7 Doodzonden van de Psychologie (see references below).

In his inaugural address on September 11th 2001, Diederik Stapel made a bold claim about the prestige and accomplishments of the field of social psychology: “Physics may have crowned itself king, but social psychology is queen”. The notion of psychological science as an infallible source of knowledge returns in Daniel Kahneman’s bestseller Thinking, Fast and Slow. Specifically, Kahneman praises psychological research on the phenomenon of behavioral priming — a famous example of such priming is that people supposedly walk more slowly after reading words such as “grey” and “false teeth”; these words activate the concept “elderly” which in turn is associated with walking more slowly. Kahneman may be impressed with this type of work, but my experience is that, upon being confronted with priming research, most audiences start to laugh. Perhaps Kahneman shares this experience, for he writes: “When I describe priming studies to audiences, the reaction is often disbelief (…) The idea you should focus on, however, is that disbelief is not an option. The results are not made up, nor are they statistical flukes. You have no choice but to accept that the major conclusions of these studies are true” (pp. 56-57). Back in the day, critique on the scientific status quo was not appreciated: when a full professor at the University of Amsterdam once suggested that certain subdisciplines of psychology were better off being bulldozed, he ended up having to apologize profusely to his deeply offended colleagues. Lese-majesty!

Rarely in the history of science did pride come before a fall so sudden and so deep. Diederik Stapel turned out to have fabricated data on an industrial scale, and –to the best of my knowledge– from the entire priming chapter in Thinking, Fast and Slow not a single experiment has been replicated successfully: disbelief turned out to be a viable option after all. The current psychological literature features the bulldozer, together with the bazooka and the bomber, as part of an accepted series of measures to cleanse the scientific record and repair the damaged trust.

How could this have happened? Why is psychological research not as replicable as we would like it to be? Do psychologists fool themselves and each other? And more importantly: how should we move on, and how can we do better? Those who seek an answer to these pressing questions ought to read the book by cognitive neuroscientist Chris Chambers. Chambers is a man on a mission: fighting scientific biases that unwittingly turn results into spin, researchers into believers, and the literature into Fox news (“Fair and Balanced”).

The first seven chapters of Deadly Sins cover each in turn: the sin of bias, the sin of hidden flexibility, the sin of unreliability, the sin of data hoarding, the sin of corruptibility, the sin of internment, and the sin of bean counting. Each of these chapters is sobering, sometimes even depressing, but in my opinion constitutes mandatory reading for the modern-day psychologist — and for the modern-day empirical researcher more generally.

Fortunately the book contains more than a list of lamentations. In Chapter 8, “redemption”, Chambers describes a concrete, simple proposal for radical improvement: the “Registered Report” format. In a Registered Report the researcher first submits to the journal a detailed proposal for an experiment. This proposal, including the analysis plan, is then assessed through peer review. Once the plan –usually after incorporating the reviewers’ feedback– has been approved the researchers obtains “in principle acceptance” (IPA). Only then are the data collected. When the data are shown to be of sufficient quality (using an outcome-independent check) then the article is published independent of the results and their statistical significance. The Registered Report format brings many advantages: it eliminates publication bias, it eliminates hindsight bias (both on the part of the authors and on the part of the reviewers), it safeguards the meaning of the statistical analyses, and it allows a sharp distinction between confirmatory hypothesis tests versus exploratory fishing expeditions, exactly as intended by the Dutch methodologist Adriaan de Groot (1956/2014). In other words: the publication of an article is determined solely by the quality of the work –the hypothesis, the experimental design– and not the outcome. This prevents journals and researchers from unwittingly cherry-picking outcomes and fooling the reader in the process. Over the last six years Chris Chambers has tirelessly promoted his Registered Reports, and currently they are on offer in over 140 journals (including Nature: Human Behaviour).

So there is hope after all. Maybe Diederik Stapel was right, and psychology is indeed the queen of science — but then the vain and cruel queen from the fairy tale of Snow White, handing out non-replicable apples all across the land. And just as in the fairy tale, the story does not end with the apple, because party due to the efforts of Chris Chambers the field of psychology has finally risen from its slumber. The desired culture reform is now in full swing; having learned from past mistakes, psychology now plays the part of scientific pioneer. And this is something to be proud of.

References

Chambers, C. D. (2017). The seven deadly sins of psychology: A manifesto for reforming the culture of scientific practice. Princeton: Princeton University Press. Recently translated in Dutch as De 7 doodzonden van de psychologie: Pleidooi voor een cultuuromslag in de wetenschappelijke praktijk.

De Groot, A. D. (1956/2014). The meaning of “significance” for different types of research. Translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, & Han L. J. van der Maas. Acta Psychologica, 148, 188-194.

Kahneman, D. (2011). Thinking, fast and slow. London: Allen Lane

About The Authors

Eric-Jan Wagenmakers

Eric-Jan (EJ) Wagenmakers is professor at the Psychological Methods Group at the University of Amsterdam.