The Bayesian Chef

Over the past year or so I’ve been working on a book provisionally titled “Bayesian bedtime stories”. Below is a part of the preface. The next post continues the cooking analogy by introducing the frequentist chef.

Even though the book [Bayesian Bedtime Stories] addresses a large variety of questions, the method of reasoning is always based on the same principle: contradictions and internal inconsistencies are not allowed. For instance, the propositions ‘Linda is a bank teller’ and ‘Linda is a feminist’ are each necessarily more plausible than the conjunction ‘Linda is a feminist bank teller’. Any method of reasoning that leads to a different conclusion is seriously deficient.

In order for our reasoning to be reasonable we therefore need to exclude relentlessly from consideration all methods, however beguiling or familiar, that produce internal inconsistencies. When we remove the debris only a single method remains. This method, known as Bayesian inference, stipulates that when we reason with uncertainty, we should obey the laws of probability theory. Simple and elegant, these laws lay the foundation for a reasoning process that cannot be improved upon; it is perfect — the reasoning process of the gods.1

‘Thou shalt not contradict thyself’. The equation in the clouds shows Bayes’ rule: the only way to reason with uncertainty while not contradicting yourself. Bayes’ rule states that our prior opinions are updated by data in proportion to predictive success: opinions that predicted the data better than average receive a boost in credibility, whereas opinions that predicted the data worse than average suffer a decline (Wagenmakers et al., 2016). In other words, the learning process is driven by relative prediction errors. CC-BY: Artwork by Viktor Beekman, concept by Eric-Jan Wagenmakers.

As may be expected, adopting the reasoning process of a god brings several advantages. One of these advantages is that only the ingredients of the reasoning process are up for debate; that is, one may discuss how exactly a particular model of the world is to be constructed — how ideas are translated to numbers and equations. The proper design of statistical models is an art that requires both training and talent. One may also discuss what data are relevant for the model. But once the ingredients –model and data– are in place, the reasoning process itself unfolds in a single unique way. No discussion about that process is possible. Given the model of the world and the data available, the gods’ method of reasoning is unwavering and will infallibly lead to the same conclusion. That conclusion is misleading only to the extent that the ingredients were misleading.

Let’s emphasize this important advantage by further exploiting the cooking analogy. Suppose that, given particular ingredients, there exists a single unique way of preparing the best meal. You may have poor ingredients at your disposal –six ounces of half-rotten meat, two old potatoes, and a molded piece of cheese– but given these ingredients, you can follow a few simple rules and create the single best meal, a meal that even Andhrimnir, the Norse god of cooking, could not improve upon. What chef would deviate from these rules and willingly create an inferior dish? 2

The gods’ reasoning process is named after the reverend Thomas Bayes who first discovered the main theorem. What Bayes’ theorem (henceforth Bayes’ rule) accomplished is to outline how prior (pre-data) uncertainties and beliefs shall be updated to posterior (post-data) uncertainties and beliefs; in short, Bayes’ rule tells us how we ought to learn from experience.

All living creatures learn from experience, and this must be done by updating knowledge in light of prediction errors: gross prediction errors necessitate large adjustments in knowledge, whereas small prediction errors necessitate only minor adjustments.

In general terms, we then have the following rule for learning from experience:

Present uncertainty about the world
Past uncertainty about the world
Predictive updating factor


The bottom line is that Bayes’ rule allows its followers to use the power of probability theory to learn about the things they are unsure of. Nevertheless, Bayesian inference is not without serious competition. In the next post, we will examine the frequentist chef (uh-oh, indigestion alert) and compare the two.


1 As documented in many science fiction stories, the universe ceases to exist at the exact moment when its creator becomes aware of an internal inconsistency.

2 We purposefully ignore the fact that Andhrimnir only prepares a single dish. At Godchecker, the entry on Andhrimnir states: “He’s an Aesir chef with only one house special. He takes the Cosmic Boar. He kills it. He cooks it. The Gods eat it. It returns to life in the night ready for use in the next set meal. It’s a real pig of a life for the boar. A little variety in the kitchen would work wonders.”


Wagenmakers, E.-J., Morey, R. D., & Lee, M. D. (2016). Bayesian benefits for the pragmatic researcher. Current Directions in Psychological Science, 25, 169-176.

About The Author

Eric-Jan Wagenmakers

Eric-Jan (EJ) Wagenmakers is professor at the Psychological Methods Group at the University of Amsterdam.

Let’s Poke a Pizza: A New Cartoon to Explain the Strength of Evidence in a Bayes Factor

In a previous post we discussed the interpretation of the strength of evidence coming from a Bayes factor. For concreteness, let’s take a binomial data set and suppose we have encountered 62 successes and 38 failures in a sample of 100 trials. We can easily enter these data directly using the JASP Summary Stats Module:



Laplace’s Demon

if there could be any mortal who could observe with his mind the interconnection of all causes, nothing indeed would escape him. For he who knows the causes of things that are to be necessarily knows all the things that are going to be. (…) For the things which are going to be do not come into existence suddenly, but the passage of time is like the unwinding of a rope, producing nothing new but unfolding what was there at first.
– Cicero, de Divinatione, 44 BC


A deterministic universe consists of causal chains that link past, present, and future in an unbreakable bond; the fact that you are reading these words right now is an inevitable consequence of domino-like cause-and-effect relationships that date back all the way to the Big Bang. If this is true, Laplace argued, then complete knowledge of the universe at any particular time allows one to perfectly predict the future and flawlessly retrace the past. The hypothetical intelligence that would possess such complete knowledge has become known as ‘Laplace’s demon’.


Limitations of Bayesian Leave-One-Out Cross-Validation for Model Selection

This post is an extended synopsis of a preprint that is available on PsyArXiv.

“[…] if you can’t do simple problems, how can you do complicated ones?” — Dennis Lindley (1985, p. 65)

Cross-validation (CV) is increasingly popular as a generic method to adjudicate between mathematical models of cognition and behavior. In order to measure model generalizability, CV quantifies out-of-sample predictive performance, and the CV preference goes to the model that predicted the out-of-sample data best. The advantages of CV include theoretic simplicity and practical feasibility. Despite its prominence, however, the limitations of CV are often underappreciated. We demonstrate with three concrete examples how Bayesian leave-one-out cross-validation (referred to as LOO) can yield conclusions that appear undesirable.


On the Importance of Avoiding Shortcuts in Applying Cognitive Models to Hierarchical Data

This post summarizes the content of an article that is in press for Behavior Research Methods. The preprint is available on PsyArXiv.

Psychological experiments often yield data that are hierarchically structured. A number of popular shortcut strategies in cognitive modeling do not properly accommodate this structure and can result in biased conclusions. First, we considered a modeling strategy that ignores the hierarchical data structure, missing random effects on the participant-level. Our theoretical analysis indicates that this biases statistical results towards the null hypothesis. Second, we considered a modeling strategy that takes a two-step approach by first obtaining participant-level estimates from a hierarchical cognitive model and subsequently using these estimates in a follow-up statistical test. Our theoretical analysis indicates that this biases statistical results towards the alternative hypothesis. Using a simulation study for a two-group experiment, we demonstrate that both strategies result in considerable statistical biases when parameter estimation is based on little data; only hierarchical models of the multilevel data lead to correct conclusions. These results are particularly relevant for applications of hierarchical Bayesian cognitive models in settings with limitations on the size of the available data sets, such as clinical studies.


Musings on Preregistration: The Case of the Facial Feedback Effect

tl;dr. In 2016, the results of a multi-lab preregistered replication effort cast doubt on the idea, motivated by the “facial feedback hypothesis”, that holding a pen with one’s teeth (instead of with one’s lips) makes cartoons appear more amusing. The effect’s progenitor, Dr. Strack, critiqued the replication effort and suggested that the presence of a camera (put in place to confirm that participants held the pen as instructed) makes the effect disappear. This conjecture has recently received some empirical support from a preregistered experiment (Noah et al., 2018, JPSP). Overall, Noah et al. present a balanced account of their findings and acknowledge the considerable statistical uncertainty that remains. It may be the case that the camera matters, although we personally would currently still bet against it. Methodologically, we emphasize how important it is that the confirmatory analysis adheres exactly to what has been preregistered. In order to prevent the temptation of subtle changes in the planned analysis and at the same time eliminate the file drawer effect, we recommend the Registered Report over plain preregistration, although preregistration alone is vastly superior to registering nothing at all.

This tale starts in 2012. Doyen and colleagues had just published a bombshell paper, “Behavioral Priming: It’s All in the Mind, but Whose Mind?” (open access). Here is the first part of their abstract:

“The perspective that behavior is often driven by unconscious determinants has become widespread in social psychology. Bargh, Chen, and Burrows’ (1996) famous study, in which participants unwittingly exposed to the stereotype of age walked slower when exiting the laboratory, was instrumental in defining this perspective. Here, we present two experiments aimed at replicating the original study. Despite the use of automated timing methods and a larger sample, our first experiment failed to show priming. Our second experiment was aimed at manipulating the beliefs of the experimenters: Half were led to think that participants would walk slower when primed congruently, and the other half was led to expect the opposite. Strikingly, we obtained a walking speed effect, but only when experimenters believed participants would indeed walk slower.” (Doyen et al., 2012)


« Previous Entries

Powered by WordPress | Designed by Elegant Themes