Book Review of “Thinking in Bets”, Part 1 of 2

Written by Annie “The Duchess of Poker” Duke, Thinking in Bets is a national bestseller, and for good reason. The writing style is direct and to-the-point, and the advice is motivated by concrete examples taken from the author’s own experience. For instance, one anecdote concerns a bet among a group of friends on whether or not one of them, “Ira the Whale”, could eat 100 White Castle burgers in a single sitting. David Grey, one of the author’s friends, bet $200 on the Whale:
(more…)

Redefine Statistical Significance XVIII: A Shockingly Honest Counterargument

Background: the 2018 article “Redefine Statistical Significance” suggested that it is prudent to treat p-values just below .05 with a grain of salt, as such p-values provide only weak evidence against the null. By threatening the status quo, this modest proposal ruffled some feathers and elicited a number of counterarguments. As discussed in this series of posts, none of these counterarguments pass muster. Recently, however, Johnson et al. (in press, Injury) presented an empirical counterargument that we believe is new. This counterargument is brutally honest and somewhat shocking (to us, anyway).

Johnson and colleagues start off by defining the p-value and its purpose:

“The primary purpose of using a P value is to minimize type I errors — erroneous conclusions made about differences between groups when no such difference truly exists. The type I error rate is often specified a priori at 0.05, meaning that there is a 1 in 20 chance — or a 5% risk — that the difference detected is because of chance rather than attributed to the effects of the intervention.”

(more…)

A Cartoon to Explain How Blinding Works

A Cartoon to Explain How Blinding Works

The cartoon presented below is available from the artwork library of BayesianSpectacles.org under a CC-BY license. The cartoon was conceptualized by Alexandra Sarafoglou and was drawn by Viktor Beekman. It is included as an appendix in Dutilh, G., Sarafoglou, A., & Wagenmakers, E.-J. (in press). Flexible yet fair: Blinding analyses in experimental psychology. Synthese. PsyArXiv Preprint: https://psyarxiv.com/d79r8

Because the procedure of analysis blinding is relatively rare and (in our experience) easily misunderstood, we clarified the key concepts using a cartoon. Here it is, courtesy of Viktor Beekman:
(more…)

Flexible Yet Fair: Blinding Analyses in Experimental Psychology

This post is an extended synopsis of Dutilh, G., Sarafoglou, A., & Wagenmakers, E.-J. (in press). Flexible yet fair: Blinding analyses in experimental psychology. Synthese. Preprint available on PsyArXiv: https://psyarxiv.com/h39jt

Abstract

The replicability of findings in experimental psychology can be improved by distinguishing sharply between hypothesis-generating research and hypothesis-testing research. This distinction can be achieved by preregistration, a method that has recently attracted widespread attention. Although preregistration is fair in the sense that it inoculates researchers against hindsight bias and confirmation bias, preregistration does not allow researchers to analyze the data flexibly without the analysis being demoted to exploratory. To alleviate this concern we discuss how researchers may conduct blinded analyses (MacCoun & Perlmutter, 2015). As with preregistration, blinded analyses break the feedback loop between the analysis plan and analysis outcome, thereby preventing cherry-picking and significance seeking. However, blinded analyses retain the flexibility to account for unexpected peculiarities in the data. We discuss different methods of blinding, offer recommendations for blinding of popular experimental designs, and introduce the design for an online blinding protocol.
(more…)

The Liberating Feeling of Relinquishing Control: Advice for Advisors

Disclaimer: advice based purely on the life and lab of the author. May not generalize to other people and other contexts. No literature whatsoever was consulted. Take advice at your own risk.

For most of my life I have had the idea that the key to happiness is control, or at least the illusion of control. What person would delight in having to follow orders and obeying the wishes of some random overlord? This is one of the perks of academia (at least in my Dutch bubble): the complete and utter freedom to do what you want, when you want, and where you want. Of course there can be lack of control in academia as well, even in the Dutch bubble. Once I felt a burn-out looming, and it was when I imagined myself standing at the bottom of a tall mountain, with an avalanche of work hurdling its way toward me, intent on crushing and suffocating — in other words, I experienced a profound lack of control.
(more…)