LT;DR In the current era of AI, there is no longer any excuse for manuscripts that contain typographical errors, grammatical errors, unclear formulations, errors in equations, incorrect references, missing references, and a demonstrated lack of awareness for related work. A recent example of my own writing demonstrates that an AI (i.e., ChatGPT Pro) can offer high-quality feedback. Incorporating two rounds of AI feedback improved the quality of the manuscript tremendously. Submitting and publishing academic writing without first solliciting and incorporating AI feedback arguably does a disservice to the field and to the readership.
Introduction
I violently dislike the idea of AI taking over my writing. My writing is my own, and having it done by AI makes the final product lose its soul. Also, whenever I have used AI to write several paragraphs independently (which I admit to doing for bureaucratic tasks) I ended up rewriting most of it anyway.
However, over the past year or so I have become increasingly impressed with what AI can do, and rather than talk about this in abstract terms I would like to present you with a concrete demonstration that shifted my opinion a great deal.
Background
Yesterday I finalized two chapters (of about 40 pages total) that are part of a syllabus for the course Behavioural Data Science, which I co-teach with Denny Borsboom. I consider myself an expert on the topic (i.e., Bayesian inference) and I consider myself a reasonably good writer — at the very least I’ve been at it for about 30 years.
AI Feedback
Having written the chapters without AI support, I then fed the pdf to ChatGPT Pro and asked it for feedback:
After about 25 minutes, ChatGPT was done:
The result is nothing short of amazing. I am meticulous with my writing (just ask my students) but ChatGPT had found numerous mistakes. Moreover, ChatGPT had organized the feedback including an overall summary, paragraph-by-paragraph suggestions for improvement, and an indication of the level of severity of the mistakes. It turned out that ChatGPT was correct (and I was wrong) in the great majority of the cases that it flagged. Here is a screenshot of two pages of the Word document (the entire document is here):
After incorporating the feedback (and corresponding with Chat about the details of some specific suggestions) I uploaded a revised version and asked for another round of feedback. To my surprise, this resulted in another deluge of comments — again. most of them on point. Here is a screenshot of two pages from the Word document (the full document is here):
Take-Home Message
The quality and quantity of ChatGPT’s feedback left me stunned. I had no idea that I could make so many blatant mistakes. This sobering experience leads me to believe that academic authors can generally improve their manuscripts tremendously by asking an LLM for paragraph-by-paragraph feedback. So much is there to gain, in fact, that it is arguably unethical to present a manuscript to reviewers and a general readership when it is likely strewn with errors — errors, moreover, that are trivial to catch with an LLM.
I do wish to stress that asking AI for feedback on a completed product is fundamentally different from involving AI in the writing process from the very beginning. For academic writing such an early partnership is still distasteful to me.
Finally, it is important to realize that the AI feedback above was produced by ChatGPT Pro in extended thinking mode: it took 25 minutes to produce a report. I have not checked the quality of the reports that the free version of Chat would generate, but I suspect it may be much lower.
NB. This blogpost was not written with the help of AI 😉
Eric-Jan Wagenmakers
Eric-Jan (EJ) Wagenmakers is professor at the Psychological Methods Group at the University of Amsterdam.
