Better Theory Development as a Response to the Prevalence of QRPs

The Journal Club: “ReproducibiliTea” at LMU Munich continues with its ninth meeting to discuss an interesting paper on how better theory development could be an answer to current replicability problems. The organization team is happy to announce that the paper’s author, Prof. Dr. Chris Donkin, will give an introductory talk to this session and participate in the open discussion afterwards:

Participants can use the following key questions to prepare for the discussion:

• How might the flexibility of theories contribute to replicability problems?
• What are the characteristics of good theories?
• Is preregistration misguided to prevent QRPs? What could be arguments for and against such an opinion?

You can find a short abstract for the introductory talk by Prof. Dr. Chris Donkin below:

Better theory development as a response to the prevalence of QRPs

Concerns about so-called QRPs, and especially p-hacking, are typically based on concerns about false-positive statistical outcomes. However, the outcome of a statistical test is not equivalent to a scientific claim, and so elevated Type-I error rates do not speak directly to reduced replicability. We will explain why certain kinds of statistical analyses, including those forms of p-hacking that seem most egregious, will indeed lead to an inability to predict the outcome of future experiments. The problem, simply put, is that without sufficient scientific argument, any statistical analysis taken at face value is an ad hoc explanation, from which there is no good reason to expect to understand the world. We contrast such explanations with good theories and discuss what those are. Thinking about p-hacking from this perspective helps provide a way of deciding what kinds of analyses are good or bad, independent of whether they are p-hacked, ‘exploratory’, or ‘confirmatory’.

About the speaker:

Chris Donkin is a Professor of Computational Modelling in Psychology at LMU Munich. His work focuses on explaining cognitive phenomena, such as memory and decision-making, often relying on using mathematical and computational models as aids to help understand and test our theories.

Concept of the Journal Club

The interdisciplinary journal club “ReproducibiliTea” is launched every 3-4 weeks and addresses main questions and current topics concerning Open Science and the credibility of research findings. The idea - originated from Great Britain - was already developed in many other countries using the same name. During on-site meetings tea, juice and small snacks are provided to live up to the slogan. However, some meetings are held online due to organizational reasons. No matter where the sessions take place, everybody is welcome to bring their own food and drinks and thus connect the journal club with their lunch break.

The Journal Club consists of open talk sessions for discussing Open Science, reproducibility, and other related issues and is explicitly open to all disciplines and institutions to create the most interesting interchange. Each session starts with a short talk about the paper and then continues with an open discussion in which you can contribute your own thoughts and questions.

During a longer break between 2020 and 2021, the concept was slightly revised, and the organization team decided to invite the speaker(s) of the respective paper that is discussed in each session. The author herself/himself starts with the introductory talk including a short summary of the paper. Afterwards, there is enough time for direct interaction with the author(s) and, as before, an open discussion with all of the participants.

You are warmly welcome to join the next meetings. You can simply send an e-mail to reproducibilitea.munich@gmail.com to join the mailing list to get information about the next meetings and chosen papers. For each session it would be helpful to have read the announced paper, but there is no obligation to do so.

The organization team is looking forward to seeing you at one of the next Journal Club sessions.

organization team:

• Laura Goetz, Medical Student at LMU Munich
• Stephan Nuding, Psychology Student at LMU Munich
• Leonhard Schramm, Psychology Student at LMU Munich

Presenters

• Prof. Dr. Chris Donkin