Summary
A pre-registered experiment is one for which hypotheses, analysis pipelines, and interpretations of results are all specified prior to data collection. By minimising analytic flexibility, the resulting research becomes more trustworthy and more likely to replicate. A trustworthy and replicable foundation of empirical research is essential for consciousness science to be successful, because without it we cannot build good theories. Enthusiasm for pre-registration is growing: a number of journals now award “badges” for pre-registered studies and a growing number of funders are encouraging applicants to adopt such methods. However, writing a pre-registration that both specifies an appropriate analysis pipeline and successfully constrains analysis choices is difficult - indeed many are inadequate and/or poorly followed - so researchers may benefit from explicit training in these methods.
This tutorial is designed for empirical researchers at all career stages. We will guide participants through the process of writing an effective yet concise pre-registration, covering topics such as how to make pre-processing choices, how to estimate effect sizes, and how to anticipate problems with the dataset. We will also discuss tips for pre-registering more complex analyses, including neuroimaging and computational research. The tutorial will be highly interactive – participants will work in small, discipline-specific groups to design research plans together, gaining hands-on experience in writing pre-registrations in their area of expertise. By the end, we hope to equip researchers with the skills necessary to create thorough pre-registrations, ultimately enhancing the credibility and rigour of scientific research.
Rationale on speaker selection and proof of their expertise
Zoltan Dienes is at the forefront of open science, pre-registration and registered reports. He has published widely on metascience (e.g. [1-3]) and statistical best practice in psychological research, with particular emphasis on Bayes Factors (e.g. [4-5]). In 2013 he joined the first Registered Reports editorial team at Cortex. He also spearheaded the introduction of registered reports to the ASSC journal Neuroscience of Consciousness, acting as associate editor for registered reports submissions until 2024. He is a founding member of Peer Community In Registered Reports (PCI-RR) and sits on the managing board.
Maxine Sherman has pre-registered all of her empirical work since starting her first postdoctoral position and now has extensive hands-on experience in pre-registration of consciousness research, including for projects using psychophysics, neuroimaging, computational modelling, questionnaire measures, and machine learning. She is a recommender (editor) for PCI Registered Reports.
Proof:
Zoltan Dienes
- https://rr.peercommunityin.org/public/user_public_page?userId=5
- https://scholar.google.co.uk/citations?user=1n48dTUAAAAJ&hl=en
Maxine Sherman
- https://rr.peercommunityin.org/public/user_public_page?userId=2007
- https://osf.io/yn3r7/
Desired educational expectations
1. To understand the information that needs to be present in a pre-registration
2. To understand how to design effective analysis plans prior to seeing the data
3. To be able to successfully identify all sources of analytic flexibility
4. To be equipped with strategies for planning complex analyses, e.g. on neuroimaging data
5. To be equipped with strategies for dealing with unexpected results
6. To gain experience writing an effective pre-registration
Proposed audience engagement
After participants sign up for the tutorial we will ask them to fill out a very brief survey that asks about the kinds of techniques they would like to pre-register (e.g. reaction time studies, psychophysics, physiological recordings, EEG, fMRI, computational modelling). We will use this information to allocate participants into technique-specific small groups. Audience participation will be central throughout the tutorial: as well as discussing topics altogether, in all sections of the session the small groups will work together on guided exercises centered around each key component of a pre-registration. At the end of each exercise another component of the pre-registration will have been planned. By the end of the session, each small group will have designed a full pre-registration for a hypothetical (or actual) study in their domain.
Planned structure
The tutorial will be split into multiple sections (with a break half-way), each addressing a different component of a pre-registration. Sections will begin with an introduction and tips and tricks for how to write that component. Participants will then work through exercises in their small groups and. Finally, will whole-group discussions will reflect on elements that were challenging, how decisions were made and issues that were identified.
The structure will be as follows:
1) Introduction
2) Designing a strong analysis plan
We will work through how to commit to answers to the following questions:
- What does my theory predict?
- What manipulation checks do I need to conduct?
- What will my data look like and what statistical model should I use?
- What patterns of results might I find and how would I interpret them?
3) Effect sizes, power, priors and nulls
Here we will cover:
- How to estimate effect sizes and minimum effects of interest
- How to determine sample size
- How to assert the null
4) Data Preprocessing & Anticipating problems
Anticipating problems is arguably the most important part of writing an effective pre-registration because there is substantial analytical flexibility in how data are preprocessed and/or excluded. This section will cover:
- How to anticipate and deal with issues such as poor task performance, failed manipulation checks, and issues specific to neuroimaging/psychophysiology/modelling.
- Exclusion and inclusion criteria
- Preprocessing choices
5) Conclusions
Rationale on panel inclusivity
The panel is balanced across gender and career stage but not across ethnicity, disability, socioeconomic class or other factors. This would be difficult to achieve with only two panelists, but we hope to be able to expand the session in future years.
The theme of the tutorial honours diversity because open, transparent science is more democratic. A democratic scientific system encourages diversity by virtue of the fact that it counters academic oligarchy [see 3].
References
[1] Schumann, F., Smolka, M., Dienes, Z., Lübbert, A., Lukas, W., Rees, M. G., ... & Van Vugt, M. (2023). Beyond kindness: a proposal for the flourishing of science and scientists alike. Royal Society Open Science, 10(11), 230728.
[2] Dienes, Z. (2024). The inner workings of Registered Reports. In Austin Lee Nichols & John E. Edlund (Eds), Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences, Volume 2
[3] Dienes, Z. (2023). The credibility crisis and democratic governance: How to reform university governance to be compatible with the nature of science. Royal Society Open Science, 10(1), 220808.
[4] Catriona, S., Dienes, Z., & Wonnacott, E. (2024). Bayes factors for logistic (mixed effect) models. Psychological Methods.
[5] Dienes, Z. (2023). Testing theories with Bayes factors. In Austin Lee Nichols & John E. Edlund (Eds), Cambridge Handbook of Research Methods and Statistics for the Social and Behavioral Sciences
