Humans and other animals are naturally uncertain about their environment. They therefore need to explore. A large body of work in psychology has identified exploration as an intrinsically motivated behaviour that animals tend to engage in, even in the absence of immediately apparent ecological gains. In the context of decision-making, where exploration efficiency can be assessed, animals often display highly efficient, near-optimal exploration. However, surprisingly little is known about the algorithmic structure of exploratory choices made by animals, which would allow their exploration to be so efficient. We generalize an influential normative theory of hippocampal replay to cases of partial observability and show how hippocampal replay can play a role in arranging for approximately optimal exploration. In this talk, I will present this theory and some predictions for hippocampal replay we might expect in animals when faced with epistemic uncertainty, and how it could occasion efficient exploration.

Replay of hippocampal place cell sequences in rodents has been suggested to be involved in spatial memory consolidation and path planning. However, navigation rarely happens in isolation and often includes knowledge about anticipated or past movement of others, for instance, to avoid collision in physical space. While previous studies investigated individual spatial cognition, we know little about coordinated multi-agent navigation and whether replay plays any role in it, by prospectively simulating future trajectories of others and retrospectively replaying their past trajectories. Here, using recent methodological advances that can track replay in fMRI, we investigated pairs of participants who navigated a previously learned abstract state space in an interactive setting, trying to reach a common goal state while avoiding collisions, while their brain activity was measured using 3T fMRI hyperscanning. Behavioral results (N = 60) showed that participants improved their performance over trials, using their knowledge of the state space to navigate in the interactive setting successfully and reached the goal more often without colliding than expected by chance. fMRI data was analyzed using representational similarity analysis to show where the state space is represented in the brain. Furthermore, we identified replay-like events as the sequential reactivation of state representations by multivariate fMRI classification. Here, we looked at the prospective simulation of self and others’ future trajectories during planning to avoid collisions, and the retrospective replay of their past ones for learning, indicative of a credit assignment mechanism. Our study sheds light on the role of replay in multi-agent navigation.

A number of computational frameworks - most prominently reinforcement learning - have conceptualised reward as the driver of human and animal learning. However, animals and humans not only show reward-seeking behaviour in response to directly rewarded stimuli, but also in response to stimuli that have not been directly reinforced. These behaviours appear to draw on alternative neural mechanisms that allow reward, or credit, to be assigned to stimuli that have not been paired with reward directly. A candidate mechanism for assigning credit to stimuli that have not been directly paired with reward is replay. Replay can be observed in the spiking activity of neurons in the hippocampus, during periods of rest/sleep. Replay can be defined as the reactivation of temporally sequenced memories in the original or reverse order to previous behavioural experience. Here we investigated the relationship between neural reactivation and credit-assignment in humans, taking advantage of near whole brain imaging at 7T. Participants learned trajectories from a starting location to a target location. In the second stage of the task, some of the target locations were rewarded. Crucially, however, there were also ‘foil’ locations, which were only statistically correlated with the starting cues. Some of these foil locations also contained reward, allowing us to look at backpropagation of value to the starting cues for both causal and statistically correlated trajectories. Using ultra-high field fMRI in combination with decoding of neural activity during rest, we show evidence for value propagation during memory reactivation in periods of rest.

Reasoning flexibly composes known elements to solve novel problems. Recent theories suggest the brain uses the axis of time to compose elements for reasoning. By packaging elements into fast neural sequences, the brain could rapidly explore implications of different compositions. Using magnetoencephalography, we tested this idea while participants mentally executed programs. Each program contained a set of steps linked to operations, the implications of which had to be computed. By end of execution, inferred program solutions were represented in prefrontal and parietal cortices. We identified a possible mechanism by which these solutions were computed: during reasoning, representations of program steps reactivated in fast sequences, consistent with sampling of candidate partial solutions. Further evidence suggested these reactivations were accompanied by representations of their operations and followed by neural patterns reflecting their computed implications. Together, these results suggest replaying sequences supports program execution and reveal a highly organized temporal microarchitecture of reasoning.

Effective memory consolidation during sleep is thought to rely on the transfer of reactivated memory traces from the hippocampus to the cortex. However, the mechanisms supporting this essential dialogue across brain areas remain poorly understood. Here, we recorded single-neuron activity (n = 1097) in the medial temporal lobe (MTL) of 10 epilepsy patients across 21 nights of sleep. Before sleep, patients engaged in a memory task with stimuli eliciting concept-neuron responses. During non-rapid eye movement (NREM) sleep, neuronal firing locked to sharp wave–ripples (SWR) revealed a directed flow of information from the hippocampus to cortical targets. Notably, within SWR-driven activity, experience-dependent concept neurons exhibited elevated co-activation both within and across MTL regions, indicating selective reactivation of behaviorally relevant information. Cross-regional co-activation of concept cells was markedly enhanced when hippocampal SWRs coincided with cortical slow oscillation–spindle complexes, suggesting an active role of the cortex in shaping interregional communication. These findings provide evidence that SWRs in humans selectively reactivate experience-related neural ensembles across the hippocampal–cortical network, while synergistic interactions with cortical slow oscillation–spindle events might facilitate effective memory consolidation.