In a study published in the scientific journal Neuron, scientists pin down a brain area crucial for this type of learning and demonstrate how its activity encodes multiple aspects of the decision-making process.
Predicting the outcomes of actions in order to make good decisions is a critical role of brain function. This process is thought to work through two fundamentally different mechanisms called "model-free" and "model-based" learning. Though fundamental for flexible and adaptive behaviour, the neurobiology of model-based learning remains poorly understood.
“Model-free and the model-based learning are distinct, but complementary,” said lead author Thomas Akam, a researcher at Oxford University, who worked together on this study with Rui Costa, investigator at the Champalimaud Centre for the Unknown and now Director and CEO of Columbia's Zuckerman Institute, and Peter Dayan, Director of the Max Planck Institute for Biological Cybernetics in Tübingen
The model-based approach relies on understanding the underlying structure of the problem and creating a plan, for instance figuring out the best route to get to a new restaurant, while the model-free approach allows you to act quickly with less mental effort in familiar situations.
According to the authors, the brain switches between these modes of acting all the time without even realising. For instance, if you find a closed road on your habitual route to the restaurant, you may quickly transition to the model-based approach to come up with an alternative. To isolate the contribution of these two cognitive schemes, the researchers developed a novel experimental task.
Mice were used to study the brain mechanisms. The mice would initiate a trial by poking their noses into one of two central ports, located one above the other. This would light up one of two side ports where the mice could collect a water reward.
To do the task well, the mice had to figure out two key variables. The first was which side-port was more likely to offer a reward. The second was which of the central ports activated the more rewarding side port. Once the mice learned the task, they would opt for the action sequence that offered the best outcome.
Though this task may seem artificial, Akam pointed out that it captures certain important features of real-world decision-making.
To promote flexible learning strategies, every now and then, one of two changes would happen. “One manipulation was to switch the mapping between the central and the side ports. The other was to change which of the side ports had a higher probability of giving reward,” Akam explained.
“In principle, the task can be solved by either model-free or model-based learning; mice could simply learn the model-free prediction 'top is good', or they could learn a model of the task 'top leads to left, left to reward.” Said Akam. “However, these different strategies would generate different patterns of choices. By looking at the subjects' behaviour we were able to assess the contribution of either approach.”
When the team analysed the results - about 230,000 individual decisions - they learned that the mice were using both approaches in parallel. Dr Costa shared that this confirmed that the task was suitable for studying the neural basis of these mechanisms.
The team focused on a brain region called anterior cingulate cortex (ACC), which is well-established to be in action selection and provided some evidence that it could be involved in model-based predictions. But its involvement in a task designed to differentiate between these different types of learning has yet to be studied. The researchers discovered that the activity of the neurons created a map that represented various aspects of the behaviour of the mice by looking at the pattern of activity across the population.
In addition to representing the animal's current location in the task, ACC neurons also encoded which state was likely to come next, providing evidence of its involvement in making model-based predictions of specific consequences of actions.
Moreover, ACC neurons also represented whether the outcome of actions was expected or surprising, potentially providing a mechanism for updating predictions when they turn out to be wrong.
Finally, to test whether the ACC was needed for model-based decision-making, the team silenced ACC neurons in individual trials while the animals were deciding what option to choose. As a result, “mice failed to correctly update their strategy, suggesting that silencing ACC prevents the animals from using model-based predictions. Consistent with this interpretation, ACC silencing had a stronger effect on subjects who relied more on a model-based strategy,” explained Akam.
“These data identify the anterior cingulate cortex as a key brain region in model-based decision-making, more specifically in predicting what will happen in the world if we choose to do a particular action versus another.” Said Dr Costa. According to the authors, a big challenge in contemporary neuroscience is understanding how the brain controls complex behaviours like planning and sequential decision making.
“These results will allow us and others to use the powerful tools for monitoring and manipulating brain activity available in this species to build mechanistic understanding of flexible decision making.” Akam concluded.