Opponent learning with different representations in the cortico-basal ganglia circuits.

The direct and indirect pathways of the basal ganglia (BG) have been suggested to learn mainly from positive and negative feedbacks, respectively. Since these pathways unevenly receive inputs from different cortical neuron types and/or regions, they may preferentially use different state/action representations. We explored whether such a combined use of different representations, coupled with different learning rates from positive and negative reward prediction errors (RPEs), has computational benefits. We modeled animal as an agent equipped with two learning systems, each of which adopted individual representation (IR) or successor representation (SR) of states. With varying the combination of IR or SR and also the learning rates from positive and negative RPEs in each system, we examined how the agent performed in a dynamic reward navigation task. We found that combination of SR-based system learning mainly from positive RPEs and IR-based system learning mainly from negative RPEs could achieve a good performance in the task, as compared to other combinations. In such a combination of appetitive SR-based and aversive IR-based systems, both systems show activities of comparable magnitudes with opposite signs, consistent with the suggested profiles of the two BG pathways. Moreover, the architecture of such a combination provides a novel coherent explanation for the functional significance and underlying mechanism of diverse findings about the cortico-BG circuits. These results suggest that particularly combining different representations with appetitive and aversive learning could be an effective learning strategy in certain dynamic environments, and it might actually be implemented in the cortico-BG circuits.Animals can learn the value of states/actions from both positive and negative feedbacks. For learning, animals need to represent each state/action, individually (like representing a person by her/his identity only) or in a relation-based manner (like representing a person by friends or descendants). Different brain circuits may learn from positive and negative feedbacks with different rates, and may represent states/actions in different ways. We explored what combination of the feedback valence-dependent learning rates and the ways of state representation performs well in a dynamic reward navigation task. We found that a particular combination performed well, and we propose that several known anatomical and physiological properties of the cortico-basal ganglia circuits may indicate implementation of such a combination.