Where, When & Which Concepts Does AlphaZero Learn? Lessons from the Game of Hex

AlphaZero, an approach to reinforcement learning that couples neural networks and Monte Carlo tree search (MCTS), has produced state-of-the-art strategies for traditional board games like Chess, Go, and Hex. While researchers and game commentators have suggested that AlphaZero uses concepts humans consider important, it is unclear how these concepts are represented in the network. We investigate AlphaZero’s representations in Hex using both model probing and behavioral tests. We find that the MCTS search initially finds important concepts, and then the neural network learns to encode these concepts. Concepts related to short-term end-game planning are best encoded in the final layers of the model, whereas concepts related to long-term planning are encoded in the middle layers of the model.