On the Road to Perfection? Evaluating Leela Chess Zero Against Endgame Tablebases

Board game research has pursued two distinct but linked objectives: solving games, and strong play using heuristics. In our case study in the game of chess, we analyze how current AlphaZero type architectures learn and play late chess endgames, for which perfect play tablebases are available. We study the open source program Leela Chess Zero in three and four piece chess endgames. We quantify the program’s move decision errors for both an intermediate and a strong version, and for both the raw policy network and the full MCTS-based player. We discuss a number of interesting types of errors by using examples, explain how they come about, and present evidence-based conjectures on the types of positions that still cause problems for these impressive engines.