EarlGAN: An enhanced actor–critic reinforcement learning agent-driven GAN for de novo drug design

Deep generative models, such as Generative Adversarial Networks (GANs), have attracted the attention of researchers in the denovo drug design field. However, traditional GANs are typically used for image processing. Therefore, they are unsuitable for Simplified Molecular-Input Line-Entry System (SMILES) strings due to their discrete nature. Previous studies addressed this problem by combining Reinforcement Learning (RL) approaches with Monte Carlo tree search. However, for large chemical datasets, the molecule generation process is time-consuming due to the lengthy atom-by-atom sampling process with cumulative reward, an essence of the Monte Carlo tree search-based RL approaches. To address this problem, we propose an enhanced actor–critic RL agent-driven GAN, called EarlGAN, for denovo drug design. Specifically, EarlGAN’s generator acts as an actor to generate SMILES strings, and the discriminator acts as a critic to perform discrimination. EarlGAN makes autoregressive predictions at the atomic level. While the generator is based on previously generated atoms, the discriminator discriminates using a bidirectional pass over the atoms, including the current atom that is being predicted. We integrate moment, global-level discrimination rewards, and information entropy maximization. The moment rewards reduce the computation time, and the global-level rewards ensure the consistency of the molecule, whereas the information entropy maximization leads to a more diverse sample generation. Experiments and ablation studies verify the effectiveness of EarlGAN for denovo drug design on the QM9 and ZINC datasets. In addition, the visualization analysis provides insight into EarlGAN and supports our conclusion.