Enhancing Code Prediction Transformer with AST Structural Matrix.

Deep learning Transformer architectures play a critical role in developing advanced code prediction models, which are essential in modern Integrated Development Environments (IDEs). Nevertheless, these architectures encounter a significant challenge in effectively capturing and utilizing the structural information present in Abstract Syntax Trees (ASTs). To tackle this challenge, we propose an innovative approach that leverages AST structural matrices to enhance Transformers for source code prediction. Specifically, we integrate three types of AST structural matrices - the R matrix, A&S matrix, and MVG matrix - into the attention module of the Transformer to effectively capture the structural information within ASTs. To ensure optimal utilization, we have devised a range of strategies and integration methods tailored specifically for the attention module. To assess the effectiveness of our proposal, we conduct empirical studies using a standard Python dataset. The results demonstrate that incorporating AST structural matrices significantly enhances the accuracy of code prediction models, leading to an overall improvement from 73.18% to 75.10%. Furthermore, we conduct an in-depth analysis of the impact of each matrix type, offering a comprehensive understanding of their application scenarios. This insightful analysis provides valuable guidance on how to effectively leverage each matrix type in various contexts.