Q-learning and hyper-heuristic based algorithm recommendation for changing environments

A considerable amount of research has been devoted to solving static optimization problems via bio-inspired metaheuristic algorithms. However, most of the algorithms assume that all problem-related data remain unchanged during the optimization process, which is not a realistic assumption. Recently, dynamic optimization problems (DOPs) grabbed remarkable attention from the research community. However, the literature still lacks clear guidelines on selecting the most appropriate bio-inspired algorithm under changing environments. Due to the availability of many design choices, the selection of a suitable bio-inspired metaheuristic algorithm becomes an immediate challenge. This study proposes an algorithm recommendation architecture using Q-learning and hyper-heuristic approaches to help decision-makers select the most suitable bio-inspired algorithm for a given problem. To this end, Artificial Bee Colony (ABC), Manta Ray Foraging Optimization (MRFO), Salp Swarm Algorithm (SSA), and Whale Optimization Algorithm (WOA) are employed as low-level optimizers so that the Q-learning and hyper-heuristic automatically select the optimizer in each cycle of the optimization process. The proposed algorithms are implemented in dynamic multidimensional knapsack problems, a natural extension of the well-known 0–1 knapsack problem. The performances of the recommender and standalone bio-inspired algorithms are evaluated through a comprehensive experimental analysis including appropriate statistical tests. Thus, the significant differences among the algorithms are revealed. The obtained results point out the efficiencies of the Q-learning-based algorithm recommender and MRFO in solving the dynamic multidimensional knapsack problem.