A new Q-function structure for model-free adaptive optimal tracking control with asymmetric constrained inputs

This article aims to design a model-free adaptive tracking controller for discrete-time nonlinear systems with unknown dynamics and asymmetric control constraints. First, a new Q-function structure is designed by introducing the control input into the tracking error of the next moment, in order to eliminate the final tracking error, avoid the steady control, and ignore the discount factor. Second, via system transformation, a general performance index is developed to overcome the challenge caused by asymmetric constraints of implicit control inputs. By this operation, the constrained tracking problem is converted to an unconstrained optimal tracking problem without the traditional nonquadratic performance function that is only applicable to explicit control inputs. Then, a value-iteration-based Q-learning (VIQL) algorithm is derived to seek the optimal Q-function and the optimal control policy by using offline data rather than the mathematical model. Next, the convergence, monotonicity, and stability properties of VIQL are investigated to demonstrate that the iterative Q-function sequence can converge to the optimal Q-function under ideal conditions. To realize the VIQL algorithm, the critic neural network is employed to approximate the Q-function. Finally, simulation results and comparative experiments are conducted to demonstrate the validity and effectiveness of the present VIQL scheme.