AoI-minimal Power Adjustment in RF-EH-powered Industrial IoT Networks: A Soft Actor-Critic-Based Method

This paper investigates the radio-frequency-energy-harvesting-powered (RF-EH-powered) wireless Industrial Internet of Things (IIoT) networks, where multiple sensor nodes (SNs) are first powered by a wireless power station (WPS), and then collect status updates from the industrial environment and finally transmit the collected data to the monitor with their harvested energy. To enhance the timeliness of data, age of information (AoI) is used as a metric to optimize the system. Particularly, an expected sum AoI (ESA) minimization problem is formulated by optimizing the power adjustment policy for the SNs under multiple practical constraints, including the EH, the minimal signal-to-noise-plus-interference ratio (SINR) and the battery capacity constraints. To solve the non-convex problem with no explicit AoI expression, we transform it into a Markov decision problem (MDP) with continuous state space and action space. Then, inspired by the Soft Actor-Critic (SAC) framework in deep reinforcement learning, a SAC-based age-aware power adjustment (SAPA) method is proposed by modeling the power adjustment as a stochastic strategy. Furthermore, to reduce the communication overhead of SAPA, a multi-agent version of SAPA, i.e., MSAPA, is proposed, with which each SN is able to adjust its transmit power based on its local observations. The communication overhead of SAPA and MSAPA is also analyzed theoretically. Simulation results show that the proposed SAPA and MSAPA converge well with different numbers of SNs. It is also shown that the ESA achieved by the proposed SAPA and MSAPA is lower than that achieved by the baseline methods.