Framework for Complex SoS Emergent Behavior Evolution Using Deep Reinforcement Learning

Abstract Advances in technology have made it easy to integrate multiple modern systems to form complex system‐of‐systems (SoS) to achieve unparalleled levels of functionality that are otherwise not achievable by the constituent systems in isolation. In fact, with the recent explosion of machine learning techniques to build autonomous systems such as drones and self‐driving cars, there is a pressing need to ensure that they collaboratively and safely operate in an SoS context. However, in general, the characteristic emergent behaviors of complex SoS – that directly impact its operational measures of success or Measures of Effectiveness (MOEs) – is very difficult, if not impossible, to manually explore, anticipate, and arbitrate just from knowledge of its underlying systems. Further, there are multiple scenarios of evolution in such complex SoS, including evolution in the emergent behavior of the SoS. The continuous, continual, and evolving nature of the SoS and constituent system environment's state and possible actions, adds further complexity. In this paper, we present a novel approach that leverages Reinforcement Learning, a machine learning approach, to inculcate adaptable intelligence in constituent systems to adapt their behaviors in tandem with the evolution of emergent behavior at the SoS level. By augmenting the reward mechanism of RL by leveraging SoS‐Constituent System MOE Relationship, that relates and ranks System MOEs vs. SoS MOEs, we inculcate an Intelligent‐Behavior Evolution Agent, with the necessary constraints to learn to maximize the SoS and system‐level MOEs, while adapting itself to the evolution in SoS. We illustrate our approach and demonstrate its feasibility and potential by applying it to a power grid SoS case example. The effectiveness and performance of the approach are quantified.