Auto-COP: Adaptation generation in Context-oriented Programming using Reinforcement Learning options

Context: Self-adaptive software systems continuously adapt in response to internal and external changes in their execution environment, captured as contexts. The Context-oriented Programming (COP) paradigm posits a technique for the development of self-adaptive systems, capturing their main characteristics with specialized programming language constructs. In COP, adaptations are specified as independent modules that are composed in and out of the base system as contexts are activated and deactivated in response to sensed circumstances from the surrounding environment. However, the definition of adaptations, their contexts and associated specialized behavior, need to be specified at design time. In complex cyber-physical systems this is intractable, if not impossible, due to new unpredicted operating conditions arising.Objective: In this paper, we propose Auto-COP, a new technique to enable generation of adaptations at run time. Auto-COP uses Reinforcement Learning (RL) options to build action sequences, based on the previous instances of the system execution (for example, atomic system actions enacted by human operators). Options are further explored in interaction with the environment, and the most suitable options for each context are used to generate the adaptations, exploiting COP abstractions.Method: To validate Auto-COP, we present two case studies exhibiting different system characteristics and application domains: a driving assistant and a robot delivery system. We present examples of Auto-COP to illustrate the types of circumstances (contexts) requiring adaptation at run time, and the corresponding generated adaptations for each context. Results: We confirm that the generated adaptations exhibit correct system behavior measured by domain specific performance metrics (e.g., conformance to specified speed limit), while reducing the number of required execution/actuation steps by a factor of two showing that the adaptations are regularly selected by the running system as adaptive behavior is more appropriate than the execution of atomic actions.Conclusion: Therefore, we demonstrate that Auto-COP is able to increase system adaptivity by enabling runtime generation of new adaptations for conditions detected at run time, while retaining the modularity offered by COP languages, and reducing the upfront specification required by system developers.