Experimenting With A Multi-Approach Testing Strategy For Adaptive Systems

Context: Testing adaptive systems (ASs) is particularly challenging due to certain characteristics such as the high number of possible configurations, runtime adaptations and the interactions between the system and its surrounding environment. Therefore, the combination of different testing approaches in order to compose a strategy is expected to improve the quality of the designed test suites. Objective: To devise and experiment with a testing strategy for ASs that relies on particular characteristics of these systems. Method: We ranked testing approaches for ASs and devised a strategy that is composed of the three top-ranked ones. The rankings address the challenges that can be mitigated by the approaches, activities from a typical testing process, and characteristics observed in some AS implementations. The strategy was applied to two adaptive systems for mobile devices. Results: The approach was applied to both systems. We observed partial gains in terms of fault detection and structural coverage when results are analysed separately for each system, even though no improvements were obtained with the application of the third approach. Conclusion: The strategy, despite being incipient, is promising and motivates a deeper analysis of results and new experiment rounds. Furthermore, it can evolve as long as the rankings are updated with new approaches.