Testing Requirements via User and System Stories in Agent Systems.

Agile software development is a popular and widely adopted practice due to its flexible and iterative nature that facilitates rapid prototyping. Recent work presented an agile approach to capturing requirements in agent systems via user and system stories. User and system stories present the requirements from the user and system perspective, respectively. Each story contains a set of acceptance criteria, which are a set of statements that identify the conditions under which the system behaviour can be accepted by the users or stakeholders. In this paper, we present a novel approach to testing the requirements that are specified via User and System stories in an agent system. We do this by developing a systematic approach to validating the execution traces output by the system against the specified acceptance criteria for each story. The approach identifies acceptance criteria that are met successfully in execution and those that fail. We present a fault model that categorizes the failures providing insight to the developers to address the failed cases. We classify three kinds of faults for a given acceptance criterion: (a) the trigger condition is never met; (b) when the trigger occurs the preconditions are not met; or (c) the trigger and preconditions are met but the resulting actions are not as expected. The motivating application of our work, which is also the test-bed for evaluation, is an agent-based simulation application for modelling the behaviours of civilians in a bushfire emergency scenario that is used in practice. We show our approach is able to successfully test and uncover requirements that were not met in this application.