Modeling an Anomaly-Based Intrusion Prevention System Using Game Theory.

In Cloud Computing environment, the availability, authentication and integrity became a more challenging problem. Indeed, the classical solutions of security based on intrusion detection system and firewalls are easily bypassed by experienced attackers. In addition, the use of different technologies in term of security didn't mitigate the attack considerably. To achieve network system's security with the complexity and the diversity of attack types is too difficult and costly. However, to make them more resistant to attacks, anomaly-based Intrusion Prevention System (IPS) are used. Such systems take into consideration the probability of legitimacy of a packet if it didn't match any signature of malicious packets. In this paper, a competitive normal form game is developed based on the probability of packets' legitimacy and the trust that an IPS has over the owner of the packet. Furthermore, a decision is made about dropping, accepting or testing packet in the network, and different Nash Equilibriums are calculated based on the system's parameters. Our approach demonstrated its feasibility in term of prediction of the cases in which the system could be compromised and the actions that should be performed in case of an intrusion.