Propagating malicious codes: theory and experiments

Propagating malicious codes (malcodes) such as worms and bots are presenting one of the most serious threats for the security research community today. Every year, along with other malwares, they are estimated to cause more than $10 billions of damage [1].Despite a large amount of research that deals with existing malcodes, there have been very few studies on hypothetical, yet enormously destructive and powerful ones. A solid understanding of such malcodes, meanwhile, is fruitful in many aspects: it allows us to understand the limit (or potential) of malcodes that current forms have yet to reach; provides a benchmark to evaluate the effectiveness of existing defense systems; and gives some insight into developing the next generation defense systems.This dissertation aims to fill the aforementioned gap by studying several classes of hypothetical malcodes: In the first part of the dissertation, we studied in depth a class of super-fast and resilient worms. Specifically, we studied the problem of optimizing worm performance in a rigorous theoretical framework. Two optimization problems were formulated: (a) Minimum Time Malicious Propagation-MTMP to minimize infection time of a given population; (b) Maximum Expansion Malicious Propagation-MEMP to maximize the population to be infected within a given time. We showed that MTMP is NP-hard under general conditions but solvable in a relaxed context. Compared to Flashworm, the fastest worm in an earlier research, our worm is shown using both simulation and analytical analysis to offer: (a) much shorter infection time under same bandwidth conditions; (b) much less bandwidth to achieve the same infection time. Next we focused on designing infection topologies that are resilient to node failure. We first showed the optimally resilient topology, yet this topology suffers from inefficient infection time. We thus proposed a structure that strikes a good balance between resiliency and infection time. When incorporating extractors, our structure can achieve resiliency with near absolute certainty.In the second part of the dissertation, we continued to explore the application of extractors to designing resilient botnets. Due to their simplicity and ease of implementation/deployment, command and control (C&C) botnets were the first and most popular type of botnets. However, C&C structures are prone to C&C removal, considered the Achilles heel for such botnets. We showed that the effect of C&C removal could be reduced exponentially under a simple approach and can be further improved using our extractor-based construction. This gives rise to a new class of botnets called CRESTBOT, which employ this structure. Our analysis showed that under packet lossless condition, UDP CRESTBOT can deliver messages three times faster than the traditional TCP botnets. This result is verified through our extensive experiments on Emulab, a realistic testbed. The experiments also showed that under packet lossy conditions, UDP CRESTBOT could achieve message passing time gain of more than three times. In the third and final part of this dissertation, we turned our attention to P2P botnets. While P2P traffic offers an excellent channel for bots to hide their communications, current P2P protocols do not enforce security mechanisms. This creates a promising avenue for researchers to exploit known P2P vulnerabilities, to monitor or disrupt bot communications. Efficient deployment of such approaches, however, requires a good understanding of P2P structures and traffic flows on the P2P network. Towards this goal, we first developed a new simulation testbed that combined distributed simulation techniques and a real P2P software client (Kademlia's). Using this testbed, we recreated a Kademlia network and performed detailed structural analysis of this network. We also measured the effectiveness of a select set of well-known centrality measures. Our results showed all but betweenness performed poorly regarding efficient traffic monitoring. They also revealed the dynamic and dependency of bot traffic on bot locations, rendering any monitoring scheme ineffective. When bot locations are fixed, we formulated an optimization problem of minimum sensor deployment called Bot Communication Monitoring–BCM and proved its hardness. Finally, we showed that of several known P2P attacks: command poisoning, sybil and bot eclipsing, only the first is effective.Our findings presented in this dissertation show that: (1) Malcodes have yet to reach their most powerful and resilient forms, at which disruption is very costly; (2) Early detection and intervention are crucial in preventing them from doing so; (3) Some existing trend of malcodes evolution, in fact, does not offer a panacea for malcodes, contrary to public perception.