BenchPress: Analyzing Android App Vulnerability Benchmark Suites

In recent years, various benchmark suites have been developed to evaluate the efficacy of Android security analysis tools. Tool developers often choose such suites based on the availability and popularity of suites and not on their characteristics and relevance due to the lack of information about them. In this context, based on a recent effort, we empirically evaluated four Android-specific benchmark suites: DroidBench, Ghera, ICCBench, and UBCBench. For each benchmark suite, we identified the APIs used by the suite that were discussed on Stack Overflow in the context of Android app development and measured the usage of these APIs in a sample of 227K real-world apps (coverage). We also identified security-related APIs used in real-world apps but not in any of the above benchmark suites to assess the opportunities to extend benchmark suites (gaps).