Towards more efficient meta-heuristic algorithms for combinatorial test generation

ESEC/SIGSOFT FSE, pp. 212-222, 2019.

Cited by: 1|Bibtex|Views57|Links
EI
Keywords:
combinatorial test generationinteraction testingscore computation methodreal worldsoftware systemMore(12+)
Weibo:
We propose a much more efficient method for score computation

Abstract:

Combinatorial interaction testing (CIT) is a popular approach to detecting faults in highly configurable software systems. The core task of CIT is to generate a small test suite called a t-way covering array (CA), where t is the covering strength. Many meta-heuristic algorithms have been proposed to solve the constrained covering array ge...More

Code:

Data:

0
Introduction
  • With the increasing requirement on customizable software, modern software systems usually have many configurable options.
  • The cost of development may be reduced significantly by reusing the systematic components among different variants
  • These benefits are in the wake of challenges for the validation of highly-configurable softwares, as failures may be caused by some combinations of options [18, 37].
  • As a response to this significant requirement, the combinatorial interaction testing (CIT) approach has emerged as a popular paradigm for detecting option-combination faults of configurable software system.
  • A recent research suggests that CIT is more desirable when system faults are hard to detect [34]
Highlights
  • With the increasing requirement on customizable software, modern software systems usually have many configurable options
  • As almost all meta-heuristic algorithms for constrained covering array generating (CCAG) are based on local search, we provide the basic concepts used in local search for solving CCAG
  • When the cutoff time is limited to 1 000 seconds, as can be seen from the tables, FastCA significantly outperforms other solvers on both the real-world and synthetic benchmarks
  • We study the importance of the lightweight score computation by comparing FastCA with its alternative FastCAp implemented with the previous score computation method on 26 real-world instances of 3-way CCAG. 15 instances are easy for both algorithms - they all find the same sized covering array (CA) in 10 seconds
  • A big issue in meta-heuristic algorithms for CCAG is the heavy time consumption of score computation, which accounts for the low speed of the algorithms, and hinders the applications of many good algorithmic ideas relying on scores
  • Experiments on a broad range of real-world and synthetic benchmarks show that FastCA is faster and finds better solutions than existing algorithms, which remains justified even when the cutoff time (100s) for FastCA is set to 10% of that (1000s) for the competitors
Methods
  • The authors conduct extensive experiments to evaluate the efficiency and effectiveness of FastCA by comparing it with 4 stateof-the-art algorithms for CCAG.
  • Since 2-way CCAG can be solved effectively by existing algorithms [36], the experiments focus on 3-way CCAG.
  • The authors investigate the performance of current state-of-the-art algorithms for solving 4-way CCAG, and compare that with the performance of FastCA.
  • RQ2: How does FastCA compare against current state-ofthe-art algorithms for 3-way CCAG?.
  • The authors set the time limit to 1 000 seconds and compare FastCA against its competitors
Results
  • The authors present the experimental results to answer each research question mentioned in Section 5.1.
  • For the Real-world and IBM benchmarks, the authors focus on the ‘difficult’ instances which need more than 100 seconds averaged runtime for all solvers excepting the greedy algorithm ACTS
  • On these instances, the size of CAs found by FastCA are usually much smaller than its competitors.
  • FastCA and its competitors for 4-way CCAG (RQ4): Table 7 presents the results of 4-way CCAG solving on the Real-world and IBM benchmarks; the results on the Synthetic benchmarks are available online8.
  • For many instances, such as spinv, Healthcare4 and Storage4, the sizes of CAs found by FastCA is several hundreds or even thousands smaller than those found by other solvers, indicating the superiority of FastCA on 4-way CCAG
Conclusion
  • Discussion on the results of

    FastCA for 2-way CCAG: For comprehensive evaluation, the authors summarize the empirical results for 2-way CCAG here.
  • A big issue in meta-heuristic algorithms for CCAG is the heavy time consumption of score computation, which accounts for the low speed of the algorithms, and hinders the applications of many good algorithmic ideas relying on scores.
  • This paper addressed this issue by proposing a lightweight method for score computation.
  • Experiments on a broad range of real-world and synthetic benchmarks show that FastCA is faster and finds better solutions than existing algorithms, which remains justified even when the cutoff time (100s) for FastCA is set to 10% of that (1000s) for the competitors
Summary
  • Introduction:

    With the increasing requirement on customizable software, modern software systems usually have many configurable options.
  • The cost of development may be reduced significantly by reusing the systematic components among different variants
  • These benefits are in the wake of challenges for the validation of highly-configurable softwares, as failures may be caused by some combinations of options [18, 37].
  • As a response to this significant requirement, the combinatorial interaction testing (CIT) approach has emerged as a popular paradigm for detecting option-combination faults of configurable software system.
  • A recent research suggests that CIT is more desirable when system faults are hard to detect [34]
  • Methods:

    The authors conduct extensive experiments to evaluate the efficiency and effectiveness of FastCA by comparing it with 4 stateof-the-art algorithms for CCAG.
  • Since 2-way CCAG can be solved effectively by existing algorithms [36], the experiments focus on 3-way CCAG.
  • The authors investigate the performance of current state-of-the-art algorithms for solving 4-way CCAG, and compare that with the performance of FastCA.
  • RQ2: How does FastCA compare against current state-ofthe-art algorithms for 3-way CCAG?.
  • The authors set the time limit to 1 000 seconds and compare FastCA against its competitors
  • Results:

    The authors present the experimental results to answer each research question mentioned in Section 5.1.
  • For the Real-world and IBM benchmarks, the authors focus on the ‘difficult’ instances which need more than 100 seconds averaged runtime for all solvers excepting the greedy algorithm ACTS
  • On these instances, the size of CAs found by FastCA are usually much smaller than its competitors.
  • FastCA and its competitors for 4-way CCAG (RQ4): Table 7 presents the results of 4-way CCAG solving on the Real-world and IBM benchmarks; the results on the Synthetic benchmarks are available online8.
  • For many instances, such as spinv, Healthcare4 and Storage4, the sizes of CAs found by FastCA is several hundreds or even thousands smaller than those found by other solvers, indicating the superiority of FastCA on 4-way CCAG
  • Conclusion:

    Discussion on the results of

    FastCA for 2-way CCAG: For comprehensive evaluation, the authors summarize the empirical results for 2-way CCAG here.
  • A big issue in meta-heuristic algorithms for CCAG is the heavy time consumption of score computation, which accounts for the low speed of the algorithms, and hinders the applications of many good algorithmic ideas relying on scores.
  • This paper addressed this issue by proposing a lightweight method for score computation.
  • Experiments on a broad range of real-world and synthetic benchmarks show that FastCA is faster and finds better solutions than existing algorithms, which remains justified even when the cutoff time (100s) for FastCA is set to 10% of that (1000s) for the competitors
Tables
  • Table1: The averaged time consumption (in seconds) of score computation on each benchmark, running TCA on each instance with time budget 1000 seconds
  • Table2: The averaged number of uncovered tuples and 1covered tuples over the procedure of TCA when solving 3way CCAG in 1 000 seconds
  • Table3: TCA+ vs TCA on the Real-world and IBM benchmarks
  • Table4: TCA+ vs TCA on the Synthetic benchmark
  • Table5: Comparing FastCA against state-of-the-art competitors for 3-way CCAG on the Real-world and IBM benchmarks. The running time is measured in CPU second. The cutoff time is set to 100s for FastCA, and 1000s for the other solvers
  • Table6: Comparing FastCA against state-of-the-art competitors for 3-way CCAG on the Synthetic benchmarks. The running time is measured in CPU second. The cutoff time is set to 1000s for all solvers
  • Table7: Comparing FastCA against state-of-the-art competitors for 4-way CCAG on the Real-world and IBM benchmarks. The running time is measured in CPU second. The cutoff time is set to 1000s for all solvers
Download tables as Excel
Related work
  • Practical algorithms for solving CCAG can be roughly categorized into three main groups: constraint-encoding algorithms, greedy algorithms and meta-heuristic algorithms. Constraint-encoding algorithms focus on efficient methods to encode CCAG into constrained optimization problem, such as SAT and MaxSat problems [1, 36]. While constraint-encoding algorithms are effective for solving 2way CCAG, it required improvements to handle 3-way CCAG [36].

    Greedy algorithms can usually generate CAs in a short time. Although the size of CAs is not necessarily small, greedy algorithms show its superior in some scenarios where highly optimized test suite is not the primary consideration. One-test-at-a-time (OTAT ) and in-parameter-order (IPO) are two main approaches of greedy algorithms. The well-known AETG algorithm is the first one using the OTAT strategy [4], and since then a number variants were proposed to improve its performance [2, 3, 7, 33, 35, 40]. The IOP
Funding
  • This work was supported by Youth Innovation Promotion Association, Chinese Academy of Sciences (No 2017150)
Reference
  • Mutsunori Banbara, Haruki Matsunaka, Naoyuki Tamura, and Katsumi Inoue.
    Google ScholarFindings
  • 2010. Generating Combinatorial Test Cases by Efficient SAT Encodings Suitable for CDCL SAT Solvers. In Proceedings of LPAR 2010. 112–126.
    Google ScholarLocate open access versionFindings
  • [2] Renée C. Bryce and Charles J. Colbourn. 2007. The density algorithm for pairwise interaction testing. Software Testing, Verification and Reliability 17, 3 (2007), 159– 182.
    Google ScholarLocate open access versionFindings
  • [3] Renée C. Bryce, Charles J. Colbourn, and Myra B. Cohen. 2005. A framework of greedy methods for constructing interaction test suites. In Proceedings of ICSE 2005. 146–155.
    Google ScholarLocate open access versionFindings
  • [4] David M. Cohen, Siddhartha R. Dalal, Michael L. Fredman, and Gardner C. Patton.
    Google ScholarFindings
  • 1997. The AETG System: An Approach to Testing Based on Combinatiorial Design. IEEE Transactions on Software Engineering 23, 7 (1997), 437–444.
    Google ScholarLocate open access versionFindings
  • [5] Myra B. Cohen, Matthew B. Dwyer, and Jiangfan Shi. 200Interaction testing of highly-configurable systems in the presence of constraints. In Proceedings of ISSTA 2007. 129–139.
    Google ScholarLocate open access versionFindings
  • [6] Myra B. Cohen, Matthew B. Dwyer, and Jiangfan Shi. 200Constructing Interaction Test Suites for Highly-Configurable Systems in the Presence of Constraints: A Greedy Approach. IEEE Transactions on Software Engineering 34, 5 (2008), 633–650.
    Google ScholarLocate open access versionFindings
  • [7] Jacek Czerwonka. 2008. Pairwise testing in the real world: Practical extensions to test-case scenarios. Microsoft Corporation, Software Testing Technical Articles (2008).
    Google ScholarLocate open access versionFindings
  • [8] Philippe Galinier, Segla Kpodjedo, and Giulio Antoniol. 2017. A penalty-based Tabu search for constrained covering arrays. In Proceedings of GECCO 2017. 1288–1294.
    Google ScholarLocate open access versionFindings
  • [9] Brady J. Garvin, Myra B. Cohen, and Matthew B. Dwyer. 2009. An Improved Meta-Heuristic Search for Constrained Interaction Testing. In Proceedings of 1st International Symposium on Search Based Software Engineering. 13–22.
    Google ScholarLocate open access versionFindings
  • [10] Brady J. Garvin, Myra B. Cohen, and Matthew B. Dwyer. 2011. Evaluating improvements to a meta-heuristic search for constrained interaction testing. Empirical Software Engineering 16, 1 (2011), 61–102.
    Google ScholarLocate open access versionFindings
  • [11] Jon D. Hagar, Thomas L. Wissink, D. Richard Kuhn, and Raghu Kacker. 2015. Introducing Combinatorial Testing in a Large Organization. IEEE Computer 48, 4 (2015), 64–72. https://doi.org/10.1109/MC.2015.114
    Locate open access versionFindings
  • [12] Mark Harman. 2007. The Current State and Future of Search Based Software Engineering. In Proceedings of FOSE 2007. 342–357.
    Google ScholarLocate open access versionFindings
  • [13] Mark Harman, Stephen Swift, and Kiarash Mahdavi. 2005. An empirical study of the robustness of two module clustering fitness functions. In Proceedings of GECCO 2005. 1029–1036.
    Google ScholarLocate open access versionFindings
  • [14] Mark Harman and Laurence Tratt. 2007. Pareto optimal search based refactoring at the design level. In Proceedings of GECCO 2007. 1106–1113.
    Google ScholarLocate open access versionFindings
  • [15] Christopher Henard, Mike Papadakis, Gilles Perrouin, Jacques Klein, Patrick Heymans, and Yves Le Traon. 2014. Bypassing the Combinatorial Explosion: Using Similarity to Generate and Prioritize T-Wise Test Configurations for Software Product Lines. IEEE Trans. Software Eng. 40, 7 (2014), 650–670. https://doi.org/10.1109/TSE.2014.2327020
    Locate open access versionFindings
  • [16] Yue Jia, Myra B. Cohen, Mark Harman, and Justyna Petke. 2015. Learning Combinatorial Interaction Test Generation Strategies Using Hyperheuristic Search. In Proceedings of ICSE 2015. 540–550.
    Google ScholarLocate open access versionFindings
  • [17] D. Richard Kuhn and Vadim Okun. 2006. Pseudo-Exhaustive Testing for Software. In Proceedings of SEW 2006. 153–158.
    Google ScholarLocate open access versionFindings
  • [18] D. Richard Kuhn, Dolores R. Wallace, and Albert M. Gallo. 2004. Software Fault Interactions and Implications for Software Testing. IEEE Transactions on Software Engineering 30, 6 (2004), 418–421.
    Google ScholarLocate open access versionFindings
  • [19] Yu Lei, Richard H. Carver, Raghu Kacker, and David Chenho Kung. 2007. A combinatorial testing strategy for concurrent programs. Softw. Test., Verif. Reliab. 17, 4 (2007), 207–225. https://doi.org/10.1002/stvr.369
    Locate open access versionFindings
  • [20] Yu Lei, Raghu Kacker, D. Richard Kuhn, Vadim Okun, and James Lawrence. 2007. IPOG: A General Strategy for T-Way Software Testing. In Proceedings of ECBS 2007. 549–556.
    Google ScholarLocate open access versionFindings
  • [21] Yu Lei, Raghu Kacker, D. Richard Kuhn, Vadim Okun, and James Lawrence. 2008. IPOG/IPOG-D: efficient test generation for multi-way combinatorial testing. Software Testing, Verification and Reliability 18, 3 (2008), 125–148.
    Google ScholarLocate open access versionFindings
  • [22] Yu Lei and Kuo-Chung Tai. 1998. In-Parameter-Order: A Test Generation Strategy for Pairwise Testing. In Proceedings of HASE 1998. 254–261.
    Google ScholarLocate open access versionFindings
  • [23] Chu Min Li and Wenqi Huang. 2005. Diversification and Determinism in Local Search for Satisfiability. In Proceedings of SAT 2005. 158–172.
    Google ScholarLocate open access versionFindings
  • [24] Zheng Li, Mark Harman, and Robert M. Hierons. 2007. Search Algorithms for Regression Test Case Prioritization. IEEE Transactions on Software Engineering 33, 4 (2007), 225–237.
    Google ScholarLocate open access versionFindings
  • [25] Jinkun Lin, Chuan Luo, Shaowei Cai, Kaile Su, Dan Hao, and Lu Zhang. 2015. TCA: An Efficient Two-Mode Meta-Heuristic Algorithm for Combinatorial Test Generation. In Proceedings of ASE 2015. 494–505.
    Google ScholarLocate open access versionFindings
  • [26] Chuan Luo, Shaowei Cai, Wei Wu, Zhong Jie, and Kaile Su. 2015. CCLS: An Efficient Local Search Algorithm for Weighted Maximum Satisfiability. IEEE Trans. Comput. 64, 7 (2015), 1830–1843.
    Google ScholarLocate open access versionFindings
  • [27] Phil McMinn. 2004. Search-based software test data generation: a survey. Software Testing, Verification and Reliability 14, 2 (2004), 105–156.
    Google ScholarFindings
  • [28] Hanefi Mercan, Cemal Yilmaz, and Kamer Kaya. 2018. CHiP: A Configurable Hybrid Parallel Covering Array Constructor. IEEE Transactions on Software Engineering (2018), To appear.
    Google ScholarLocate open access versionFindings
  • [29] Kari J. Nurmela. 2004. Upper bounds for covering arrays by tabu search. Discrete Applied Mathematics 138, 1-2 (2004), 143–152.
    Google ScholarLocate open access versionFindings
  • [30] Olivier Roussel. 2011. Controlling a Solver Execution with the runsolver Tool. Journal on Satisfiability, Boolean Modeling and Computation 7, 4 (2011), 139–144.
    Google ScholarLocate open access versionFindings
  • [31] Itai Segall, Rachel Tzoref-Brill, and Eitan Farchi. 2011. Using binary decision diagrams for combinatorial test design. In Proceedings of ISSTA 2011. 254–264.
    Google ScholarLocate open access versionFindings
  • [32] Ole Tange. 2018. GNU Parallel 2018. Ole Tange. https://doi.org/10.5281/zenodo.1146014
    Findings
  • [33] Yu-Wen Tung and Wafa S Aldiwan. 2000. Automating test case generation for the new generation mission software system. In Proceedings of IEEE Aerospace Conference 2000, Vol. 1. IEEE, 431–437.
    Google ScholarLocate open access versionFindings
  • [34] Huayao Wu, Justyna Petke, Yue Jia, Mark Harman, et al. 2018. An Empirical Comparison of Combinatorial Testing, Random Testing and Adaptive Random Testing. IEEE Transactions on Software Engineering (2018).
    Google ScholarLocate open access versionFindings
  • [35] Akihisa Yamada, Armin Biere, Cyrille Artho, Takashi Kitamura, and Eun-Hye Choi. 2016. Greedy combinatorial test case generation using unsatisfiable cores. In Proceedings of ASE 2016. 614–624.
    Google ScholarLocate open access versionFindings
  • [36] Akihisa Yamada, Takashi Kitamura, Cyrille Artho, Eun-Hye Choi, Yutaka Oiwa, and Armin Biere. 2015. Optimization of Combinatorial Testing by Incremental SAT Solving. In Proceedings of ICST 2015. 1–10.
    Google ScholarLocate open access versionFindings
  • [37] Cemal Yilmaz, Myra B. Cohen, and Adam A. Porter. 2006. Covering Arrays for Efficient Fault Characterization in Complex Configuration Spaces. IEEE Transactions on Software Engineering 32, 1 (2006), 20–34.
    Google ScholarLocate open access versionFindings
  • [38] Linbin Yu, Yu Lei, Mehra Nouroz Borazjany, Raghu Kacker, and D. Richard Kuhn.
    Google ScholarFindings
  • 2013. An Efficient Algorithm for Constraint Handling in Combinatorial Test Generation. In Sixth IEEE International Conference on Software Testing, Verification and Validation, ICST 2013, Luxembourg, Luxembourg, March 18-22, 2013. 242–251.
    Google ScholarLocate open access versionFindings
  • [39] Xun Yuan, Myra B. Cohen, and Atif M. Memon. 2011. GUI Interaction Testing: Incorporating Event Context. IEEE Trans. Software Eng. 37, 4 (2011), 559–574. https://doi.org/10.1109/TSE.2010.50
    Locate open access versionFindings
  • [40] Zhiqiang Zhang, Jun Yan, Yong Zhao, and Jian Zhang. 2014. Generating combinatorial test suite using combinatorial optimization. Journal of Systems and Software 98 (2014), 191–207.
    Google ScholarLocate open access versionFindings
Your rating :
0

 

Tags
Comments