AI helps you reading Science

AI generates interpretation videos

AI extracts and analyses the key points of the paper to generate videos automatically


pub
Go Generating

AI Traceability

AI parses the academic lineage of this thesis


Master Reading Tree
Generate MRT

AI Insight

AI extracts a summary of this paper


Weibo:
We find that test suites prioritized by coverage-based strategies are better than those from other strategies in terms of the effectiveness of fault localization

How Well Do Test Case Prioritization Techniques Support Statistical Fault Localization

COMPSAC (1), pp.99-106, (2009)

Cited by: 79|Views128
EI

Abstract

In continuous integration, a tight integration of test case prioritization techniques and fault-localization techniques may both expose failures faster and locate faults more effectively. Statistical fault-localization techniques use the execution information collected during testing to locate faults. Executing a small fraction of a prior...More

Code:

Data:

0
Introduction
  • Continuous Integration (CI) [11] refers to a software process, in which developers integrate their software artifacts with a CI agent frequently, such as many times a day.
  • Multiple developers may submit their artifacts to the CI agent at various chosen times, resulting in one or more integrations within each period.
  • Multiple developers may concurrently submit their code and every developer expects the CI agent to run the regression test suite of the same baseline version to verify their individual submissions.
  • The overall integration process is heavily loaded.
  • It is, necessary to optimize this activity
Highlights
  • Continuous Integration (CI) [11] refers to a software process, in which developers integrate their software artifacts with a CI agent frequently, such as many times a day
  • The main contribution of the paper is threefold. (i) We report the first empirical study to evaluate the impact of test case prioritization techniques on statistical fault-localization techniques. (ii) The empirical results interestingly show that the coverage-based test case prioritization strategy is less sensitive than the other studied strategies in supporting such integration
  • Combined with the result in Section 2), we find that additional statement (AS) can be promising in providing effective test case prioritization as well as supporting statistical fault localization
  • We conduct an empirical study to explore the impact of test case prioritization on statistical fault localization
  • We find that test suites prioritized by coverage-based strategies are better than those from other strategies in terms of the effectiveness of fault localization
  • The empirical results of Tarantula show that, for the AS prioritization techniques, developers only need to examine 7% more code to locate the fault when using the topmost 50% of a test suite
  • Random ordering can be less effective than the additional statement technique, no other technique can outperform random ordering
Results
  • The authors followed [9] to remove the versions whose faults are too evident.
  • The empirical results of Tarantula show that, for the AS prioritization techniques, developers only need to examine 7% more code to locate the fault when using the topmost 50% of a test suite.
  • The developer would need to examine 10% more code if another 20% of test suite is not used in a commit
Conclusion
  • The total time allowed for testing and fault localization is limited.
  • The authors conduct an empirical study to explore the impact of test case prioritization on statistical fault localization.
  • The authors find that test suites prioritized by coverage-based strategies are better than those from other strategies in terms of the effectiveness of fault localization.
  • The authors' result provides a strong piece of evidence to clear the misconception on random prioritization ⎯ random ordering can be effective in supporting such integration
Tables
  • Table1: PRIORITIZATION TECHNIQUES
  • Table2: SUBJECT PROGRAMS
  • Table3: MEAN RELATIVE EXPENSE
  • Table4: MEAN EFFECTIVENESS RELATIVE TO RANDOM ORDERING
Download tables as Excel
Related work
  • This section reviews related work that has not been discussed in previous sections.

    Wong et al [28] proposed an approach that combines test suite minimization and prioritization to select cases according to the cost per additional coverage. Srivastava et al [26] developed a binary matching technique to calculate the changes in program at the basic block level and prioritize test cases to optimally cover the affected program changes. Walcott et al [27] studied a time-aware test suite prioritization technique based on genetic algorithms to order test cases under testing time constraints. Li et al [20] evaluated various search algorithms for test case prioritization.

    Apart from distribution-based techniques, Leon and colleagues [3][18][19] also proposed a family of failurepursuit sampling techniques. They select one initial sample per cluster and, when a failure is found, k nearest neighbors are selected and checked. If additional failures are found, the process will be repeated. We do not evaluate this technique because it requires the outcomes of test cases on the modified version and is, therefore, not suitable for continuous integration, in which fast turnaround is required.
Funding
  • * This research is supported in part by the General Research Fund of the Research Grants Council of Hong Kong (project nos. 716507 and 717308) and a Discovery Grant of the Australian Research Council
Reference
  • R. Abreu, P. Zoeteweij, and A. J. C. van Gemund. On the accuracy of spectrum-based fault localization. In Proceedings of the Testing: Academic and Industrial Conference: Practice And Research Techniques (TAICPART-MUTATION 2007), pages 89–98. IEEE Computer Society Press, Los Alamitos, CA, 2007.
    Google ScholarLocate open access versionFindings
  • H. Cleve and A. Zeller. Locating causes of program failures. In Proceedings of the 27th International Conference on Software Engineering (ICSE 2005), pages 342–351. ACM Press, New York, NY, 2005.
    Google ScholarLocate open access versionFindings
  • W. Dickinson, D. Leon, and A. Podgurski. Pursuing failure: the distribution of program failures in a profile space. In Proceedings of the Joint 8th European Software Engineering Conference and 9th ACM SIGSOFT International Symposium on Foundation of Software Engineering (ESEC 2001/ FSE-9), pages 246–255. ACM Press, New York, NY, 2001.
    Google ScholarLocate open access versionFindings
  • H. Do, S. G. Elbaum, and G. Rothermel. Supporting controlled experimentation with testing techniques: an infrastructure and its potential impact. Empirical Software Engineering, 10 (4): 405–435, 2005.
    Google ScholarLocate open access versionFindings
  • H. Do, G. Rothermel, and A. Kinneer. Prioritizing JUnit test cases: an empirical assessment and cost-benefits analysis. Empirical Software Engineering, 11: 33–70, 2006.
    Google ScholarLocate open access versionFindings
  • P. M. Duvall, S. Matyas, and A. Glover. Continuous Integration: Improving Software Quality and Reducing Risk. Addison Wesley, Upper Saddle River, NJ, 2007.
    Google ScholarFindings
  • S. G. Elbaum, A. G. Malishevsky, and G. Rothermel. Prioritizing test cases for regression testing. ACM SIGSOFT Software Engineering Notes, 25 (5): 102–112, 2000.
    Google ScholarLocate open access versionFindings
  • S. G. Elbaum, A. G. Malishevsky, and G. Rothermel. Test case prioritization: a family of empirical studies. IEEE Transactions on Software Engineering, 28 (2): 159–182, 2002.
    Google ScholarLocate open access versionFindings
  • S. G. Elbaum, G. Rothermel, S. Kanduri, and A. G. Malishevsky. Selecting a cost-effective test case prioritization technique. Software Quality Control, 12 (3): 185–210, 2004.
    Google ScholarLocate open access versionFindings
  • D. Farley. The Development Pipeline. http://studios.thoughtworks.com/assets/2007/5/11/The-DeploymentPipeline-by-Dave-Farley-2007.pdf. Last accessed 2008.
    Findings
  • M. Fowler. Continuous Integration. http://martinfowler.com/articles/continuousIntegration.html. Last accessed 2008.
    Findings
  • A. K. Jain and R. C. Dubes. Algorithms for Clustering Data. Prentice-Hall, Upper Saddle River, NJ, 1988.
    Google ScholarFindings
  • D. Jeffrey, N. Gupta, and R. Gupta. Fault localization using value replacement. In Proceedings of the 2008 ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2008), pages 167–178. ACM Press, New York, NY, 2008.
    Google ScholarLocate open access versionFindings
  • J. A. Jones and M. J. Harrold. Empirical evaluation of the Tarantula automatic fault-localization technique. In Proceedings of the 20th IEEE/ACM International Conference on Automated Software Engineering (ASE 2005), pages 273–282. ACM Press, New York, NY, 2005.
    Google ScholarLocate open access versionFindings
  • J. A. Jones, M. J. Harrold, and J. F. Bowring. Debugging in parallel. In Proceedings of the 2007 ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2007), pages 16–26. ACM Press, New York, NY, 2007.
    Google ScholarLocate open access versionFindings
  • J. A. Jones, M. J. Harrold, and J. Stasko. Visualization of test information to assist fault localization. In Proceedings of the 24th International Conference on Software Engineering (ICSE 2002), pages 467–477. ACM Press, New York, NY, 2002.
    Google ScholarLocate open access versionFindings
  • J.-M. Kim and A. Porter. A history-based test prioritization technique for regression testing in resource constrained environments. In Proceedings of the 24th International Conference on Software Engineering (ICSE 2002), pages 119– 129. ACM Press, New York, NY, 2002.
    Google ScholarLocate open access versionFindings
  • D. Leon, W. Masri, and A. Podgurski. An empirical evaluation of test case filtering techniques based on exercising complex information flows. In Proceedings of the 27th International Conference on Software Engineering (ICSE 2005), pages 412–421. ACM Press, New York, NY, 2005.
    Google ScholarLocate open access versionFindings
  • D. Leon and A. Podgurski. A comparison of coverage-based and distribution-based techniques for filtering and prioritizing test cases. In Proceedings of the 14th International Symposium on Software Reliability Engineering (ISSRE 2003), pages 442–453. IEEE Computer Society Press, Los Alamitos, CA, 2003.
    Google ScholarLocate open access versionFindings
  • Z. Li, M. Harman, and R. M. Hierons. Search algorithms for regression test case prioritization. IEEE Transactions on Software Engineering, 33 (4): 225–237, 2007.
    Google ScholarLocate open access versionFindings
  • B. Liblit, M. Naik, A. X. Zheng, A. Aiken, and M. I. Jordan. Scalable statistical bug isolation. In Proceedings of the 2005 ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI 2005), pages 15–26. ACM Press, New York, NY, 2005.
    Google ScholarLocate open access versionFindings
  • C. Liu, X. Yan, L. Fei, J. Han, and S. P. Midkiff. SOBER: statistical model-based bug localization. In Proceedings of the Joint 10th European Software Engineering Conference and 13th ACM SIGSOFT International Symposium on Foundation of Software Engineering (ESEC 2005/FSE-13), pages 286–295. ACM Press, New York, NY, 2005.
    Google ScholarLocate open access versionFindings
  • M. Renieris and S. P. Reiss. Fault localization with nearest neighbor queries. In Proceedings of the 18th IEEE International Conference on Automated Software Engineering (ASE 2003), pages 30–39. IEEE Computer Society Press, Los Alamitos, CA, 2003.
    Google ScholarLocate open access versionFindings
  • G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold. Test case prioritization: an empirical study. In Proceedings of the 15th IEEE International Conference on Software Maintenance (ICSM ’99), pages 179–188. IEEE Computer Society Press, Los Alamitos, CA, 1999.
    Google ScholarLocate open access versionFindings
  • G. Rothermel, R. H. Untch, C. Chu, and M. J. Harrold. Prioritizing test cases for regression testing. IEEE Transactions on Software Engineering, 27 (10): 929–948, 2001.
    Google ScholarLocate open access versionFindings
  • A. Srivastava and J. Thiagarajan. Effectively prioritizing tests in development environment. In Proceedings of the 2002 ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2002), pages 97–106. ACM Press, New York, NY, 2002.
    Google ScholarLocate open access versionFindings
  • K. R. Walcott, M. L. Soffa, G. M. Kapfhammer, and R. S. Roos. TimeAware test suite prioritization. In Proceedings of the 2006 ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2006), pages 1–12. ACM Press, New York, NY, 2006.
    Google ScholarLocate open access versionFindings
  • W. E. Wong, J. R. Horgan, S. London, and H. Agrawal. A study of effective regression testing in practice. In Proceedings of the 8th International Symposium on Software Reliability Engineering (ISSRE ’97), pages 264–274. IEEE Computer Society Press, Los Alamitos, CA, 1997.
    Google ScholarLocate open access versionFindings
  • Y. Yu, J. A. Jones, and M. J. Harrold. An empirical study of the effects of test-suite reduction on fault localization. In Proceedings of the 30th International Conference on Software Engineering (ICSE 2008), pages 201–210. ACM Press, New York, NY, 2008.
    Google ScholarLocate open access versionFindings
Your rating :
0

 

Tags
Comments
数据免责声明
页面数据均来自互联网公开来源、合作出版商和通过AI技术自动分析结果,我们不对页面数据的有效性、准确性、正确性、可靠性、完整性和及时性做出任何承诺和保证。若有疑问,可以通过电子邮件方式联系我们:report@aminer.cn
小科