Taming Timeout Flakiness: An Empirical Study of SAP HANA
CoRR(2024)
摘要
Regression testing aims to prevent code changes from breaking existing
features. Flaky tests negatively affect regression testing because they result
in test failures that are not necessarily caused by code changes, thus
providing an ambiguous signal. Test timeouts are one contributing factor to
such flaky test failures. With the goal of reducing test flakiness in SAP HANA,
we empirically study the impact of test timeouts on flakiness in system tests.
We evaluate different approaches to automatically adjust timeout values,
assessing their suitability for reducing execution time costs and improving
build turnaround times. We collect metadata on SAP HANA's test executions by
repeatedly executing tests on the same code revision over a period of six
months. We analyze the test flakiness rate, investigate the evolution of test
timeout values, and evaluate different approaches for optimizing timeout
values. The test flakiness rate ranges from 49
of repeated test executions. Test timeouts account for 70
failures. Developers typically react to flaky timeouts by manually increasing
timeout values or splitting long-running tests. However, manually adjusting
timeout values is a tedious task. Our approach for timeout optimization reduces
timeout-related flaky failures by 80
value by 25
major contributing factor to flakiness in system tests. It is challenging for
developers to effectively mitigate this problem manually. Our technique for
optimizing timeout values reduces flaky failures while minimizing test costs.
Practitioners working on large-scale industrial software systems can use our
findings to increase the effectiveness of their system tests while reducing the
burden on developers to manually maintain appropriate timeout values.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要