A Scalable t-Wise Coverage Estimator: Algorithms and Applications
Article Type
Research Article
Publication Title
IEEE Transactions on Software Engineering
Abstract
Owing to the pervasiveness of software in our modern lives, software systems have evolved to be highly configurable. Combinatorial testing has emerged as a dominant paradigm for testing highly configurable systems. Often constraints are employed to define the environments where a given system is expected to work. Therefore, there has been a sustained interest in designing constraint-based test suite generation techniques. A significant goal of test suite generation techniques is to achieve t-wise coverage for higher values of t. Therefore, designing scalable techniques that can estimate t-wise coverage for a given set of tests and/or the estimation of maximum achievable t-wise coverage under a given set of constraints is of crucial importance. The existing estimation techniques face significant scalability hurdles. We designed scalable algorithms with mathematical guarantees to estimate (i) t-wise coverage for a given set of tests, and (ii) maximum t-wise coverage for a given set of constraints. In particular, ApproxCov takes in a test set U and returns an estimate of the t-wise coverage of U that is guaranteed to be within (1±ϵ)-factor of the ground truth with probability at least 1-δ for a given tolerance parameter ϵ and a confidence parameter δ. A scalable framework ApproxMaxCov for a given formula F outputs an approximation which is guaranteed to be within (1±ϵ) factor of the maximum achievable t-wise coverage under F, with probability ≥1-δ for a given tolerance parameter ϵ and a confidence parameter δ. Our comprehensive evaluation demonstrates that ApproxCov and ApproxMaxCov can handle benchmarks that are beyond the reach of current state-of-the-art approaches. In this paper we present proofs of correctness of ApproxCov, ApproxMaxCov, and of their generalizations. We show how the algorithms can improve the scalability of a test suite generator while maintaining its effectiveness. In addition, we compare several test suite generators on different feature combination sizes t.
First Page
2021
Last Page
2039
DOI
10.1109/TSE.2024.3419919
Publication Date
1-1-2024
Recommended Citation
Baranov, Eduard; Chakraborty, Sourav; Legay, Axel; Meel, Kuldeep S.; and Vinodchandran, N. Variyam, "A Scalable t-Wise Coverage Estimator: Algorithms and Applications" (2024). Journal Articles. 4569.
https://digitalcommons.isical.ac.in/journal-articles/4569