![](bg8.png)
x-axis shows the programs and the y-axis shows the
time in seconds (logarithmic format). We can observe
the red line (PTime) is most of the time below the
blue line (CTime). This shows that PTime is very less
and hence PICT is beneficial to use in VeriCombTest
framework. Time for GCoV (GTime) is the time con-
sumed by the GCOV tool for both modes. The Ag-
gregate of 38 programs, CTime, PTime, and GTime
are 546.99 min, 2.67 min, 5.69 min respectively. Fig.
4 shows the graph for time consumption in mutation
analysis by Mode1 and Mode2. The x-axis shows the
programs and the y-axis shows the time in seconds
(logarithmic format).
From the graph, it is very clear that the blue line
is mostly below the red line. For very few programs
the time difference is closer. In the Aggregate of 38
programs, the value of Mutation Analysis Time for
Mode 1 (MTime1) is 496.07 min and Mutation Anal-
ysis Time for Mode 2 (MTime2) is 3960.17 min. The
VeriCombtest i.e. Mode2 is clearly worse in muta-
tion analysis time by 7.98× as compared to baseline
CBMC i.e. Mode1. Since the PICT produced more
number of test cases that makes mutation time anal-
ysis worse. Improvement of the mutation time is the
out of the scope of this paper. From this work, we
would like to highlight that there is a scope for im-
provement for a verifier i.e. CBMC wrt. test cases.
5 CONCLUSION
From the literature survey we can observe that the
works such as Veritesting (Avgerinos et al., 2014),
Veriabs (Afzal et al., 2019; Darke et al., 2018), and
VeriFuzz (Basak Chowdhury et al., 2019) used the
common mechanism of combining different tools and
techniques. Motivating by these works, we com-
bine two techniques viz. Verification and Combi-
natorial Testing. For verification, we have consid-
ered CBMC and for Combinatorial Testing we con-
sidered PICT. We experimented with 38 C-Programs
from The RERS challenge repository. For 38 pro-
grams PICT only consumed 2.6 Minutes to popu-
late the test inputs, however, CBMC which is a Static
Symbolic Executor consumed 546.99 Minutes to gen-
erate the test input. The VeriCombTest (Mode2)
has 355 extra killed mutants as compared to base-
line CBMC (Mode1). Finally, as discussed in cover-
age analysis section VeriCombTest has 21.05% pro-
grams to achieve higher Branch Coverage and in mu-
tation analysis VeriCombTest has 94.73% programs
to achieve higher mutation score, these show the
strength of our proposed work.
REFERENCES
Afzal, M., Asia, A., Chauhan, A., Chimdyalwar, B., Darke,
P., Datar, A., Kumar, S., and Venkatesh, R. (2019). Ve-
riabs : Verification by abstraction and test generation.
In 2019 34th IEEE/ACM International Conference on
Automated Software Engineering (ASE), pages 1138–
1141.
Armando, A., Mantovani, J., and Platania, L. (2006).
Bounded model checking of software using SMT
solvers instead of SAT solvers. In 13th SPIN, volume
3925 of Lecture Notes in Computer Science, pages
146–162. Springer.
Avgerinos, T., Rebert, A., Cha, S. K., and Brumley, D.
(2014). Enhancing symbolic execution with veritest-
ing. In Proceedings of the 36th International Con-
ference on Software Engineering, ICSE 2014, page
1083–1094, New York, NY, USA. Association for
Computing Machinery.
Basak Chowdhury, A., Medicherla, R. K., and R, V. (2019).
Verifuzz: Program aware fuzzing. In Beyer, D., Huis-
man, M., Kordon, F., and Steffen, B., editors, Tools
and Algorithms for the Construction and Analysis
of Systems, pages 244–249, Cham. Springer Interna-
tional Publishing.
Clarke, E., Kroening, D., and Yorav, K. (2003). Behavioral
consistency of C and Verilog programs using bounded
model checking. In 40th DAC, pages 368–371. ACM
Press.
Darke, P., Prabhu, S., Chimdyalwar, B., Chauhan, A., Ku-
mar, S., Basakchowdhury, A., Venkatesh, R., Datar,
A., and Medicherla, R. K. (2018). Veriabs: Verifica-
tion by abstraction and test generation. In Beyer, D.
and Huisman, M., editors, Tools and Algorithms for
the Construction and Analysis of Systems, pages 457–
462, Cham. Springer International Publishing.
Mall, R. (2018). Fundamentals of software engineering.
PHI Learning Pvt. Ltd.
Mathur, A. P. and Wong, W. E. (1993). Comparing the fault
detection e ectiveness of mutation and data flow test-
ing: An empirical study.
Nie, C. and Leung, H. (2011). A survey of combinatorial
testing. ACM Comput. Surv., 43(2).
RERS (2018). RERS:. http://rers-challenge.org/.
VeriCombTest-Artifacts (2021). Raw experimnetal data.
https://figshare.com/s/0d412445b4411208ae68.
VeriCombTest: Automated Test Case Generation Technique Using a Combination of Verification and Combinatorial Testing
313