Using the Juliet Test Suite to Compare Static Security Scanners

Andreas Wagner, Johannes Sametinger

2014

Abstract

Security issues arise permanently in different software products. Making software secure is a challenging endeavour. Static analysis of the source code can help eliminate various security bugs. The better a scanner is, the more bugs can be found and eliminated. The quality of security scanners can be determined by letting them scan code with known vulnerabilities. Thus, it is easy to see how much they have (not) found. We have used the Juliet Test Suite to test various scanners. This test suite contains test cases with a set of security bugs that should be found by security scanners. We have automated the process of scanning the test suite and of comparing the generated results. With one exception, we have only used freely available source code scanners. These scanners were not primarily targeted at security, yielding disappointing results at first sight. We will report on the findings, on the barriers for automatic scanning and comparing, as well as on the detailed results.

References

  1. Black, P.E., 2009. Static Analyzers in Software Engineering. CROSSTALK-The Journal of Defense Software Engineering. https://buildsecurityin.us-cert.gov/resources/crosstalkseries/static-analyzers-in-software-engineering
  2. Boland, T., Black, P.E., 2012. Juliet 1.1 C/C++ and Java Test Suite, Computer, vol. 45, no. 10, pp. 88-90, DOI: 10.1109/MC.2012.345
  3. Center for Assured Software, 2011. CAS Static Analysis Tool Study - Methodology, December 2011. http://samate.nist.gov/docs/CAS%202011%20Static% 20Analysis%20Tool%20Study%20Methodology.pdf
  4. Hofer T., 2010. Evaluating Static Source Code Analysis Tools, Master Thesis, InfoScience 2010. http://infoscience.epfl.ch/ record/153107
  5. Howard M., Lipner S., 2006. The Security Development Life-Cycle, Microsoft Press.
  6. McCullagh D., 2014. Klocwork: Our source code analyzer caught Apple's 'gotofail' bug, c|net February 28. http://news.cnet.com/8301-1009_3-57619754- 83/klocwork-our-source-code-analyzer-caught-applesgotofail-bug/
  7. McGraw G., 2004. Software Security, IEEE Security & Privacy, vol. 2, no. 2, pp. 80-83. doi:10.1109/MSECP.2004.1281254
  8. McGraw G., 2009. Software Security: Building Security In, 5th edition, Addison-Wesley.
  9. MITRE 2011. CWE/SANS Top 25 Most Dangerous Software Errors, Version 1.0.3. http://cwe.mitre.org/top25/
  10. National Institute of Standards and Technology 2012. SAMATE Reference Dataset. http://samate.nist.gov/SRD/testsuite.php.
  11. Plösch, R., et al., 2008. Tool Support for a Method to Evaluate Internal Software Product Quality by Static Code Analysis, Software Quality Professional Journal, American Society for Quality, Volume 10, Issue 4.
  12. Rutar, N., Almazan, C.B., Foster, J.S., 2004. A Comparison of Bug Finding Tools for Java, IEEE, ISBN 0- 7695-2215-7, 245-256.
  13. Wagner, S., et al., 2012. The Quamoco Product Quality Modelling and Assessment Approach, Proceedings of 34th International Conference on Software Engineering (ICSE 2012), Zurich.
Download


Paper Citation


in Harvard Style

Wagner A. and Sametinger J. (2014). Using the Juliet Test Suite to Compare Static Security Scanners . In Proceedings of the 11th International Conference on Security and Cryptography - Volume 1: SECRYPT, (ICETE 2014) ISBN 978-989-758-045-1, pages 244-252. DOI: 10.5220/0005032902440252


in Bibtex Style

@conference{secrypt14,
author={Andreas Wagner and Johannes Sametinger},
title={Using the Juliet Test Suite to Compare Static Security Scanners},
booktitle={Proceedings of the 11th International Conference on Security and Cryptography - Volume 1: SECRYPT, (ICETE 2014)},
year={2014},
pages={244-252},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005032902440252},
isbn={978-989-758-045-1},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 11th International Conference on Security and Cryptography - Volume 1: SECRYPT, (ICETE 2014)
TI - Using the Juliet Test Suite to Compare Static Security Scanners
SN - 978-989-758-045-1
AU - Wagner A.
AU - Sametinger J.
PY - 2014
SP - 244
EP - 252
DO - 10.5220/0005032902440252