Automatically Generating Tests from Natural Language Descriptions of Software Behavior

Sunil Kamalakar, Stephen H. Edwards, Tung M. Dao

2013

Abstract

Behavior-Driven Development (BDD) is an emerging agile development approach where all stakeholders (including developers and customers) work together to write user stories in structured natural language to capture a software application’s functionality in terms of required `behaviors.' Developers can then manually write `glue' code so that these scenarios can be translated into executable software tests. This glue code represents individual steps within unit and acceptance test cases, and tools exist that automate the mapping from scenario descriptions to manually written code steps (typically using regular expressions). This paper takes the position that, instead of requiring programmers to write manual glue code, it is practical to convert natural language scenario descriptions into executable software tests %{\it fully automatically}%. To show feasibility, this paper presents preliminary results from a tool called Kirby that uses natural language processing techniques to automatically generate executable software tests from structured English scenario descriptions. Kirby relieves the developer from the laborious work of writing code for the individual steps described in scenarios, so that both developers and customers can both focus on the scenarios as pure behavior descriptions (understandable to all, not just programmers). Preliminary results from assessing the per-formance and accuracy of this technique are presented.

References

  1. Beck, K. 2002. Test Driven Development: By Example. Addison-Wesley Longman Publishing Co., Inc., Boston, MA, USA.
  2. Budinsky, F. J., Finnie, M. A., Vlissides, J. M., and Yu, P. S. 1996. Automatic code generation from design patterns. IBM Systems Journal, 35(2):151-171, May 1996.
  3. CodeModel. 2013. http://codemodel.java.net [Accessed May 15, 2013].
  4. Cucumber. 2013. http://cukes.info/ [Accessed May 15, 2013].
  5. Edwards, S. H., Shams, Z., Cogswell, M., and Senkbeil, R. C. 2012. Running students' software tests against each others' code: New life for an old “gimmick”. In Proceedings of the 43rd ACM technical symposium on Computer Science Education, SIGCSE 7812, pp. 221- 226, ACM, New York, NY, USA.
  6. Fellbaum, C. (ed.). 1998. WordNet: An Electronic Lexical Database. MIT Press, Cambridge, MA, USA.
  7. Gherkin. 2013. https://github.com/cucumber/cucumber/ wiki/Gherkin [Accessed May 15, 2013].
  8. JBehave. 2013. http://jbehave.org/, Accessed May 15, 2013.
  9. JDave. 2013. http://jdave.org/ [Accessed May 15, 2013].
  10. Keogh, E. 2010. BDD: A lean toolkit. In Proceedings of Lean Software Systems Conference, 2010.
  11. Kolb, P. 2008. DISCO: A multilingual database of distributionally similar words. In Proceedings of KONVENS-2008, Berlin.
  12. Koskela, L. 2007. Test Driven: Practical TDD and Acceptance TDD for Java Developers. Manning Publications Co., Greenwich, CT, USA.
  13. Lazr, I., Motogna, S., and Pírv, B. 2010. Behaviourdriven development of foundational UML components. Electronic Notes in Theoretical Computer Science, 264(1): 91-105, Aug. 2010.
  14. Nagappan, N., Maximilien, E. M., Bhat, T., and Williams, L. 2008. Realizing quality improvement through test driven development: Results and experiences of four industrial teams. Empirical Software Engineering, 13(3): 289-302, June 2008.
  15. NBehave. 2013. http://nbehave.org/ [Accessed May 15, 2013].
  16. North, D. 2013. Introduction to BDD. http://dannorth.net/ introducing-bdd/ [Accessed May 15, 2013].
  17. PHPSpec. 2013. http://www.phpspec.net/ [Accessed May 15, 2013].
  18. Soeken, M., Wille R., and Drechsler, R. 2012. Assisted behavior driven development using natural language processing. In Proceedings of the 50th International Conference on Objects, Models, Components, Patterns (TOOLS'12), Springer-Verlag, Berlin, Heidelberg, pp. 269-287.
  19. Solis, C. and Wang, X. 2011. A study of the characteristics of behavior driven development. In 37th EUROMICRO Conference on Software Engineering and Advanced Applications (SEAA), pp. 383-387.
  20. Soukoreff, R. W., and MacKenzie, I. S. 2001. Measuring errors in text entry tasks: An application of the Levenshtein string distance statistic. In CHI'01 Extended Abstracts on Human Factors in Computing Systems. ACM, New York, NY, USA, pp. 319-320.
  21. Stanford NLP Group. 2013. http://nlp.stanford.edu/ [Accessed May 15, 2013].
  22. Tata, S., and Patel, J.M. 2007. Estimating the selectivity of tf-idf based cosine similarity predicates. ACM SIGMOD Record, 36(2): 7-12, June 2007.
  23. Yu, J.J.-B., and Fleming, A.M. 2010. Automatic code generation via natural language processing, U.S. Patent 7765097, July 27, 2010.
  24. Wynne, M., and Hellesøy, A. 2012. The Cucumber Book: Behaviour-Driven Development for Testers and Developers. Pragmatic Programmers, LLC.
Download


Paper Citation


in Harvard Style

Kamalakar S., H. Edwards S. and M. Dao T. (2013). Automatically Generating Tests from Natural Language Descriptions of Software Behavior . In Proceedings of the 8th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE, ISBN 978-989-8565-62-4, pages 238-245. DOI: 10.5220/0004566002380245


in Bibtex Style

@conference{enase13,
author={Sunil Kamalakar and Stephen H. Edwards and Tung M. Dao},
title={Automatically Generating Tests from Natural Language Descriptions of Software Behavior},
booktitle={Proceedings of the 8th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE,},
year={2013},
pages={238-245},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0004566002380245},
isbn={978-989-8565-62-4},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 8th International Conference on Evaluation of Novel Approaches to Software Engineering - Volume 1: ENASE,
TI - Automatically Generating Tests from Natural Language Descriptions of Software Behavior
SN - 978-989-8565-62-4
AU - Kamalakar S.
AU - H. Edwards S.
AU - M. Dao T.
PY - 2013
SP - 238
EP - 245
DO - 10.5220/0004566002380245