Software Testing Process in a Test Factory - From Ad hoc Activities to an Organizational Standard

Rossana Maria de Castro Andrade, Ismayle de Sousa Santos, Valéria Lelli, Káthia Marçal de Oliveira, Ana Regina Rocha

Abstract

Software testing is undoubtedly essential for any software development. However, testing is an expensive activity, usually costing more than 50% of the development budget. Thus, to save resources while performing tests with high quality, many software development companies are hiring test factories, which are specialized enterprises for the delivery of outsourced testing services for other companies. Although this kind of organization is common in the industry, we have found few empirical studies concerning test factories. In this paper, we report our experience in the definition, use, and improvement of a software testing process within a test factory. To support the implantation of the test factory, we applied the PDCA (Plan-Do-Check-Act) cycle using the lessons learned in the PDCA check phase to improve the testing process. As a result, we have decreased the number of failures found after the software delivery and thus a higher value for DRE (Defect Removal Efficiency) measure. We also present 12 lessons learned that may be applicable by other test factories.

References

  1. Afzal, W., Alone, S., Glocksien, K., and Torkar, R. (2016). Software test process improvement approaches: A systematic literature review and an industrial case study. Journal of Systems and Software, 111:1 - 33.
  2. Andrade, R. M. C., Carvalho, R. M., Oliveira, K. M., Maia, M. E. F., and Arajo, I. L. (2017). What changes from ubiquitous computing to internet of things in interaction evaluation? In 5th International Conference on Distributed, Ambient and Pervasive Interactions, DAPI 2017. Held as Part of the 19th International Conference on Human-Computer Interaction 2017.
  3. Bezerra, C., Andrade, R. M. C., Santos, R. M., Abed, M., de Oliveira, K. M., Monteiro, J. M., Santos, I., and Ezzedine, H. (2014). Challenges for usability testing in ubiquitous systems. In Proceedings of the 26th Conference on L'Interaction Homme-Machine, IHM 7814, pages 183-188, New York, NY, USA. ACM.
  4. Carvalho, R. M., Andrade, R. M. C., Oliveira, K. M., Santos, I. S., and Bezerra, C. I. M. (2016). Quality characteristics and measures for human-computer interaction evaluation in ubiquitous systems. Software Quality Journal, pages 1-53.
  5. Collins, E. F. and de Lucena, Jr., V. F. (2012). Software test automation practices in agile development environment: An industry experience report. In Proceedings of the 7th International Workshop on Automation of Software Test, AST 7812, pages 57-63, Piscataway, NJ, USA. IEEE Press.
  6. Cooper-Brown, B.; Ludhani, C. C. S. (2015). Test factory setup for sap applications. https://www.infosys.com/ IT-services/independent-validation-testing-services/ white-papers/Documents/test-factory-setup.pdf.
  7. CWI (2017). CWI's test factory. Available: http://www.cwi. com.br/Services/TestFactory. [Online, Accessed: 19- Feb-2017].
  8. Cyber:con (2017). Cyber:con's test factory. Available: http://www.cybercon.de/en GB/testing. [Online, Accessed: 19-Feb-2017].
  9. Dantas, V. L. L., Marinho, F. G., da Costa, A. L., and Andrade, R. M. C. (2009). Testing requirements for mobile applications. In 2009 24th International Symposium on Computer and Information Sciences, pages 555-560.
  10. Engström, E. and Runeson, P. (2010). A qualitative survey of regression testing practices. In Proceedings of the 11th International Conference on Product-Focused Software Process Improvement, PROFES'10, pages 3-16, Berlin, Heidelberg. Springer-Verlag.
  11. FH (2017). Fh's test factory. Available: http://www.fh. com.br/en/servicos/technology/software-development/ test-factory/. [Online, Accessed: 19-Feb-2017].
  12. IEEE (2015). IEEE Draft International Standard for Software and Systems Engineering-Software TestingPart 4: Test Techniques. ISO/IEC/IEEE P29119-4- FDIS April 2015, pages 1-147.
  13. Johnson, C. N. (2002). The benefits of pdca. Quality Progress, 35:120-121.
  14. Jones, C. (1996). Software defect-removal efficiency. Computer, 29(4):94-95.
  15. Lelli, V., Blouin, A., and Baudry, B. (2015a). Classifying and qualifying gui defects. In 2015 IEEE 8th International Conference on Software Testing, Verification and Validation (ICST), pages 1-10.
  16. Lelli, V., Blouin, A., Baudry, B., and Coulon, F. (2015b). On model-based testing advanced guis. In 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pages 1-10.
  17. Myers, G. J., Sandler, C., and Badgett, T. (2011). The Art of Software Testing. Wiley Publishing, 3rd edition.
  18. Ramler, R. and Felderer, M. (2015). A process for riskbased test strategy development and its industrial evaluation. In Proceedings of the 16th International Conference on Product-Focused Software Process Improvement - Volume 9459, PROFES 2015, pages 355- 371, New York, NY, USA. Springer-Verlag New York, Inc.
  19. Rocha, A. R., Montoni, M., Santos, G., Mafra, S., Figueiredo, S., Albuquerque, A., and Mian, P. (2005). Reference model for software process improvement: A brazilian experience. In Proceedings of the 12th European Conference on Software Process Improvement, EuroSPI'05, pages 130-141, Berlin, Heidelberg. Springer-Verlag.
  20. Rocha, L. C., Andrade, R. M. C., Sampaio, A. L., and Lelli, V. (2017). Heuristics to evaluate the usability of ubiquitous systems. In 5th International Conference on Distributed, Ambient and Pervasive Interactions, DAPI 2017. Held as Part of the 19th International Conference on Human-Computer Interaction 2017.
  21. Santos, A. M., Karlsson, B. F., Cavalcante, A. M., Correia, I. B., and Silva, E. (2011). Testing in an agile product development environment: An industry experience report. In Latin American Test Workshop, pages 1-6.
  22. Sanz, A., García, J., Salda n˜a, J., and Amescua, A. (2009). A proposal of a process model to create a test factory. In Proceedings of the Seventh ICSE Conference on Software Quality, WOSQ'09, pages 65-70, Washington, DC, USA. IEEE Computer Society.
  23. Schwaber, K. and Sutherland, J. (2016). MPS.BR - Melhoria de Processo do Software Brasileiro, Guia Geral (v. 1.1). http://www.Scrumguides.org/docs/ Scrumguide/v1/Scrum-guide-us.pdf.
  24. Shamsoddin-motlagh, E. (2012). Article: A review of automatic test cases generation. International Journal of Computer Applications, 57(13):25-29. Full text available.
  25. SOFTEX (2006). The scrum guide - the definitive guide to scrum: The rules of the game. http://www.softex.br/ mpsbr/.
  26. Xia, X., Lo, D., Kochhar, P. S., Xing, Z., Wang, X., and Li, S. (2015). Experience report: An industrial experience report on test outsourcing practices. In 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE), pages 370-380.
  27. Zieris, F. and Prechelt, L. (2016). Observations on knowledge transfer of professional software developers during pair programming. In Proceedings of the 38th International Conference on Software Engineering Companion, ICSE 7816, pages 242-250, New York, NY, USA. ACM.
Download


Paper Citation


in Harvard Style

Maria de Castro Andrade R., de Sousa Santos I., Lelli V., Marçal de Oliveira K. and Regina Rocha A. (2017). Software Testing Process in a Test Factory - From Ad hoc Activities to an Organizational Standard . In Proceedings of the 19th International Conference on Enterprise Information Systems - Volume 2: ICEIS, ISBN 978-989-758-248-6, pages 132-143. DOI: 10.5220/0006333301320143


in Bibtex Style

@conference{iceis17,
author={Rossana Maria de Castro Andrade and Ismayle de Sousa Santos and Valéria Lelli and Káthia Marçal de Oliveira and Ana Regina Rocha},
title={Software Testing Process in a Test Factory - From Ad hoc Activities to an Organizational Standard},
booktitle={Proceedings of the 19th International Conference on Enterprise Information Systems - Volume 2: ICEIS,},
year={2017},
pages={132-143},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0006333301320143},
isbn={978-989-758-248-6},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 19th International Conference on Enterprise Information Systems - Volume 2: ICEIS,
TI - Software Testing Process in a Test Factory - From Ad hoc Activities to an Organizational Standard
SN - 978-989-758-248-6
AU - Maria de Castro Andrade R.
AU - de Sousa Santos I.
AU - Lelli V.
AU - Marçal de Oliveira K.
AU - Regina Rocha A.
PY - 2017
SP - 132
EP - 143
DO - 10.5220/0006333301320143