A Structured Approach for Conducting a Series of Controlled Experiments in Software Visualization

Richard Müller, Pascal Kovacs, Jan Schilbach, Ulrich W. Eisenecker, Dirk Zeckzer, Gerik Scheuermann

2014

Abstract

In the field of software visualization controlled experiments are an important instrument to investigate the specific reasons, why some software visualizations excel the expectations on providing insights and ease task solving while others fail doing so. Despite this, controlled experiments in software visualization are rare. A reason for this is the fact that performing such evaluations in general, and particularly performing them in a way that minimizes the threats to validity, is hard to accomplish. In this paper, we present a structured approach on how to conduct a series of controlled experiments in order to give empirical evidence for advantages and disadvantages of software visualizations in general and of 2D vs. 3D software visualizations in particular.

References

  1. Amar, R., Eagan, J., and Stasko, J. (2005). Low-level components of analytic activity in information visualization. In Pro. 2005 IEEE Symp. Inf. Vis., page 15. IEEE Computer Society.
  2. Andrews, K. (2008). Evaluation comes in many guises. In Proc. 2008 AVI Work. BEyond Time Errors Nov. Eval. Method. Inf. Vis., pages 8-10.
  3. Balogh, G. and Beszedes, A. (2013). CodeMetropolis - a Minecraft based collaboration tool for developers. In Proc. 1st IEEE Work. Conf. Softw. Vis., pages 1-4.
  4. Basili, V. R., Selby, R. W., and Hutchens, D. H. (1986). Experimentation in software engineering. IEEE Trans. Softw. Eng., 12(7):733-743.
  5. Benomar, O. and Poulin, P. (2013). Visualizing Software Dynamicities with Heat Maps. In 1st IEEE Work. Conf. Softw. Vis. IEEE.
  6. Bull, R. I., Storey, M.-A., Favre, J.-M., and Litoiu, M. (2006). An Architecture to Support Model Driven Software Visualization. In 14th Int. Conf. Progr. Compr., pages 100-106. IEEE Computer Society.
  7. Carpendale, S. (2008). Evaluating information visualizations. In Kerren, A., Stasko, J. T., Fekete, J.-D., and North, C., editors, Information Visualization, volume 4950, pages 19-45. Springer, Berlin, Heidelberg.
  8. Cockburn, A. and McKenzie, B. (2001). 3D or not 3D?: evaluating the effect of the third dimension in a document management system. In Proc. SIGCHI Conf. Hum. factors Comput. Syst., pages 434-441. ACM.
  9. Cockburn, A. and McKenzie, B. (2002). Evaluating the effectiveness of spatial memory in 2D and 3D physical and virtual environments. In Proc. SIGCHI Conf. Hum. Factors Comput. Syst., pages 203-210. ACM.
  10. Di Penta, M., Stirewalt, R. E. K., and Kraemer, E. (2007). Designing your Next Empirical Study on Program Comprehension. 15th Int. Conf. Progr. Compr., pages 281-285.
  11. Diehl, S. (2007). Software visualization: Visualizing the Structure, Behaviour, and Evolution of Software. Springer.
  12. Ducasse, S., G?irba, T., and Favre, J. (2004). Modeling software evolution by treating history as a first class entity. In Proc. Work. Softw. Evol. Through Transform., pages 75-86. Elsevier.
  13. Dwyer, T. (2001). Three dimensional UML using force directed layout. In Proc. 2001 Asia-Pacific Symp. Inf. Vis., volume 9, pages 77-85. Australian Computer Society.
  14. Fittkau, F., Waller, J., Wulf, C., and Hasselbring, W. (2013). Live Trace Visualization for Comprehending Large Software Landscapes : The ExplorViz Approach. In 1st IEEE Work. Conf. Softw. Vis., pages 1-4.
  15. Gallagher, K., Hatch, A., and Munro, M. (2008). Software Architecture Visualization: An Evaluation Framework and Its Application. IEEE Trans. Softw. Eng., 34(2):260-270.
  16. Grac?anin, D., Matkovic, K., and Eltoweissy, M. (2005). Software Visualization. Innov. Syst. Softw. Eng., 1(2):59-63.
  17. Greevy, O. (2007). Dynamix - a meta-model to support feature-centric analysis. In 1st Int. Work. FAMIX Moose Reeng.
  18. Herman, I., Melancon, G., and Marshall, M. S. (2000). Graph visualization and navigation in information visualization: A survey. IEEE Trans. Vis. Comput. Graph., 6(1):24-43.
  19. Höffler, T. N. and Leutner, D. (2007). Instructional animation versus static pictures: A meta-analysis. Learn. Instr., 17(6):722-738.
  20. Huynh-Thu, Q., Barkowsky, M., and Le Callet, P. (2011). The Importance of Visual Attention in Improving the 3D-TV Viewing Experience: Overview and New Perspectives. IEEE Trans. Broadcast., 57(2):421-431.
  21. Irani, P. and Ware, C. (2000). Diagrams based on structural object perception. In Proc. Work. Conf. Adv. Vis. Interf., pages 61-67. ACM.
  22. Irani, P. and Ware, C. (2003). Diagramming information structures using 3D perceptual primitives. ACM Trans. Comput. Hum. Interact., 10(1):1-19.
  23. Isenberg, T., Isenberg, P., Chen, J., Sedlmair, M., and Möller, T. (2013). A systematic review on the practice of evaluating visualization. IEEE Trans. Vis. Comput. Graph., 19(12):2818-27.
  24. Keim, D. A., Kohlhammer, J., Ellis, G., and Mansmann, F. (2010). Mastering The Information Age-Solving Problems with Visual Analytics. Eurographics Association.
  25. Keim, D. A. and Schneidewind, J. (2007). Introduction to the Special Issue on Visual Analytics. SIGKDD Explorations, 9(2):3-4.
  26. Koike, H. and Chu, H. (1998). How does 3-D visualization work in software engineering?: empirical study of a 3- D version/module visualization system. In Proc. 20th Int. Conf. Softw. Eng., pages 516-519. IEEE Computer Society.
  27. Kosara, R., Healey, C. G., Interrante, V., Laidlaw, D. H., and Ware, C. (2003). User Studies: Why, How, and When? IEEE Comput. Graph. Appl., 23(4):20-25.
  28. Myers, B. A. (1990). Taxonomies of visual programming and program visualization. J. Vis. Lang. Comput., 1(1):97-123.
  29. Nierstrasz, O., Ducasse, S., and G?irba, T. (2005). The story of moose: an agile reengineering environment. In Proc. 10th Eur. Softw. Eng. Conf., pages 1-10. ACM.
  30. Pfleeger, S. L. (1995). Experimental design and analysis in software engineering. Ann. Softw. Eng., 1(1):219-253.
  31. Price, B. A., Baecker, R. M., and Small, I. S. (1993). A Principled Taxonomy of Software Visualization. J. Vis. Lang. Comput., 4(3):211-266.
  32. Roman, G.-C. and Cox, K. C. (1993). A taxonomy of program visualization systems. Computer, 26(12):11-24.
  33. Sensalire, M., Ogao, P., and Telea, A. (2009). Evaluation of software visualization tools: Lessons learned. In 5th Int. Work. Vis. Softw. Underst. Anal., pages 19-26.
  34. Sharif, B. and Jetty, G. (2013). An Empirical Study Assessing the Effect of SeeIT 3D on Comprehension. In 1st IEEE Work. Conf. Softw. Vis.
  35. Shneiderman, B. (1996). The eyes have it: A task by data type taxonomy for information visualizations. In Proc. 1996 IEEE Symp. Vis. Lang., pages 336-343.
  36. Siegmund, J. (2012). Framework for Measuring Program Comprehension. Phd thesis, Otto-von-GuerickeUniversität Magdeburg.
  37. Sjoberg, D. I. K., Dyba°, T., and Jorgensen, M. (2007). The Future of Empirical Methods in Software Engineering Research. In Futur. Softw. Eng. (FOSE 7807), pages 358-378.
  38. Smisek, J., Jancosek, M., and Pajdla, T. (2011). 3D with Kinect. In 2011 IEEE Int. Conf. Comput. Vis. Work., pages 1154-1160.
  39. Stasko, J. and Patterson, C. (1993). Understanding and Characterizing Program Visualization Systems. Technical report, Georgia Institute of Technology, Atlanta.
  40. Stasko, J. and Wehrli, J. (1993). Three-dimensional computation visualization. Proc. 1993 IEEE Symp. Vis. Lang., pages 100-107.
  41. Storey, M.-A. D., C?ubranic, D., and German, D. M. (2005). On the use of visualization to support awareness of human activities in software development: a survey and a framework. In Proc. 2005 ACM Symp. Softw. Vis., pages 193-202. ACM.
  42. Teyseyre, A. R. and Campo, M. R. (2009). An overview of 3D software visualization. IEEE Trans. Vis. Comput. Graph., 15(1):87-105.
  43. Tichy, W. and Padberg, F. (2007). Empirische Methodik in der Softwaretechnik im Allgemeinen und bei der Software-Visualisierung im Besonderen. In Softw. Eng. 2007 Beitr. Work., pages 211-222. Gesellschaft für Informatik.
  44. Waller, J., Wulf, C., Fittkau, F., Döhring, P., and Hasselbring, W. (2013). SynchroVis: 3D Visualization of Monitoring Traces in the City Metaphor for Analyzing Concurrency. In 1st IEEE Work. Conf. Softw. Vis., pages 7-10.
  45. Ware, C. and Franck, G. (1994). Viewing a graph in a virtual reality display is three times as good as a 2D diagram. IEEE Symp. Vis. Lang., pages 182-183.
  46. Ware, C. and Franck, G. (1996). Evaluating stereo and motion cues for visualizing information nets in three dimensions. ACM Trans. Graph., 15(2):121-140.
  47. Ware, C., Hui, D., and Franck, G. (1993). Visualizing object oriented software in three dimensions. In Proc. 1993 Conf. Cent. Adv. Stud. Collab. Res. Softw. Eng., pages 612-620. IBM Press.
  48. Ware, C. and Mitchell, P. (2008). Visualizing graphs in three dimensions. ACM Trans. Appl. Percept., 5(1):1- 15.
  49. Weichert, F., Bachmann, D., Rudak, B., and Fisseler, D. (2013). Analysis of the accuracy and robustness of the leap motion controller. Sensors, 13:6380-6393.
  50. Wettel, R., Lanza, M., and Robbes, R. (2011). Software systems as cities: A controlled experiment. In Proc. 33rd Int. Conf. Softw. Eng., pages 551-560. ACM.
  51. Wohlin, C., Runeson, P., Höst, M., Ohlsson, M. C., Regnell, B., and Wesslén, A. (2012). Experimentation in Software Engineering. Springer.
  52. Yi, J. S., ah Kang, Y., Stasko, J. T., and Jacko, J. A. (2007). Toward a deeper understanding of the role of interaction in information visualization. IEEE Trans. Vis. Comput. Graph., 13(6):1224-1231.
  53. Zorrilla, M., Martin, A., Sanchez, J. R., Tamayo, I., and Olaizola, I. G. (2013). HTML5-based system for interoperable 3D digital home applications. Multimed. Tools Appl., pages 1-21.
Download


Paper Citation


in Harvard Style

Müller R., Kovacs P., Schilbach J., Eisenecker U., Zeckzer D. and Scheuermann G. (2014). A Structured Approach for Conducting a Series of Controlled Experiments in Software Visualization . In Proceedings of the 5th International Conference on Information Visualization Theory and Applications - Volume 1: IVAPP, (VISIGRAPP 2014) ISBN 978-989-758-005-5, pages 204-209. DOI: 10.5220/0004835202040209


in Bibtex Style

@conference{ivapp14,
author={Richard Müller and Pascal Kovacs and Jan Schilbach and Ulrich W. Eisenecker and Dirk Zeckzer and Gerik Scheuermann},
title={A Structured Approach for Conducting a Series of Controlled Experiments in Software Visualization},
booktitle={Proceedings of the 5th International Conference on Information Visualization Theory and Applications - Volume 1: IVAPP, (VISIGRAPP 2014)},
year={2014},
pages={204-209},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0004835202040209},
isbn={978-989-758-005-5},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 5th International Conference on Information Visualization Theory and Applications - Volume 1: IVAPP, (VISIGRAPP 2014)
TI - A Structured Approach for Conducting a Series of Controlled Experiments in Software Visualization
SN - 978-989-758-005-5
AU - Müller R.
AU - Kovacs P.
AU - Schilbach J.
AU - Eisenecker U.
AU - Zeckzer D.
AU - Scheuermann G.
PY - 2014
SP - 204
EP - 209
DO - 10.5220/0004835202040209