Validating Search Processes in Systematic Literature Reviews

Barbara Kitchenham, Zhi Li, Andrew Burn

2011

Abstract

Context: Systematic Literature Reviews (SLRs) need to employ a search process that is as complete as possible. It has been suggested that an existing set of known papers can be used to help develop an appropriate strategy. However, it is still not clear how to evaluate the completeness of the resulting search process. Aim: We suggest a means of assessing the completeness of a search process by evaluating the search results on an independent set of known papers. Method: We assess the results of a search process developed using a known set of papers by seeing whether it was able to identify papers from a different set of known papers. Results: Using a second set of known papers, we were able to show that a search process, which was based on a first set of known papers, was unlikely to be complete, even though the search process found all the papers in the first known set. Conclusions: When using a set of known papers to develop a search process, keep a “hold-out” sample to evaluate probable completeness.

References

  1. Dieste, O. and Padua, A. G., 2007. Developing Search Strategies for Detecting Relevant Experiments for Systematic Reviews. Proceedings 1st International Symposium on Empirical Software Engineering and Measurement, ESEM 7807, IEEE Computer Society.
  2. , Mailshevsky, A. G. and Rothermel, G., 2000. Prioritizing Test Cases for Regression Testing. Proc. Int'l Symp. Software Testing and Analysis, ACM Press, pp. 102- 112.
  3. Engström, E., Runeson, P. and Skoglund, M., 2010. A systematic review on regression test selection techniques. Information and Software Technology, 52, pp 14-30.
  4. Jia, Y. and Harmon, M. In press. An Analysis and Survey of the Development of Mutation Testing, IEEE TSE. Doi 10.1109/TSE.2010.62.
  5. Juristo, N. Moreno, A. M., Vigas, S. and Solari, M., 2006. In Search of What We Experimentally Know about Unit Testing, IEEE Software, vol. 23, pp. 72-80.
  6. Kitchenham, B. A. and Charters, S., 2007. Guidelines for performing Systematic Literature. Reviews in Software Engineering, Technical Report EBSE-2007-01.
  7. Kitchenham, B., Pretorius, R., Budgen. D., Brereton, P., Turner, M., Niazi, M., and Linkman, S. G., 2010. Systematic Literature Reviews in Software Engineering - A Tertiary Study. Information and Software Technology, 52(8), pp 934-944.
  8. Kitchenham, B. A. Breerton, O. P. Li, Z, Budgen, D., Burn, A., 2011. Repeatability of systematic literature reviews, EASE 20011.
  9. Leon, D., Masri, W. and Podgurski, A., 2005. An Empirical Evaluation of Test Case Filtering Techniques Based on Exercising Complex Information Flows, Proc. 27th Int'l Conf. Software Eng. (ICSE 05), IEEE CS Press, pp. 412-421.
  10. Rothermel, G. Untch, R. H. Chengyun C, Harrold, M. J., 1999. Test case prioritization: an empirical study, Proc. Int'l Conf. Software Maintenance, IEEE CS Press, pp. 179-188.
  11. Skoglund, M. and Runeson, P., 2009. Reference-based search strategies in systematic reviews. Proceedings EASE'09, BCS eWic.
  12. Spoor, P., Alrey, M., Bennett, C., Greensill, J. and Williams, R. Using capture-recapture technique to evaluate the completeness of systematic literature searches. British Medical Journal, 1996, 313(7053), p 342.
Download


Paper Citation


in Harvard Style

Kitchenham B., Li Z. and Burn A. (2011). Validating Search Processes in Systematic Literature Reviews . In Proceeding of the 1st International Workshop on Evidential Assessment of Software Technologies - Volume 1: EAST, (ENASE 2011) ISBN 978-989-8425-58-4, pages 3-9


in Bibtex Style

@conference{east11,
author={Barbara Kitchenham and Zhi Li and Andrew Burn},
title={Validating Search Processes in Systematic Literature Reviews},
booktitle={Proceeding of the 1st International Workshop on Evidential Assessment of Software Technologies - Volume 1: EAST, (ENASE 2011)},
year={2011},
pages={3-9},
publisher={SciTePress},
organization={INSTICC},
doi={},
isbn={978-989-8425-58-4},
}


in EndNote Style

TY - CONF
JO - Proceeding of the 1st International Workshop on Evidential Assessment of Software Technologies - Volume 1: EAST, (ENASE 2011)
TI - Validating Search Processes in Systematic Literature Reviews
SN - 978-989-8425-58-4
AU - Kitchenham B.
AU - Li Z.
AU - Burn A.
PY - 2011
SP - 3
EP - 9
DO -