Balanced Scoring Method for Multiple-mark Questions

Darya Tarasowa, Sören Auer

2013

Abstract

Advantages and disadvantages of a learning assessment based on multiple-choice questions (MCQs) are a long and widely discussed issue in the scientific community. However, in practice this type of questions is very popular due to the possibility of automatic evaluation and scoring. Consequently, an important research question is to exploiting the strengths and mitigate the weaknesses of MCQs. In this work we discuss one particularly important issue of MCQs, namely methods for scoring results in the case, when the MCQ has several correct alternatives (multiple-mark questions, MMQs). We propose a general approach and mathematical model to score MMQs, that aims at recognizing guessing while at the same time resulting in a balanced score. In our approach conventional MCQs are viewed as a particular case of multiple-mark questions, thus, the formulas can be applied to tests mixing MCQs and MMQs. The rational of our approach is that scoring should be based on the guessing level of the question. Our approach can be added as an option, or even as a replacement for manual penalization. We show that our scoring method outperforms existing methods and demonstrate that with synthetic and real experiments.

References

  1. Bauer, D., Holzer, M., Kopp, V., and Fischer, M. R. (2011). Pick-N multiple choice-exams: a comparison of scoring algorithms. Advances in health sciences education : theory and practice, 16(2):211-21.
  2. Ben-Simon, A., Budescu, D. V., and Nevo, B. (1997). A Comparative Study of Measures of Partial Knowledge in Multiple-Choice Tests. Applied Psychological Measurement, 21(1):65-88.
  3. Cronbach, L. J. (1941). An experimental comparison of the multiple true-false and multiple multiple-choice tests. Journal of Educational Psychology, 32:533-543.
  4. Dressel, P. and Schmid, J. (1953). Some modifications of the multiple-choice item. Educational and Psychological Measurement, 13(4):574-595.
  5. Farthing, D., Jones, D., and McPhee, D. (1998). Permutational multiple-choice questions: an objective and efficient alternative to essay-type examination questions. ACM SIGCSE Bulletin, 30(3):81-85.
  6. Frisbie, D. A. (1992). The multiple true-false item format: A status review. Educational Measurement: Issues and Practice, 11(4):21-26.
  7. Hohensinn, C. and Kubinger, K. D. (2011). Applying Item Response Theory Methods to Examine the Impact of Different Response Formats. Educational and Psychological Measurement, 71(4):732-746.
  8. Itten, S. and Krebs, R. (1997). Messqualitaet der verschiedenen MC-Itemtypen in den beiden Vorpruefungen des Medizinstudiums an der Universitaet Bern 1997/2. Bern: IAWF.
  9. Jiao, H., Liu, J., and Haynie, K. (2012). Comparison Between Dichotomous and Polytomous Scoring of Innovative Items in a Large-Scale Computerized Adaptive Test. Educational and Psychological Measurement, 72(3):493-509.
  10. Khalili, A., Auer, S., Tarasowa, D., and Ermilov, I. (2012). Slidewiki: Elicitation and sharing of corporate knowledge using presentations. In Proceedings of the EKAW 2012, pages 302-316. Springer.
  11. Morgan, M. (1979). MCQ: An interactive computer program for multiple-choice self-testing. Biochemical Education, 7(3):67-69.
  12. Pomplun, M. and Omar, M. H. (1997). Multiple-Mark Items: An Alternative Objective Item Format? Educational and Psychological Measurement, 57(6):949-962.
  13. Ripkey, D. and Case, S. (1996). A" new" item format for assessing aspects of clinical competence. Academic Medicine, 71(10):34-36.
  14. Serlin, R. and Kaiser, H. (1978). A method for increasing the reliability of a short multiple-choice test. Educational and Psychological Measurement, 38(2):337-340.
  15. Tsai, F. and Suen, H. (1993). A brief report on a comparison of six scoring methods for multiple true-false items. Educational and psychological measurement, 53(2):399-404.
Download


Paper Citation


in Harvard Style

Tarasowa D. and Auer S. (2013). Balanced Scoring Method for Multiple-mark Questions . In Proceedings of the 5th International Conference on Computer Supported Education - Volume 1: CSEDU, ISBN 978-989-8565-53-2, pages 411-416. DOI: 10.5220/0004384304110416


in Bibtex Style

@conference{csedu13,
author={Darya Tarasowa and Sören Auer},
title={Balanced Scoring Method for Multiple-mark Questions},
booktitle={Proceedings of the 5th International Conference on Computer Supported Education - Volume 1: CSEDU,},
year={2013},
pages={411-416},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0004384304110416},
isbn={978-989-8565-53-2},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 5th International Conference on Computer Supported Education - Volume 1: CSEDU,
TI - Balanced Scoring Method for Multiple-mark Questions
SN - 978-989-8565-53-2
AU - Tarasowa D.
AU - Auer S.
PY - 2013
SP - 411
EP - 416
DO - 10.5220/0004384304110416