Quantifying Depth and Complexity of Thinking and Knowledge

Tamal T. Biswas, Kenneth W. Regan

2015

Abstract

Qualitative approaches to cognitive rigor and depth and complexity are broadly represented by Webb’s Depth of Knowledge and Bloom’s Taxonomy. Quantitative approaches have been relatively scant, and some have been based on ancillary measures such as the thinking time expended to answer test items. In competitive chess and other games amenable to incremental search and expert evaluation of options, we show how depth and complexity can be quantified naturally. We synthesize our depth and complexity metrics for chess into measures of difficulty and discrimination, and analyze thousands of games played by humans and computers by these metrics. We show the extent to which human players of various skill levels evince shallow versus deep thinking, and how they cope with ‘difficult’ versus ‘easy’ move decisions. The goal is to transfer these measures and results to application areas such as multiple-choice testing that enjoy a close correspondence in form and item values to the problem of finding good moves in chess positions.

References

  1. Andersen, E. (1973). Conditional inference for multiplechoice questionnaires. Brit. J. Math. Stat. Psych., 26:31-44.
  2. Anderson, L. and Krathwol, D. (2001). A Taxonomy for Learning, Teaching, and Assessing: A revision of Blooms taxonomy of educational objectives: complete edition. Longman, New York.
  3. Andrich, D. (1978). A rating scale formulation for ordered response categories. Psychometrika, 43:561-573.
  4. Andrich, D. (1988). Rasch Models for Measurement. Sage Publications, Beverly Hills, California.
  5. Baker, F. B. (2001). The Basics of Item Response Theory. ERIC Clearinghouse on Assessment and Evaluation.
  6. Bloom, B. (1956). Taxonomy of Educational Objectives, Handbook I: The Cognitive Domain. David McKay Co., New York.
  7. Bransford, J. D., Brown, A., and Cocking, R., editors (2000). How People Learn: expanded edition. The National Academies Press, Washington, D.C.
  8. Busemeyer, J. R. and Townsend, J. T. (1993). Decision field theory: a dynamic-cognitive approach to decision making in an uncertain environment. Psychological review, 100(3):432.
  9. Chabris, C. and Hearst, E. (2003). Visualization, pattern recognition, and forward search: Effects of playing speed and sight of the position on grandmaster chess errors. Cognitive Science, 27:637-648.
  10. Donovan, M. S. and Bransford, J. D. (2005). How Students Learn. The National Academies Press, Washington, D.C.
  11. Elo, A. (1978). The Rating of Chessplayers, Past and Present. Arco Pub., New York.
  12. Ferreira, D. (2013). The impact of search depth on chess playing strength. ICGA Journal, 36(2):67-80.
  13. Glickman, M. E. (1999). Parameter estimation in large dynamic paired comparison experiments. Applied Statistics, 48:377-394.
  14. Guid, M. and Bratko, I. (2006). Computer analysis of world chess champions. ICGA Journal, 29(2):65-73.
  15. Guid, M. and Bratko, I. (2011). Using heuristic-search based engines for estimating human skill at chess. ICGA Journal, 34(2):71-81.
  16. Haworth, G. (2003). Reference fallible endgame play. ICGA Journal, 26:81-91.
  17. Hotiu, A. (2006). The relationship between item difficulty and discrimination indices in multiple-choice tests in a physical science course. M.Sc. thesis.
  18. Krathwohl, D., Bloom, B., and Bertram, B. (1973). Taxonomy of Educational Objectives, the Classification of Educational Goals. Handbook II: Affective Domain. David McKay Co., New York.
  19. Linacre, J. M. (2006). Rasch analysis of rank-ordered data. Journal of Applied Measurement, 7(1).
  20. Masters, G. (1982). A Rasch model for partial credit scoring. Psychometrika, 47:149-174.
  21. Morris, G. A., Branum-Martin, L., Harshman, N., Baker, S. D., Mazur, E., Dutta, S., Mzoughi, T., and McCauley, V. (2006). Testing the test: Item response curves and test quality. American Journal of Physics, 74(5):449-453.
  22. Moxley, J. H., Ericsson, K. A., Charness, N., and Krampe, R. T. (2012). The role of intuition and deliberative thinking in experts' superior tactical decision-making. Cognition, 124(1):72 - 78.
  23. Ostini, R. and Nering, M. (2006). Polytomous Item Response Theory Models. Sage Publications, Thousand Oaks, California.
  24. Rasch, G. (1961). On general laws and the meaning of measurement in psychology. In Proceedings, Fourth Berkeley Symposium on Mathematical Statistics and Probability, pages 321-334. University of California Press.
  25. Regan, K. and Haworth, G. (2011). Intrinsic chess ratings. In Proceedings of AAAI 2011, San Francisco.
  26. Thorpe, G. L. and Favia, A. (2012). Data analysis using item response theory methodology: An introduction to selected programs and applications. Psychology Faculty Scholarship, page 20.
  27. Tversky, A. and Kahneman, D. (1992). Advances in prospect theory: Cumulative representation of uncertainty. Journal of Risk and Uncertainty, 5:297-323.
  28. Webb, N. (1997). Criteria for Alignment of Expectations and Assessments on Mathematics and Science Education. Monograph No. 6. CCSSO, Washington, DC.
Download


Paper Citation


in Harvard Style

Biswas T. and Regan K. (2015). Quantifying Depth and Complexity of Thinking and Knowledge . In Proceedings of the International Conference on Agents and Artificial Intelligence - Volume 2: ICAART, ISBN 978-989-758-074-1, pages 602-607. DOI: 10.5220/0005288306020607


in Bibtex Style

@conference{icaart15,
author={Tamal T. Biswas and Kenneth W. Regan},
title={Quantifying Depth and Complexity of Thinking and Knowledge},
booktitle={Proceedings of the International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,},
year={2015},
pages={602-607},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005288306020607},
isbn={978-989-758-074-1},
}


in EndNote Style

TY - CONF
JO - Proceedings of the International Conference on Agents and Artificial Intelligence - Volume 2: ICAART,
TI - Quantifying Depth and Complexity of Thinking and Knowledge
SN - 978-989-758-074-1
AU - Biswas T.
AU - Regan K.
PY - 2015
SP - 602
EP - 607
DO - 10.5220/0005288306020607