CLICKERS AND DEEP LEARNING IN A LARGE

UNDERGRADUATE MANAGEMENT COURSE?

Jennifer D. E. Thomas

Pace University, Ivan Seidenberg School of CSIS, 1Pace Plaza, New York, NY 10038, U.S.A.

Danielle Morin, Marylène Gagné

Concordia University, John Molson School of Business, 1455 de Maisonneuve West, Qc. H3G 1M8, Montreal, Canada

Keywords: Higher-order learning, Critical thinking, Deep learning, Technology and learning, Clickers.

Abstract: The idea that clicker technology, a type of electronic polling technology, could have any relationship to

students’ acquisition of higher-order learning skills is seen by many as highly unlikely, especially in large

classes. Nonetheless, that is precisely what the results of this study seem to indicate. In a study of a large

undergraduate Management course in Organizational Behaviour (OB) which blended clicker technology

use, classroom lecture, and online course management content, students’ perceptions of the acquisition of

higher-order thinking skills and team-building skills from the integration of these various resources in the

course were solicited. Clicker technology, the aspect of the course reported in this paper, was favourably

rated for the acquisition of critical thinking skills and problem-solving skills; it was somewhat less so for

acquisition of research skills and creative idea generation, and the team-building skills. They also reported a

preference for learning with clickers than without and felt its use increased student engagement.

1 INTRODUCTION

Clickers, also known as Audience Response Systems

(ARS), have been used in many settings, courses and

levels (King and Robinson, 2009; Watkins and

Sabella, 2008; Mayer, et al., 2009; Berry, 2009;

Morgan, 2008; Trees and Jackson, 2007; Herreid,

2006; Barnett, 2006; Len 2006), but are largely

viewed as more useful for acquiring and testing

shallow knowledge than higher-order learning

(Dangel and Wang, 2008; Radosevich, et al., 2008;

Morgan, 2008). The concept of clicker technology

takes advantage of an electronic polling system

integrated with presentation software, to analyze and

display the distribution of results of responses

obtained from the input of those responding to

multiple-choice or yes/no dichotomous type

questions. This would seem to belie any possibility

of application to more robust acquisition of

knowledge, and of breaking the isolation so often

felt by students in large classes.

The study presented in this paper, sought to

examine these issues by soliciting students’

perceptions of their acquisition of higher-order

learning skills, defined as higher-order thinking

skills and team-building skills, as a consequence of

the integration of clicker technology in a large

undergraduate Organizational Behaviour (OB)

Management course. The results indicate that

students do favourably view the integration of

clickers into their course, and perceive that they

contribute to their engagement and acquisition of

some of the higher-order thinking skills, namely

critical thinking skills and problem-solving skills,

but less so for research skills and creative idea

generation, and for the team-building skills.

2 BACKGROUND

Deep learning can be defined as “the intention to

extract meaning produces active learning processes

that involve relating ideas and looking for patterns

and principles on the one hand (a holist strategy -

Pask, 1976, 1988), and using evidence and

examining the logic of the argument on the other

(serialist). The approach also involves monitoring

the development of one’s own understanding

123

D. E. Thomas J., Morin D. and Gagné M..

CLICKERS AND DEEP LEARNING IN A LARGE UNDERGRADUATE MANAGEMENT COURSE?.

DOI: 10.5220/0003354601230127

In Proceedings of the 3rd International Conference on Computer Supported Education (CSEDU-2011), pages 123-127

ISBN: 978-989-8425-50-8

 2011 SCITEPRESS (Science and Technology Publications, Lda.)

(Entwistle, McCune & Walker, 2000)”. (Entwistle,

2000, p.2). These are the skills that students are

expected to acquire through their tenure in

university, and ultimately to take with them into

their careers. The issue of how to provide

opportunities for this learning in large university

classes is quite vexing, as a consequence,

universities are looking at if, and how, various

technologies might be able to overcome the

obstacles posed to this type of learning, and to

student engagement, by large classes. (Mayer, et al.,

2009; King and Robinson, 2009; Caldwell, 2007;

Trees and Jackson, 2007; Barnett, 2006; Herreid,

2006; Hoffman and Goodwin, 2006). Clickers, are

one such technology that has been adopted but, as

noted previously, are deemed more appropriate for

superficial learning. (Czekanski and Roux, 2008;

Dangel and Wang, 2008; Radosevich, et al., 2008;

Morgan, 2008; Knight and Wood, 2005). The notion

of using clickers in a large undergraduate course to

encourage deep learning therefore, on the surface,

seems ill-advised, or is it?

An investigation of the research literature

produces mixed results. Mayer, et al. (2009) found

significant improvements in students’ exam scores

with the use of clickers in responding to discussion

questions in large psychology classes over those

without use of clickers, and over those without use

of either clickers or discussion questions. Watkins

and Sabella (2008) on their side found that

understanding of questions exhibited in the

classroom using the clickers, did not transfer to

similar questions posed on the actual exam, whereas

Carnaghan & Webb, 2007 and Radosevich, et al.

(2008) found better retention scores at the end of the

semester. Berry’s (2009) study, albeit exploratory,

found that their second exam and final course grades

were significantly higher with the use of clickers,

and that students’ satisfaction feedback supported

their use. Beckes, 2007 found the technology

increased participation and class discussion and was

viewed by students as a favourable way to promote

active learning. These are just a few of the range of

results that research with clickers tends to produce.

For succinct summaries outlining the pros and cons

from the perspective of students and instructors, as

well as best practices recommendations gleaned

from their own studies, and that of other researchers,

see Medina, et al., 2008 and Nelson and Hauck,

2008.

As the field is still wide open, the study in this

paper has focused on students’ perceptions of the

deep learning and engagement they were acquiring

and experiencing with the integration of clicker

technology into their undergraduate Organizational

Behaviour (OB) course. The possibility of

measuring deep learning in any objective way is

open to debate (Entwistle, 2000). Additionally,

instruments to measure deep learning, often because

of their length, are difficult to administer and also to

interpret (see Follman, Lavely and Berger, 1997, for

a comprehensive inventory of instruments). For

these reasons, in this study, the perception of

students was used as a surrogate measure. The main

instrument used was developed by Thomas, 2001

and has been used in several prior studies on clickers

(Morin, et al., 2009; Thomas et al., 2009) and other

technologies (Thomas and Morin, 2006; Thomas,

2005 and Thomas 2001).

Deep learning here was defined as higher-order

thinking skills such as: critical thinking, problem-

solving, research, and creative idea generation, and

student engagement was measured by team-building

skills fostered, such as: communication skills, work

coordination, and team cooperation (Thomas, 2001).

These concepts of higher learning are consistent

with those advocated by Chickering and Gamson,

1987 in Dangel and Wang, 2008, Facione, 2004,

Bloom & Krathwohl, 1956, Anderson & Krathwohl,

2001. These were consistent with the learning

objectives set by the instructor for the course, which

was for students to be able to: 1) discuss the major

concepts relating to human behavior in organizations

and the interrelationships between these concepts; 2)

evaluate these concepts critically in terms of their

utility, applications, and limitations; 3) diagnose and

solve organizational problems by applying material

learned in the course; 4) communicate ideas related

to organizational behavior, both orally and in

writing; 5) effectively collaborate on team projects.

The motivation for this study stems from a need,

at the university in question, as with many other

institutions, to seek cost-cutting measures through

larger classes, without jeopardizing the student

experience. One teacher will teach a larger group of

students with the support of a few teaching assistants

offering tutorials which is more economical than

multi-sections of the same course to be taught by

several instructors. It was hoped that technology,

such as clickers, could provide this bridge. Based on

the nature of the clicker technology, research skills

and coordinating work skills are not expected to be

developed with this tool, as no tasks related to these

skills is performed with the technology. On the other

hand, based on our prior research, it is expected to

affect communication skills and collaboration and,

especially, problem-solving and critical thinking,

while the contribution to collaboration and creative

idea generation is expected to be moderate (Morin et

al., 2009 and Thomas et al. 2009).

CSEDU 2011 - 3rd International Conference on Computer Supported Education

124

3 METHODOLOGY

The course was presented in lecture format during

which the professor gave all information necessary

concerning theories of organizational behavior,

using PowerPoint presentations, which sometimes

included videos and other visuals. The professor also

used the i>clicker

system to increase student

participation. i>clickers

allow students to try out

some questions given by the professor and give

feedback to the professor about their understanding

of the material which was then used to stimulate

class discussion in teams, and as a class. It

represents a tool to increase student participation and

also provides a way to take attendance, as each

clicker has a unique bar code matched with student

names. The professor was also available to meet

with students by appointment and made good use of

the FirstClass

course management system to send

announcements to students. All class materials

(PowerPoint notes, syllabus, project instructions,

etc.) were available on FirstClass

. Lectures were

supported by tutorials outside of class-time, given by

graduate students who were supervised by the

instructor. These tutorials included review of

discussion questions.

Students’ perceptions of the higher-order

thinking skills being developed such as: critical

thinking, problem-solving, research, and creative

idea generation, and student engagement in team-

building skills, such as: communication skills, work

coordination, and team cooperation, were solicited

via a questionnaire. The instrument was composed

of sixteen learning objective questions on a three-

point scale - a lot, moderate and not at all, and eight

questions on general clicker use on a 4-point scale -

agree, strongly agree, disagree, strongly disagree

(Thomas 2001). Six relevant questions from the

official university course evaluation instrument on

satisfaction with the instructional method used in the

course were also collected. Five were based on a 5-

point scale – strongly agree, agree, neither agree nor

disagree, disagree, strongly disagree, and one

question on how students rate the course overall was

based on - excellent, very good, good, fair, poor.

4 RESULTS

4.1 Demographics

There were 149 respondents from a possible 218

students. Males outnumbered females, 54% to 46%,

and the majority were 20-29 years old (72%), with

moderate computer experience (64%).

4.2 Students’ Perceptions of General

Clicker Use

With respect to the distributions of general

perceptions of clicker use, on a 4-point scale - agree,

strongly agree, disagree, strongly disagree, most

students agreed, or strongly agreed, that the use of

clickers in the course contributed positively to their

learning experience. The first four questions were

given this rating by 77%, 86%, 71%, and 77% of the

students, respectively. Eighty (80) percent thought

there was the right amount of use of the technology

in the course, 88% had a positive view of its

contribution to their learning, and 94% felt it was

used to give immediate feedback to students. The

majority, 76% preferred learning with clickers.

4.3 Students’ Perceptions of Deep

Learning and Engagement

The distribution of the learning objective questions

were given on a three-point scale - a lot, moderate

and not at all, and the means column and standard

deviations were calculated by assigning a score 1 to

‘A lot’, a score of 2 to ‘Moderate’ and a score of 3 to

‘Not at all’ and taking the average. By combining

the frequencies corresponding to ‘A lot’ and to

‘Moderate’, the percentage of students who thought

the clickers had a positive impact were calculated.

Most of the students had a positive perception of

the contribution of clickers to their development of

critical thinking and problem-solving skills, 72%

and 65%, respectively. They were approximately

evenly split in their perception of the contributions

to creative idea generation, 50%, and to the team-

building skills – communication skills, 46%,

coordinating work, 47%, cooperation among

students, 45%. They were not at all convinced of its

contribution to developing research skills. Only 32%

felt there was a contribution.

Students had positive perceptions of the class

discussions that ensued following the question and

answer sessions with the clickers, only research

skills was still somewhat lower at 52%. Critical

thinking skills and problem-solving skills was again

the highest at 85% and 79%, respectively. Creative

idea generation was positively perceived by 67%.

The team-building skills were in the 58- 64% range.

CLICKERS AND DEEP LEARNING IN A LARGE UNDERGRADUATE MANAGEMENT COURSE?

125

4.4 Student Perceptions from Course

Evaluation

Responses from questions posed to students on the

university course evaluations were also analysed.

There were 111 complete course evaluations from a

possible 218 students. Students felt they learned a

great deal in the course and that the instructional

method, which included the clicker technology, was

effective and encouraged student engagement. Mean

scores closer to 1 indicate more favourable

perceptions. They also indicate, with means of 2.11,

that learning in a large lecture format, augmented

with tutorials and clickers, was as effective as

courses with smaller class sizes, and would

recommend this format over smaller classes to

fellow students. From a range of 1, being Excellent

to 5, being Poor, the perception of the overall course

produced a mean of 2.37.

5 DISCUSSION

The results of this study are in line with those

observed in similar previous studies (Morin et al.,

2009; Thomas et al., 2009; Berry, 2009; Beckes,

2007). Clickers are, once again, shown to be

favourably viewed by students as supporting the

aspects of deep learning associated with critical

thinking and problem-solving skills, with less

support seen for research skills and creative idea

generation, and student engagement in team-

building skills in general. This favourable view is

enhanced, across all the learning objectives, with the

classroom discussions with peers, and as a class,

which stemmed from the post-polling results

displayed from the use of the clickers. This increase

was by as much as 13-20 percentage points, the most

marked increase being in the perception of research

skills support, 20%, which is interesting. Further,

most students perceived benefits to learning in

general, and to student engagement, from the use of

clickers, and preferred learning with them. They

would also encourage students to take the course in

this large class-size format, with tutorials and

clickers, over smaller classes without them. Overall

then, clickers are viewed as a positive contribution

to the learning experience.

6 CONCLUSIONS

It is clear from the results of this study that clickers

have a role to play in large classes, and in fostering

deep learning and active student engagement,

especially when combined with subsequent

discussions with peers, and the class. Evidently, the

learner-centered approach to integration of

technology per Mayer (2001) is the distinguishing

factor. Success depends on the ability of the

technology to support students’ cognitive processes,

and by the instructional method employed to take

advantage of these, and not by any inherit merit of

the technology, in and of itself. Further research

continues to be needed in this area. The marked

improvement to the perceptions of support for

research skills acquisition when clicker polling is

coupled with subsequent class discussion is

particularly intriguing. It also would be interesting in

future research to link students’ performance in the

course to their perceptions, along with the

instructors’ perceptions.

REFERENCES

Anderson, Lorin, & Krathwohl, D. R. (Eds.) (2001). A

taxonomy for learning, teaching and assessing: a

revision of Bloom's Taxonomy of educational

objectives: Complete edition, New York: Longman.

Barnett, John. 2006. Implementation of personal response

units in very large lecture classes: Student perceptions.

Australasian Journal of Educational Technology,

22(4), 474-494.

Beckes, W. 2006. The “Millionaire” method for

encouraging participation. Active Learn. Higher

Education. 7(1), 25–36.

Bloom, Benjamin S., & David R Krathwohl, 1956.

Taxonomy of educational objectives: The

classification of educational goals, by a committee of

college and university examiners. Handbook 1:

Cognitive Domain, New York: Longman.

Caldwell, J. E 2007. Clickers in the Large Classroom:

Current Research and Best-Practice Tips. CBE-Life

Sciences Education, 6, 9-20.

Carnagan, C. & Webb, A., 2007. Investigating the effects

of group response systems on student satisfaction,

learning, and engagement in accounting education.

Issues in Accounting Education. 22(3): 391-409.

Czekanski, A. J., & Roux, D.-M. P., 2008. The Use of

Clicker Technology to Evaluate Short- and Long-Term

Concept Retention. Paper presented at the American

Society for Engineering Education Middle Atlantic

Section Spring Conference, Baltimore, Maryland.

Dangel, H. L., & Wang, C. X., 2008. Student Response

Systems in Higher Education: Moving Beyond Linear

Teaching and Surface Learning. Journal of

Educational Technology Development and Exchange,

1(1), 93-104.

Entwistle, N., 2000. Promoting deep learning through

teaching and assessment: conceptual frameworks and

CSEDU 2011 - 3rd International Conference on Computer Supported Education

126

educational contexts. Paper presented at the TLRP

Conference, Leicester, England, November, 1-11.

Entwistle, N. J., McCune, V. & Walker, P., 2000.

Conceptions, styles and approaches within higher

education: analytic abstractions and everyday

experience. In R. J. Sternberg & L-F. Zhang (Eds.),

Perspectives on Cognitive, Learning, and Thinking

Styles. Mahwah, N. J.: Lawrence Erlbaum.

Facione, P. A. (2004). Retrieved October 31, 2010, from:

http://www.insightassessment.com

FirstClass

. Retrieved October 31, 2010 from:

http://www.firstclass.com/

Herreid, C. F., 2006. “Clicker” Cases: Introducing Case

Study Teaching Into Large Classrooms. Journal of

College Science Teaching, 36(2), 43-47.

Hoffmann, C. & Goodwin, S., 2006. A clicker for your

thoughts: Technology for active learning. New Library

World. 107(9/10): 422-433.

Hoffmann, C. & Goodwin, S., 2006. A clicker for your

thoughts: Technology for active learning. New Library

World. 107(9/10): 422-433.

i>clicker

. Retrieved October 31, 2010 from:

http://www.iclicker.com/dnn/

King, Samuel O. & Robinson, Carol L., 2009. ‘Pretty

Lights’ and Maths! Increasing student engagement and

enhancing learning through the use of electronic

voting systems. Computers & Education, 53, 189-199.

Mayer, R.E., 2001. Multimedia Learning. New York,

Cambridge University Press.

Mayer, R.E., et al., 2009. Clickers in college classrooms:

Fostering learning with questioning methods in large

lecture classes. Contemporary Educational

Psychology, 34, 51-57.

Mazur, Eric, 1997. Peer Instruction: A User’s Manual.

Prentice Hall Series in Educational Innovation, Upper

Saddle River, NJ, Prentice Hall

Medina, M. S., Medina, P. J., Wanzer, D. S., Wilson, J. E.,

Er, N., & Britton, M. L., 2008. Use of an Audience

Response System (ARS) in a Dual-Campus Classroom

Environment. American Journal of Pharmaceutical

Education, 72(2), Article 38.

Morgan, R. K., 2008. Exploring the Pedagogical

Effectiveness of Clickers. Insight: A Journal of

Scholar Teaching, 3, 31-36.

Morin, D., Thomas, J.D.E., Barrington, J., Dyer, L. and

Boutchkova, M., 2009. The ‘Clicker’ Project: A

Scholarly Approach to Technology Integration. Real

Learning Opportunities at Business School and

Beyond, Advances in Business Education and

Training, 2, Peter Daly and David Gijbels, eds.,

Springer Publishing, NY, 97-108.

Nelson, M. L., & Hauck, R. V., 2008, April 1. Clicking to

Learn: A Case Study of Embedding Radio-Frequency

based Clickers in an Introductory Management

Information Systems Course. Journal of Information

Systems Education, 19(1), 55-64.

Pask, G., 1988. Learning strategies, teaching strategies and

conceptual or learning style. In R.R. R.R. Schmeck

(Ed.), Learning strategies and learning styles. New

York: Plenum Press.

Pask, G., 1976. Styles and strategies of learning. British

Journal of Educational Psychology, 46, 128-148.

Radosevich, D. J., Salomon, R., Radosevich, D. M., &

Kahn, P., 2008. Using Student Response Systems to

Increase Motivation, Learning, and Knowledge

Retention. Innovate, 5(1).

Thomas, J. D. E., Morin, D. and Cassidy, R., 2009.

Clickers and Critical Thinking. Proceedings of the

IADIS International CELDA Conference, Rome, Italy,

Nov. 20-22.

Thomas, Jennifer, Danielle Morin, Janette Barrington,

Linda Dyer, Maria Boutchkova., 2008. The ‘Clicker’

Project: A Scholarly Approach to Technology

Integration. Proceedings of the 15th Educational

Innovation in Economics and Business – EDiNEB

Conference, Malaga, Spain.

Thomas, J. D. E. and Morin, D., 2006. Technological

Supports and Students' Perceptions of Acquisition of

Team-Building and Thinking Skills. Proceedings of E-

Learn 2006--World Conference on E-Learning in

Corporate, Government, Healthcare, and Higher

Education, Honolulu, Hawaii, USA, October 13-17,

2436-2441.

Thomas, J. D. E., 2005. Support of Students’ Acquisition

of Knowledge Management Skills in Academia.

Proceedings of 5th Annual Hawaii International

Conference on Business, Honolulu, Hawaii, May 26-

29, 3016-3019.

Thomas, J. D. E., 2001. “Technology Integration and

Higher-Order Learning”, in Proceedings of

Conference in Advanced Technology in Education

Conference, Banff, Calgary, Canada, May.

Trees, April R. & Jackson, Michele H., 2007. The learning

environment in clicker classrooms: student processes

of learning and involvement in large university-level

courses using student response systems. Learning,

Media and Technology, 32(1), 21-40.

Watkins, E. P. & Sabella, M. S., 2008. Examining the

Effectiveness of Clickers on Promoting Learning by

Tracking the Evolution of Student Responses.

Proceedings of Physics Education Research

Conference, Eds. C. Henderson, M. Sabella, and L.

Hsu, 223-226.

CLICKERS AND DEEP LEARNING IN A LARGE UNDERGRADUATE MANAGEMENT COURSE?

127