Are We Ready for Problem-based Learning? A Proposal of

Institutional Diagnosis in Computing Higher Education

Osmário de Carvalho Santos Filho and Simone C. dos Santos

Centro de Informática, Federal University of Pernambuco, Recife, Brazil

Keywords: Computing Higher Education, PBL, Institutional Diagnosis, Opinion Survey.

Abstract: In Computing Higher Education (CHE), the desired transformation of traditional teaching and learning

methods, almost always based on the transmission of information and content-based curricula, has been the

objective of several educational institutions that wish to combat students' demotivation and dropout. Among

successful approaches, Problem-Based Learning stands out as one of the most effective and radical methods

regarding pedagogical innovations. While the implementation of the PBL means a great opportunity to

achieve better educational performance, it also represents many challenges that can only be managed if they

are first known and understood. In this context, the motivation for this study comes from the following

research question: "How to know if an institution at CHE is ready to implement the PBL?". As a response, an

institutional diagnostic model regarding the adoption of PBL is proposed. From an opinion survey with 38

technical educational institutions in computing, involving 302 participants, the results showed that the model

reached its objective, allowing the identification of favorable, warning, and critical points regarding the

adoption of PBL in these institutions.

1 INTRODUCTION

In Computing Higher Education (CHE), the desired

transformation of traditional teaching and learning

methods, almost always based on the transmission

of information and content-based curricula, has been

the objective of several educational institutions that

aim to prevent students' demotivation and dropout

(Yang and Choi, 2017). There are diverse examples

of successful approaches, in particular those using

authentic problems, attractive technologies, and a

learning environment that reflects the labor market

to promote the engagement and motivation of the

students (Babori et. al., 2016; Kemavuthanon, 2017;

Martins et. al., 2018; Santos and Silva, 2018). Also,

several solutions seek to systematize teaching and

learning methods into methodologies, with concrete

proposals to help transform the traditional classroom

into a practical and stimulating environment. Thus,

teaching methodologies such as Problem-Based

Learning (PBL) (Yu, 2005; Jaryani et. al., Zhao and

Liu, 2011; Panwong and Kemavuthanon, 2014;

Ibrahim and Halim, 2014; Mäenpää et. al., 2017),

and its variants like Project-Based Learning (Bell,

https://orcid.org/0000-0002-7903-9981

2010), Case-Based Learning (Srinivasan et. al.,

2007) among others, have become popular in

computing education, bringing important benefits

such as increased engagement, motivation, and

development of technical and non-technical skills so

important to the professional life of the student.

Born in the medical education and defined as "a

learner-centered constructivist method that uses

real problems as a learning object" (Savery and

Duffy, 1994). PBL is considered one of the most

effective and radical methods regarding pedagogical

innovations. It advocates profound changes that

involve the institution as a whole, transforming its

learning environment, teachers' and students'

attitudes, curricula, operational and managerial

resources, infrastructure, assessment processes,

relationships with the labor market, and,

consequently, budget (Fink, Enemark and Moesby,

2002). While the implementation of the PBL means

a great opportunity to achieve better educational

performance, it also represents many challenges that

can only be managed if they are first known and

understood. In this context, it is necessary to

understand the challenges faced by all stakeholders

Filho, O. and dos Santos, S.

Are We Ready for Problem-based Learning? A Proposal of Institutional Diagnosis in Computing Higher Education.

DOI: 10.5220/0010433604130424

In Proceedings of the 13th International Conference on Computer Supported Education (CSEDU 2021) - Volume 1, pages 413-424

ISBN: 978-989-758-502-9

413

in the educational process based on PBL and raise

the requirements to be adopted this approach

effectively. With this objective, the NEXT

(iNnovative Educational eXperience in Technology)

research group has been investigated the use of the

PBL approach in computer education for more a

decade (Oliveira, Santos and Garcia, 2013; Santos et

al., 2020), developing methodologies that can

support the pedagogical team in adopting this

approach. However, in recent consultancies in CHE

institutions that wish to transform their curricula and

traditional pedagogical proposals through the PBL,

it was encountered an imminent difficulty: the

institutions were not able to adopt the PBL and, even

more critical, they did not know how to start. With

this motivation, a master's research (Santos Filho,

2020) sought to investigate solutions that could help

higher education institutions identify their ability to

implement the PBL, making evident its favorable,

critical aspects and warning signs. From a

systematic literature review concerning PBL applied

in computing education from 1999 to 2019 (Santos

et al., 2020), several frameworks for applying PBL

were identified, proposing the key elements for

implementing the method and conducting the

classroom approach as described in (Yu, 2005;

Jaryani et. al., Zhao and Liu, 2011; Panwong and

Kemavuthanon, 2014; Ibrahim and Halim, 2014;

Babori et. al., 2016; Kemavuthanon, 2017; Mäenpää

et. al., 2017; Martins et. al., 2018; Santos and Silva,

2018). But no solution was found focusing on

investigating the situation of an educational

institution regarding the requirements necessary

for/adopting the PBL. So, the motivation for this

study comes from the following research question:

RQ - "How to know if an institution at CHE is ready

to adopt the PBL?".

In this context, this study proposes a model of

institutional diagnosis intending to investigate this

situation, considering two target-public:

teachers/tutors, giving their opinion on the essential

elements for the adoption of the method under the

pedagogical aspect, also reporting their perceptions

related to students in general; course

coordinators/managers, who have a more systemic

view of the educational institution and, therefore,

can contribute with their vision under structural and

organizational aspects, complementary to the

teachers' view. From these visions, the educational

institution will be able to identify its strengths and

improvement points regarding the implementation

of PBL, supporting its decisions of changes within

its pedagogical planning and educational strategy.

To report the early results of this research, this

paper is divided into 6 sections. After this brief

introduction, Section 2 presents the main PBL

references used as the foundation of the model,

highlighting concepts, challenges, and requirements

for adopting this approach. Section 3 describes the

research methodology and its steps. The institutional

diagnosis model is presented in Section 4. To assess

the practical applicability of this model, Section 5

describes the results and discussions of an opinion

survey with 38 educational institutions in

computing, which answered the proposed diagnosis

involving their teachers and course coordinators'

team. Finally, Section 6 discusses the conclusions

and future work.

2 THEORETICAL REFERENCES

According to (Ribeiro, 2008), the PBL is a teaching

and learning method that makes use of real problems

to motivate students to learn concepts, procedures,

and attitudes that will be important for their future

performance as citizens and professionals. As

emphasized in (Melo, 2013), the potential of the PBL

in the teaching and learning process of students is

capable of developing important skills such as self-

confidence, problem-solving, and autonomy. Thus,

the focus of the teaching process is the student, who

is stimulated to learn more autonomously and

cooperatively with his colleagues. The PBL method

still requires a more active posture from students,

choosing the best way to learn, conducting research,

and using educational resources that go beyond the

classroom. This also encourages students to reflect

critically on what is proposed to solve the problem,

important skills for future computer professionals in

constant learning due to technological advances. In

this context, the role of the teacher is to monitor and

provide feedback on learning, in addition to

identifying the difficulties encountered by students,

facilitating and guiding the progress of their learning

(Enemark, Kolmos and Moesby, 2006).

PBL follows some principles such as an authentic

learning environment and simulation of the situation

found in the professional environment, the use of real

problems as a learning object, the monitoring of

evaluation by continuous feedback (Ribeiro, 2008).

In (Santos et al., 2013), ten principles were defined

for the teaching of Computing that founded a

methodology called xPBL (Santos et al., 2014), as

shown in Figure 1.

CSEDU 2021 - 13th International Conference on Computer Supported Education

414

Figure 1: Ten PBL Principles and the xPBL elements.

The xPBL methodology defines five manageable

elements for PBL planning: 1) Problem; 2) Learning

Environment; 3) Human Capital, that includes

students, pedagogical team and market partners; 4)

Content, as a guide and support to solve problems;

And 5) Processes, concerning educational objectives

and assessment processes. These elements are aligned

with the ten PBL principles (as shown in Figure 1)

that are fundamental for the implementation of an

authentic PBL in computer education.

2.1 Challenges in PBL

The adoption of the PBL approach, as it is not trivial,

requires a series of changes, both in the attitude of

teachers and students (Moesby, 2004; Santos and

Silva, 2018; Aldabbus, 2018). According to (Moesby,

2004), it is common to evidence in the application of

the PBL method the resistance of teachers, who often

lack knowledge and experience in the teaching

methodology, and for this, need to be trained to face

the obstacles of the method.

Moreover, teachers should always be aligned with

the PBL teaching process, considering that this

approach requires a learning environment with a

flexible curriculum geared to the demands of the

professional market, in which the relevant issues are

raised by the students and no longer by the teacher

(Enemark, Kolmos and Moesby, 2006). As for the

students, according to (Santos and Silva, 2018), some

difficulties were observed in the application of the

PBL method: a certain initial discomfort with the

changes; lack of bibliographical research during the

activities; lack of ability with technological

resources; little involvement of the students in

groups, and student priorities. However, there are also

factors external to the learning environment that make

it difficult to implement the PBL, as mentioned by the

students, such as adequate time management, which

causes a drop-in student performance and project

quality (Hsu and Lin, 2016). In the application of the

PBL, the dedication of students inside and outside the

classroom is fundamental, and their active

participation in the learning process.

It is also necessary to face many obstacles in the

planning for the implementation of the PBL

methodology, such as the development of problems

or projects, the planning of classes with a new

approach, the lack of technological tools and trained

professionals to practice this method in the institution

(Herold, 2019). The initial lack of security in the

process of change to the PBL method needs to be

overcome, as well as the difficulties of aligning the

time between theory and practice, the adequacy of the

curriculum, the availability of financial resources, the

evaluation, and development of skills of the tutor

teacher (Souza and Dourado, 2015). In (Aldabbus,

2018), the author reinforces these challenges

emphasizing the lack of technology, flexible

schedules, and absence of curriculum policy as

difficulties in adopting the method. The study

(Krusche et al., 2016) reports that there are

institutional factors that hinder the adoption of the

PBL method such as infrastructure, class planning,

and didactic resources, in addition to an adequate

environment for the development of team activities,

technical-pedagogical support, and hiring of

monitors/tutors.

2.2 Requirements to Apply PBL

To overcome the challenges of adopting the PBL, it

is necessary to plan all the processes and resources

before implementation, assisting in the correct use of

the method, in the alignment of theory and practice,

and respecting the principles of this approach (Santos

et al., 2013). When planning to change a teaching and

learning process, it is important to carry out a

diagnosis in order to characterize the respective

educational institution that will face the changes.

Thus, it is possible to identify the main aspects

required for the effective implementation of the

approach.

Among the aspects to be analyzed, it is important

to point out: collaboration between courses from a

curriculum that allows the integration of knowledge

acquired by students; a central committee for

curriculum planning, to manage the content and

topics to be addressed in the course; training of

teachers to improve their didactics and learn new

teaching strategies; planning of teaching in small

groups and; availability of a period for study

(Alshaye, Tasir, and Jumaat, 2019). The

implementation of PBL also requires the

understanding and participation of various actors

(teachers, students, researchers, managers, real

clients) who will be active in the processes, therefore

the need to focus on team development and

curriculum development of PBL (Fink, Enemark and

Moesby, 2002; Koray and Koray, 2013). The PBL

Are We Ready for Problem-based Learning? A Proposal of Institutional Diagnosis in Computing Higher Education

415

curriculum should have a flexible format, be student-

centered, be interdisciplinary, have real problems,

and focus on research and investigation, promoting

critical thinking and development of technical and

non-technical skills (Hmelo-Silver, 2004; Melo,

2008). Content planning should focus on practical

classes, with the learning process focused on solving

problems that stimulate discussion, challenge

students, and stimulate their creativity (Krusche,

2016; Barron and Wells, 2013).

Other prerequisites in the process of adopting the

PBL are the provision of free space for self-learning,

a review of the role of teachers and departmental

autonomy, availability of financial resources to invest

in infrastructure, and aligning student selection

criteria to the profile expected by the PBL approach.

Santos et al. (2020) reinforce that the implementation

of the PBL method in an environment with traditional

teaching triggers the need for a series of changes,

such as the adaptation of the curriculum, the

formatting and organization of the learning

environment and the use of technologies.

PBL promotes many benefits for the student

learning from the alignment of academic training with

the requirements of the professional market (Herold,

2019). At this point, the PBL requires that the

institution has approximation with companies in the

labor market to provide teachers with new ideas and

relevant problems to be addressed in the classroom.

There are organizations that perform PBL method

consulting and analyze the institution at the

organizational, pedagogical and educational level,

thus supporting the design of a PBL-based

curriculum, evaluation processes and,

implementation of organizational and pedagogical

aspects (Moesby, 2004). Moesby reports that many

educational institutions analyze the needs of students

today, and this brings about various changes at

various levels: personal, organizational, and cultural.

These changes require the development of an action

plan, involving not only principals, but all those

involved in the educational process, such as teachers,

tutors, coordinators, and managers.

3 RESEARCH METHOD

This study has used a mix of methods, with emphasis

on qualitative research and descriptive approach.

According to Patton in (Patton, 2002), research is

said to be qualitative when it aims to investigate

what people do, know, think, and feel through data

collection techniques such as observation,

interviews, questionnaires, document analysis,

among others. Conducted by the NEXT research

group focusing on studies on the PBL approach

applied to computing teaching, this research was

motivated by direct observations in consulting

activities to support educational institutions in the

transformation of traditional methods to PBL, in

general, carried out during the process of

educational innovation and curriculum changes. At

this time, an analysis of the institution is essential to

identify the requirements for the implementation of

PBL. Figure 2 shows the research steps and the main

methods.

Figure 2: Methodological scheme.

After an ad hoc literature review seeking solutions

to help this investigation, the central research

question was defined: RQ - "How to know if an

institution at CHE is ready to adopt the PBL?

To find objective answers, this research question was

divided into three secondary questions:

 Q1 - What are the favorable points for adopting

the PBL?

 Q2 - What are the critical points for PBL

adoption that can negatively impact its

implementation?

 Q3 - What are the warning points that the

institution needs to understand better to make

new decisions?

From these research questions, two objectives

were defined: (1) the conception of a model to assess

the ability to apply PBL and; (2) the application of

this model in real educational institutions to verify

its effectiveness. To achieve these objectives, two

research cycles were designed, as shown in Figure

CSEDU 2021 - 13th International Conference on Computer Supported Education

416

The first cycle had focus on the "conception of

the institutional diagnosis", which included

searching the relevant literature on the adoption of

PBL in the teaching computing (Oliveira, Santos

and Garcia, 2013; Santos et al., 2020),

understanding of the main challenges encountered

by the institutions and the essential requirements for

the successful implementation of PBL, as discussed

in sections 2.1 and 2.2, respectively. From these

references, three dimensions of the model and its

aspects were identified:

 Pedagogical, with the five aspects Problem,

Environment, Human Capital, Content and

Process, based on the methodology cited in

Section 2;

 Structural, which includes Infrastructure and

Curriculum aspects, and;

 Organizational, with the Political and

Evaluation aspects.

Besides, two target-public were defined: the

teacher/tutor and the course coordinator/manager,

both key actors in the process of transforming the

pedagogical approach. At this point, the student

inquiry was also considered, but it was decided to

capture the students' perspectives from their

teachers/tutors. As a result, a set of 85 assertions were

identified, with respective rationales and literature

references. After that, the assertions were analyzed by

two PBL specialist, discarding those outside of the

context and grouping others and, finally, defining 60

assertions distributed in two questionnaires of 30

questions each one, addressed to the two target-

public. Each assertion has its rationale and literature

references (URL in Appendix).

The second cycle was guided by the objective of

applying the model. To do so, this research has used

the opinion survey method proposed by Kitchenham

and Pfleeger (2008), defined in six stages: Setting

the objectives; Survey design; Developing the

survey instrument (i.e. the questionnaire);

Evaluating the survey instrument; Obtaining valid

data and; Analysing the data. The research objective

was to evaluate and identify the aspects required for

the adoption of PBL in higher education institutions

in computing, in both undergraduate and graduate

levels, checking the current situation with teachers

and course coordinators.

To create the research instrument, the

questionnaires and assertions defined in Cycle 1

were analyzed by seven specialists (6 Ph.D. and 1

Ph.D. student, all with more than 5 years of

experience in PBL) under the aspects of syntax,

semantics, level of the relevance of the questions

and completeness of the questionnaire. From this

evaluation, some updates, groupings, and

suggestions of new assertions were recommended:

an assertion to verify if the self-regulation and

metacognition of the students can be evidenced in

the respective institution; an assertion on feedback

in the evaluations of the teachers and other actors.

After the experts' evaluation, two forms

(Google) were created, each directed to a type of

participant (teacher or coordinator). The answers

were based on Likert's ordinal scale: 1) Totally

Disagree, 2) Partially Disagree, 3) Neither Agree

nor Disagree; 4) Partially Agree, and; 5) Totally

Agree.

For data analysis, consolidated charts were

generated for each question, with the appropriate

proportions of each answer chosen. For a qualitative

discussion of the results, three status were defined:

"favorable" (agrees), "warning" (neutral), and

"critical" (disagrees), facilitating the analysis of

responses towards the secondaries research

questions and future decision making. The results of

this assessment will be discussed in more detail in

Section 5.

3.1 Limitations and Threats to Validity

It is important to highlight some limitations of this

research. According to “Promoting Institutional &

Organisational Development” guideline in (DFID,

2003), a diagnosis must be a thorough task, based on

a careful selection of interested parties. However,

preliminary or partial analysis in the first contact

with the investigated institution can serve as a

reference base to have a more comprehensive and

in-depth diagnosis later.

Another important point is that an institutional

diagnosis is focused on a particular organization,

requiring time and effort to apply, discuss, and carry

out future interventions. This research focused

solely on the diagnosis application stage, using the

opinion survey method for large-scale application,

to understand its usefulness regarding the

implementation of PBL, making evident the

favorable, critical points and warning signs

institutions.

Finally, to facilitate the processing of the data

collected in the diagnosis, a Likert scale with five

values from "totally disagree" to "totally agree" was

used. However, the evaluation of the results adopted

a qualitative interpretation of the data, identifying the

favorable, critical points and warning signs, in

response to the secondary questions Q1 to Q3. Thus,

Are We Ready for Problem-based Learning? A Proposal of Institutional Diagnosis in Computing Higher Education

417

no statistical method was applied in this case, which

does not prevent it from being adopted in future

works.

4 PBL INSTITUTIONAL

DIAGNOSIS

The PBL institutional diagnosis is structured

according to Figure 3, while Table 1 and 2 present the

assertions for the Teacher/Tutor and

Coordinator/Manager questionaries, respectively.

Figure 3: Structure of the Institutional Diagnosis.

The scope of the diagnosis is focused on

computing courses, considering the PBL references

used for its definition and the CHE context. Two

questionnaires have created addressing collaborators

of the educational institution in the role of

teacher/tutor and coordinator/manager. Each

questionnaire has 30 assertions, in the format of

Google forms.

Regarding identification of the respondents, the

following fields were also included: name, institution,

position/function, e-mail, course modalities, course,

duration (in hours), teaching methodology, PBL

knowledge, and experience in PBL. The “course” and

“duration” fields are only included in the teacher’s

form, and the “teaching methodology” field asks what

type of methodology the teacher uses within three

alternatives (traditional, active, and hybrid), while in

the coordinator’s form asks if the institution

recommends any type of teaching methodology.

It is important to emphasize that the results of this

diagnosis provide transparency about the situation of

the institution from the perception of its employees.

With this information, discussions, and referrals can

be made, for example, regarding the training of

teachers, approximation with market companies, or

even acquisition of specific technologies.

Table 1: Teacher/Tutor Questionary.

PROBLEM

A1 The activities developed in the computing course use

real problems as a motivating element for student

learning.

A2 Students in the computing course seek to take

ownership of the problem to be solved, becoming

responsible for the learning itself.

A3 The problems, problem situations, or hypotheses

presented in the course are based on real contexts.

A4 Problems attributed to learners are stimulating as a

challenge to reasoning.

A5 In the course, problems or problem situations have

similar complexity to those found in real contexts.

A6 Students interact with real customers and users who

have problems to solve.

ENVIRONMENT

A7 The environment stimulates social skills and problem-

solving required by the labor market.

A8 The students’ learning environment seeks solutions to

real client demands in order to reflect conditions similar

to the professional market.

A9 Students demonstrate a professional attitude in the

learning environment in order to maintain the

authenticity of the labor market in the educational

environment.

A10 The students’ learning process is implemented in an

environment that provides conditions for students to

assume responsibilities assigned to certain functions

and positions in the Computing.

A11 The physical and technological infrastructure of the

learning environment stimulates and favors the

execution of learning dynamics, group work, and

collaborative activities.

A12 The learning environment fosters the understanding of

the concepts that will be aimed at solving problems in

the labor market context.

CONTENT

A13 In the course of computing, there is alignment between

theory and practice.

A14 There is integration between the courses in favor of the

educational proposal of the curriculum.

A15 The content planned to be addressed is based on

projects and practical activities.

A16 The content of the course is designed to be used as a

guide and support for problem-solving, therefore

flexible.

A17 The course structure is planned with content that will

generate dynamics in student learning inside and

outside the classroom.

A18 The course plan can be adjusted as the teaching

methodology changes.

CSEDU 2021 - 13th International Conference on Computer Supported Education

418

Table 1: Teacher/Tutor Questionary (cont.).

HUMAN CAPITAL

A19 Teachers monitor the resolution of problems by

students in order to stimulate resolution using

interdisciplinary knowledge.

A20 In solving real problems, there is the participation of the

real client to carry out the collaboration in the

requirements of the problem.

A21 The students demonstrate to be active and autonomous

in the construction of their knowledge from the solution

of the proposed problems.

A22 In problem-solving tasks, teams or small groups are

formed with 4 to 7 students to promote a greater

contribution among members.

A23 The students' learning process has a multidirectional

characteristic, that is, there is an effective relationship

between students, teacher, tutor, and real client.

A24 The pedagogical coordination seeks, together with the

teachers, to develop more collaborative and

multidirectional projects for their students.

PROCESS

A25 Teachers plan the content to be learned and evaluations

should reflect on the content assimilated.

A26 Teachers carry out continuous evaluation and

monitoring of the teaching and learning process, aiming

to help students in their own reflection on learning from

their difficulties and feedback.

A27 The institution seeks appropriate strategies for

monitoring and evaluating student learning.

A28 In the teaching-learning process, characteristics of

metacognition and self-regulation can be evidenced in

students.

A29 The teachers evaluate the learning outcomes with the

objective of following up the student in solving process,

before putting it into practice.

A30 The educational planning meets the students'

expectations regarding the objectives, goals, or

expected results of the course.

5 APPLYING PBL DIAGNOSIS:

RESULTS AND DISCUSSION

The proposed diagnosis was applied in Federal

Institutes of Education, Science, and Technology

(FITs), institutions created by law 11.892/08 to

promote basic, professional, and higher education,

but also to offer education in different teaching

modalities (Santos Filho, 2020). Each institute is led

by a hierarchy of directors made up of rector, pro-

rector, directors, coordinators, teachers, and

Table 2: Coordinator/Manager Questionary.

INFRASTRUCTURE

A1 The course environment (classroom, meeting room,

etc.) and communication tools (email, instant message,

groupware, etc.) facilitate interaction and collaboration

of students.

A2 The course environment provides free space and

sufficient time for self-learning by the student.

A3 There are sufficient materials, technological resources,

and systems in the course to assist teachers during

learning.

A4 Teachers demonstrate sufficient skills and capacity to

perform activities as tutors.

A5 Teachers play a role as facilitators of learning and

assist in the process of learner autonomy and

independence.

A6 Tutors are available in the course to provide group

mentoring.

A7 There is the role of the real client (in general, external

collaborators), participating in the teaching and

learning process, providing problems to be solved, and

accompanying their resolutions.

A8 In the course, there are trained collaborators available

to help in the tutoring process.

POLITICS

A9 The institution usually conducts training for teachers

and teaching staff for possible evolutions in the

teaching process.

A10 The course involves the participation of companies in

academic projects, to make the teaching and learning

process closer to reality.

A11 In the student selection process, criteria such as

interpersonal, autonomy, and creativity are considered,

such as teamwork skills and student proactivity.

A12 Teachers easily plan, implement, and evaluate their

courses.

A13 There is resistance from teachers when there are

changes in curriculum or pedagogical methodology.

A14 There is resistance from students when there are

changes in curriculum or pedagogical methodology.

A15 There is a culture of "learning by doing" in the

institution, stimulating professional practice.

A16 Class planning is carried out collaboratively, involving

the pedagogical team (e.g., teachers, tutors, real

client).

A17 The institution's budget includes resources for

practical approaches and continuous monitoring.

Are We Ready for Problem-based Learning? A Proposal of Institutional Diagnosis in Computing Higher Education

419

Table 2: Coordinator/Manager Questionary (cont.).

CURRICULUM

A18 The adaptation of the course curriculum to a new

pedagogical methodology may occur if it is necessary.

A19 The course brings interdisciplinarity and/or

multidisciplinarity to the classroom.

A20 The curriculum is flexible, providing a consistent body

of basic knowledge and autonomy to the student in the

choice of their specialties.

A21 The class has a flexible schedule, being able to adjust

according to the teacher's needs.

A22 There is a lack of alignment between the class load

and the content to be taught by the teacher.

A23 The course curriculum encourages problem-solving

and self-directed learning.

A24 The course is based on a curriculum that stimulates

interdisciplinary and/or multidisciplinary practice.

EVALUATION

A25 During the course, there is continuous monitoring and

feedback from teachers to students.

A26 There is effectiveness in the course evaluation process

through various evaluation criteria.

A27 The institution performs evaluations on the

pedagogical methodology and the level of tasks

required by the teachers.

A28 The evaluation system provides feedback on the

teachers' work and the students' learning.

A29 The course's subjects allow for evaluations of the

learning process and self-assessments of students.

A30 The institution performs evaluations regarding

content, process, performance, satisfaction, and results

generated from the course.

administrators. With a multi-campus structure of

more than 661 units, there are 38 FITs distributed in

various regions of Brazil based on professional and

technical education. Although not all campuses offer

courses in the area of computing, there are several

courses in this area such as Computer Technician,

Information Technology Management, Information

System Analysis and Development, Bachelor of

Computer Science, Databases, among others (Santos

Filho, 2020).

All 38 FITs responded to the survey, totalling 302

respondents (222 teachers and 80 coordinators).

Considering that teachers can teach in more than one

academic level, most of them work in technical (92%)

and higher (75%) courses, while a small portion of

them work in graduate (14%) and extension (22%)

courses. This scenario was similar to the coordinators

with technical (60%), higher (49%), graduate (3%),

and extension (8%). About 88% of the teachers stated

that they apply active methodologies in their classes,

while only 12% use traditional methodology. On the

other hand, 62.5% of the coordinators answered that

the institution where they work does not recommend

a teaching methodology, while 22.5% answered that

they recommend the active methodology and 15%

indicate the traditional methodology. As for

knowledge and experience in PBL, 57% of the

teachers and 50% of the coordinators stated that they

have median knowledge, with the level of experience

also median even lower (42% of the teachers and only

30% of the coordinators).

5.1 Results of the Teacher Survey

Figure 4 shows an overall result from the 222

teachers.

On the “Problem” aspect, most stated using real,

stimulating, and sufficiently complex problems as an

element of study (favorable), but were uncertain

about the appropriation of the problem-solving

process by the student (warning).

Figure 4: Teachers’ results.

It was also evident the lack of interaction of

students with real clients and users, compromising the

authenticity of the learning environment (critical

points). In PBL, real clients make it possible to build

effective solutions through interactions that help

students in the resolution process, providing feedback,

and evaluating partial results (Santos et al., 2020).

31,5%

12,2%

28,4%

32,9%

18,9%

7,7%

14,9%

11,3%

2,7%

16,8%

13,6%

33,8%

23,1%

23,9%

16,3%

16,7%

22,2%

24,8%

6,3%

5,4%

26,2%

10%

6,8%

36%

24,3%

21,2%

5,5%

12,7%

10%

40%

29,7%

37,8%

40,1%

34,2%

10,8%

34,7%

23,4%

16,7%

25,5%

28,6%

41,6%

47,7%

37,1%

40,1%

34,4%

32,6%

34,8%

41%

12,7%

17,6%

27,2%

16,7%

17,6%

39,2%

35,6%

41%

24,5%

38,2%

36,1%

25,2%

38,7%

24,8%

21,6%

34,2%

25,7%

32,9%

40,5%

40,8%

37,3%

29,6%

32,5%

14,8%

23,5%

27,4%

32,6%

27,2%

24,3%

19,4%

41,7%

26,3%

31,7%

21,3%

18,9%

27,5%

27,9%

42,7%

33,7%

43,4%

8,1%

14,4%

8,6%

9,5%

28,8%

14,4%

16,7%

27,1%

23,6%

18,6%

10%

2,3%

14,5%

7,7%

14%

15,4%

12,2%

9,5%

32,6

27,6%

14%

29,4%

26,2%

4,5%

9,9%

8,1%

19,1%

13,6%

7,8%

0,5%

0,4%

3,2%

27%

3,1%

8,1%

12,7%

8,6%

6,4%

2,3%

1,4%

1,8%

0,9%

2,7%

0,4%

7,7%

6,3%

12,2%

28,1%

1,4%

2,7%

1,8%

8,2%

1,8%

2,7%

0 102030405060708090100

A10

A11

A12

A13

A14

A15

A16

A17

A18

A19

A20

A21

A22

A23

A24

A25

A26

A27

A28

A29

A30

Teachers' Evaluation

Totally Agree Partially Agree Neutral

Partially Desagree Totally Desagree

CSEDU 2021 - 13th International Conference on Computer Supported Education

420

Regarding the “Environment” aspect, teachers

partially agreed that the environment stimulates

social skills, solving problems required by the

professional market and fostering important concepts

(favorable), however, the authenticity of the

environment was again questioned, considering that

problems do not always reflect real market demands,

compromising the student's professional posture

(critical). According to (Bell, 2010), the PBL

promotes the construction of knowledge and skills

considered relevant to their professional practice,

therefore, an inappropriate environment can impact

this construction.

Concerning the “Content” aspect, as main

favorable points are the alignment between theory

and practice, integration between related courses,

content approaches using practical projects, and

appropriate subjects. Being institutions focused on

professional education, it is understood that the

course curricula have already been built for this

purpose. Two points indicated a warning sign:

flexibility of content in supporting the problem-

solving process and content that promotes dynamic

learning inside and outside the classroom. In PBL, the

content needs to be flexible and focused on the

problem-solving process (Krusche, 2016), with

teaching that is much more "learning to learn" than

"knowing a concept" that, far from practice, can

easily be forgotten.

“Human Capital” was the most critical aspect.

Although teachers follow the resolution of problems

and stimulate group work with the formation of small

teams of students (favorable), many showed

uncertainty about the active posture and autonomy of

students. Critical points are the lack of participation

of the real client, a multidirectional relationship

where everyone learns from everyone and stimulation

of collaborative projects by the course coordinator.

Finally, the “Process” aspect showed as favorable

points the content planning, evaluation, and

continuous monitoring by teachers and institutions.

However, warning signs for learning with

characteristics of metacognition and self-regulation

and a course planning that meets the expectations of

the students. The characteristics of metacognition and

self-regulation are present in the PBL approach,

enabling the student to perform self-reflection and

perform monitoring and evaluation of their learning

(Santos and Silva, 2018).

5.2 Results of the Coordinators Survey

Figure 5 presents the overall result of the

coordinators' responses.

In the "Infrastructure" aspect, the following are

highlighted as favorable points: the environment and

technological tools that facilitate interaction and

collaboration; a free space and sufficient time for self-

learning of the student; the existence of resources;

teachers with skills for tutoring activities and teachers

in the role of learning facilitators. The high bias in the

agreement of these points and the rate of uncertainty

in the first five assertions indicates a warning sign for

many institutions. Critical points were the absence of

tutors to support the teacher, collaboration with real

clients, and training in tutoring activities.

Concerning the "Politics" aspect, favorable points

are the realization of the training, ease in planning,

implementing and evaluating, and the culture of

"learning by doing" present in the institution.

However, once again the lack of interaction with the

labor market became evident, besides the absence of

a student selection process more appropriate to the

pedagogical approach and the collaborative planning

of courses.

Figure 5: Coordinators’ results.

It is also worth highlighting a warning regarding

the resistance of teachers and students to changes in

the curriculum or pedagogical methodology, and the

need for a planned budget for educational practices.

Regarding the “Curriculum” aspect, the following

points stand out as favorable: the adequacy of the

curriculum, an interdisciplinary and multidisciplinary

curriculum, and a flexible curriculum. However, the

33,8%

17,5%

12,5%

15%

16,3%

8,8%

3,8%

12,5%

3,8%

13,8%

12,7%

20%

6,3%

11,3%

6,3%

43,8%

17,5%

10%

6,3%

6,1%

3,8%

21,3%

13,8%

16,3%

3,8%

10%

15%

40%

32,5%

40%

33,8%

40%

8,8%

13,8%

23,8%

11,4%

17,5%

44,3%

17,5%

26,3%

43,8%

12,5%

16,3%

30%

41,3%

33,8%

11,3%

28,8%

30%

37,5%

32,5%

28,8%

21,3%

38,8%

22,5%

21,2%

30%

27,4%

32,4%

6,1%

10%

17,4%

30%

25,3%

26,2%

30,4%

38,7%

36,1%

30%

21,2%

41,1%

23,7%

30%

28,8%

22,3%

35%

34,9%

29,9%

32,4%

21,1%

34,9%

31,2%

27,5%

3,7%

13,8%

18,8%

10%

22,5%

23,6%

17,5%

23,7%

27,9%

7,5%

10,1%

13,8%

22,5%

12,5%

35%

21,3%

2,5%

8,8%

13,7

21,3%

31,3%

22,5%

16,3%

15%

13,8%

18,8%

13,8%

12,5%

20%

1,3%

6,2%

1,3%

53,8%

46,3%

10%

31,6%

35%

2,5%

10%

8,8%

2,4%

25%

16,3%

2,4%

13,7

38,8%

16,3%

10%

2,5%

3,8%

27,5%

25%

7,5%

15%

0 102030405060708090100

A10

A11

A12

A13

A14

A15

A16

A17

A18

A19

A20

A21

A22

A23

A24

A25

A26

A27

A28

A29

A30

Coordinator's Evaluation

Totally Agree Partially Agree Neutral

Partially Desagree Totally Desagree

Are We Ready for Problem-based Learning? A Proposal of Institutional Diagnosis in Computing Higher Education

421

result also shows a warning about the curriculum's

ability to stimulate problem-solving and promote

self-directed learning. Also, the lack of flexible hours

and alignment of time to content was highlighted as

negative points.

Finally, as for the “Evaluation” aspect, the

following points stand out as favorable: monitoring

and feedback, effectiveness and evaluations of the

disciplines and the institution. Critical points were the

absence of feedback from the evaluation system on

the work of teachers and student learning, in addition

to the lack of evaluation of teaching methodology and

teacher performance.

5.3 General Discussion

Figure 6 presents an overview of the results from the

teachers' questionnaire.

Figure 6: Teachers’ results.

This figure shows a predominance of favorable

points on the Problem, Content, and Process axis,

with respect to the panorama of all the institutions

involved in the survey. Considering the education

purpose focused on professional performance, in

general, these institutions have worked with real and

relevant problems, with appropriate content for

problem-solving, and have a student evaluation

process. On the other hand, there was a greater

predominance of critical points in the Human Capital

axis, highlighting how impactful the PBL culture can

be in its adoption. Based on this understanding, PBL

training recommendations can be made regarding

each actor's responsibilities and roles in the learning

environment and the inclusion of new actors who can

make a difference, such as real clients and tutors. It is

worth noting the number of warning signs on the

Environment axis, indicating that there are still many

uncertainties regarding preparing the teaching and

learning environment for PBL in these institutions.

Figure 7 presents an overview of the questionnaire

results for the coordinators.

Comparing Figures 6 and 7, it can be clearly seen

how much the structural and organizational aspects

can negatively impact the PBL implementation,

having as main axes of concern the infrastructure and

the policy. Again, it became evident that it is

necessary to work on the institution's PBL culture,

involving educational managers and investing in

teacher training to prepare institutions for the radical

transformations that PBL brings. Other critical points

also highlight the need for flexibility of the content,

maintaining the alignment of theory with practice,

and a special concern with the continuous evaluation

process involving not only students, teachers, and

coordinators.

Figure 7: Coordinators’ results.

Finally, from this kind of overview, a discussion

with those involved in the diagnosis is recommended

with each institution to raise the points of

improvements and uncertainties, identifying and

managing the risks for implementing the PBL.

6 CONCLUSIONS

Based on the motivation of how to assess whether an

educational institution in the context of CHE is

prepared to adopt the PBL, this study proposed an

institutional diagnosis based on the pedagogical,

structural and organizational dimensions, evaluating

nine aspects with two groups of stakeholders:

teachers/tutors and coordinators/ managers. To

evaluate the model, this diagnosis was applied in 38

institutions of technical education in computing

through an opinion survey, obtaining an overview of

these institutions' situation with 302 respondents (222

teachers and 80-course coordinators in computing).

Due to these institutions' professional

characteristics, several favorable points were

identified, such as the use of real problems,

curriculum, and student evaluation. On the other

hand, critical points such as the academy's lack of

interaction with the labor market, teacher training,

and a more effective and ample assessment process

were also highlighted. In particular, some warning

CSEDU 2021 - 13th International Conference on Computer Supported Education

422

signs have also been identified regarding the

suitability of the learning environment for PBL and

the institutionalization of PBL culture.

The next steps are intended to make

improvements in questionnaires and apply them to

individual institutions, based on a careful analysis

with their stakeholders. It is also intended to apply

statistical methods for a more rigorous analysis.

ACKNOWLEDGEMENTS

The NEXT group thanks all the institutions that

participate in the survey and their respective

respondents, in addition to the group of experts who

evaluated and contributed to the production of the

survey questionnaires. Many thanks to all involved in

this study.

REFERENCES

Aldabbus, S, 2018. Project-Based Learning: Implementation

& Challenges. Int. Journal of Education, Learning and

Development, Vol.6, No.3, p. 71-79.

Alshaye, Tasir, Z. and Jumaat, N. F, 2019. The Conceptual

Framework of Online Problem-Based Learning

Towards Problem-Solving Ability and Programming

Skills. In IEEE Conference on e-Learning, e-

Management & e-Services (IC3e). IEEE, pp. 1-4.

Babori, A., Fassi, H.F, Hariri, A., Bideq, M., and Zaid, A.,

2016. Using problem based learning environment to

enhance algorithmic problem solving skill. In Global

Summit on Computer & Information Technology

(GSCIT). IEEE, pp. 60–65.

Barron, Lis; Wells, Lauren, 2013. Transitioning to the Real

World Through Problem- Based Learning: A

Collaborative Approach to Teacher Preparation. In

Journal of Learning in Higher Education.

Bell, S., 2010. Project-Based Learning for the 21st

Century: Skills for the Future. In The Clearing House,

83: 39–43, Taylor & Francis Group, LLC. ISSN: 0009-

8655print. DOI: 10.1080/00098650903505415.

DFID, 2003. Promoting Institutional & Organisational

Development. Department for International

Development, UK. ISBN 1 86192 535 2. Available in

http://www.kalidadea.org/castellano/materiales/evalua

cion/DFID%20promoting%20institutional%20develp

ment%20guide.pdf.

Enemark, S.; Kolmos, A.; Moesby, E., 2006. Promoting

and supporting PBL interests world wide: the profile of

the UICEE Centre for Problem Based Learning. In

Kolmos A. (Ed.), PBL at Aalborg University:

contributions to the International PBL Conference in

Lima July 17-24 (pp. 7-16).

Fink, F. K.; Enemark, S.; Moesby, E., 2002. UICEE Centre

for Problem Based Learning (UCPBL) at Aalborg

University. In: Proceedings of the 6th Baltic Region

Seminar on Engineering Education, Wismar, p. 6.

Herold, Robert, 2019. Problem based learning use in higher

education: a state university faculty study of pbl. Doctor-

ate in Education. California State University, Fresno.

Hmelo-Silver, Cindy E., 2004. Problem-Based Learning:

What and How Do Students Learn?. Educational

Psychology Review. 16 (3): 235–266.

Hsu, W.-C. and Lin, H.-C. K., 2016. Impact of applying

webgl technology to develop a web digital game-based

learning system for computer programming course in

flipped classroom. In EITT International Conference.

IEEE, pp. 64–69.

Ibrahim, N. and Halim, S. A., 2014. Generic framework

design of project-oriented problem-based learning

(popbl) for software engineering courses. In 8th.

Malaysian Software Engineering Conference

(MySEC). IEEE, 2014, pp. 359–364.

Jaryani, F., Zandi, B., Sahibudin, S., Salehy, S.,. Masrom,

M., and Zamani, M., 2010. Framework of a reflective

e-portfolio supported by outcome based education and

problem based learning. In Second International

Conference on Computer Research and Development.

IEEE, pp. 270–273.

Kemavuthanon, K., 2017. Integrated e-project

collaborative management system: Empirical study for

problem-based learning project. In 9th ICITEE

International Conference. IEEE, pp. 1–5.

Kitchenham, B. Pfleeger, S., 2008. Personal Opinion

Surveys. In: Guide to Advanced Empirical Software

Engineering. Springer, Cap. 3, p. 63-92.

Koray, O.; Koray, A., 2013. The effectiveness of problem-

based learning supported with computer simulations on

reasoning ability. In Procedia-Social and Behavioral

Sciences, 106, 2746 – 2755.

Krusche, S., Reichart, B., Tolstoi, P. and Bruegge, B., 2016.

Experiences from an experiential learning course on

games development. In Proceedings of the 47th ACM

Technical Symposium on Computing Science

Education. ACM, pp. 582–587.

Mäenpää, H., Varjonen, S., Hellas, A., Tarkoma, S, and

Männistö, T., 2017. Assessing iot projects in university

education: a framework for problem-based learning. In

Proceedings of the 39th International Conference on

Software Engineering: Software Engineering and

Education Track. IEEE Press, pp. 37–46.

Martins, V. F., De Almeida, I., Concilio, S., and De Paiva

Guimarães, M, 2018. Problem based learning associated

to the development of games for programming teaching.

In Computer Applications in Engineering Education, v.

26, n. 5. Scopus, 2018. pp. 1577-1589.

Melo, Roberto Correia, 2013. Estratégias de ensino e

aprendizagem baseadas em problemas (PBL) no ensino

tecnológico [Problem-based teaching and learning

(PBL) strategies in technological education]. In VIII

Workshop de Pós-Graduação e Pesquisa do Centro

Paula Souza, São Paulo.

Moesby, E., 2004. Reflections on making a change towards

Project Oriented and Problem-Based Learning

(POPBL).

Are We Ready for Problem-based Learning? A Proposal of Institutional Diagnosis in Computing Higher Education

423

Oliveira, A. M., Santos, S. C., Garcia, V. C. “PBL in

Teaching Computing: An overview of the Last 15

Years,” presented at the IEEE Frontiers in Education

Conference (FIE), Oklahoma, OK, Oct. 2013, pp. 23-

26.

Panwong, P. and Kemavuthanon, K., 2014. Problem-based

learning framework for junior software developer:

empirical study for computer programming students. In

Wireless personal communications, vol. 76, no. 3, pp.

603–613.

Patton, M. Q., 2008. Qualitative Research & Evaluation

Methods. 3. ed. California: Sage Publications.

Ribeiro, L. R. D. C., 2008. Aprendizagem Baseada em

Problemas (PBL) na educação em Engenharia

[Problem-Based Learning (PBL) in Engineering

Education]. Revista de ensino de Engenharia, 27(2),

23-32.

Richardson, I., Reid, L., Seidman, S. B., Pattinson, B., and

Delaney, Y., 2011. Educating software engineers of the

future: Software quality research through problem-

based learning”. In 24th IEEE-CS Conference on

Software Engineering Education and Training

(CSEE&T). IEEE, 2011, pp. 91–100.

Santos, D. M. B., and Silva, C. A. S, 2018. Problem-Based

Learning in a Computer Engineering Program:

Quantitative Evaluation of the Students' Perspective. In

IEEE Latin America Transactions, v. 16, n. 7, Scopus,

2018, pp. 2061-2068.

Santos, S. C., Figueredo, C. O., and Wanderley, F., 2013.

PBL-test: A model to evaluate the maturity of teaching

processes in a pbl approach, in Frontiers in Education

Conference, IEEE, 2013, pp. 595–601.

Santos, S. C., Furtado, F., and Lins, W., 2014. xPBL: A

methodology for managing pbl when teaching

computing. IEEE Frontiers in Education Conference

(FIE) Proceedings. IEEE, pp. 1–8.

Santos, S. C., Reis P. B. S., Reis J. F. S, Tavares F., 2020.

Two Decades of PBL in Teaching Computing: A

Systematic Mapping Study, IEEE Transaction on

Education, pp. 1-15.

Santos Filho, O. C., 2020. Diagnóstico Institucional para

Adoção de PBL em Cursos de Computação: Uma

Pesquisa de Opinião em Institutos Federais de

Educação, Ciência e Tecnologia do Brasil (Institutional

Diagnosis for PBL Adoption in Computer Courses: An

Opinion Survey in Federal Institutes of Education,

Science and Technology in Brazil), Master thesis,

Informatics Center, UFPE.

Savery, J. R. and Duffy, T. M., 1994. Problem based

learning: An instructional model and its constructivist

framework.

Souza, S. C.; Dourado, L, 2015. Aprendizagem baseada em

problemas (ABP): Um método de aprendizagem

inovador para o ensino educativo [Problem-based

learning (PBL): An innovative learning method for

educational teaching]. Holos, v. 5, p. 182-200.

Srinivasan, Malathi; Wilkes, Michael; Stevenson, Frazier;

Nguyen, Thuan; Slavin, Stuart, 2007. Comparing

Problem-Based Learning with Case-Based Learning:

Effects of a Major Curricular Shift at Two Institutions.

In Academic Medicine, Volume 82 – Issue 1 - p 74-82.

doi: 10.1097/01.ACM.0000249963.93776.aa.

Yang, J.-Y. and Choi, C., 2017. Enhancing interactiveness

of problem-based programming course using

anonymous chatting service. In 16th International

Conference on Information Technology Based Higher

Education and Training (ITHET). IEEE, pp. 1–6.

Yu, W., 2005. Work in progress-a mobile computing

collaborative framework for problem-based learning

environment. In Proceedings Frontiers in Education

35th Annual Conference. IEEE, pp. F4E–12.

Zhao, L. and Liu, F., 2011. A new theoretical pbl model for

mis course design and research. In International

Conference on Management and Service Science.

IEEE, pp. 1–4.

APPENDIX

A rationale and literature references for Teachers’ and

Coordinators’ questionaries are available in:

https://docs.google.com/spreadsheets/d/1E0fKh-Nim

U-OAhCed-geJ8bKpH8QzgmtqKD1VQzUupo/edit?

usp=sharing

CSEDU 2021 - 13th International Conference on Computer Supported Education

424