A Study of the Implementation Effectiveness of the CDIO Education
Model
Jing Wang
1
, Xumei Yuan
1
1
YanShan University, Qinhuangdao, P.R., China
Keywords: CDIO education model; CEM; Implementation effectiveness; empirical study.
Abstract: The CDIO education model is the latest achievement in the reform of international engineering education in
recent years. China introduced the model and initiated pilot implementation in late 2005. After more than
ten years of investigation and practical experience, the higher education institutions that participated have
developed and consolidated their own training characteristics. Through the analysis of the evaluation
standards for engineering personnel as well as elements of the CDIO curriculum, this paper employed the
catastrophe evaluation method to constructs the evaluation index system, and to assess the effectiveness of
the CDIO engineering education model at the university.
1 INTRODUCTION
The conceive-design-implement-operate (CDIO)
engineering education model was initiated by a
multinational research team comprising members
from the Massachusetts Institute of Technology
(MIT) and several Swedish universities. It focuses
on providing students with a realistic engineering
setting that embodies the entire life cycle of a
product, from conceptual design to operational
implementation and even maintenance. This allows
students to study engineering in a proactive and
hands-on way, through courses that are organically
connected, thereby enabling them to develop into
engineers capable of solving practical engineering
problems in today’s complex environment[1-2]. In
late 2005, Shantou University was the first Chinese
institution that introduced this model to
pilotimplementation in five departments of its
College of Engineering. In April 2008, a study
group, established by Department of Higher
Education of China’s Ministry of Education,
dedicated to develop the research and practice
related to the CDIO engineering education model[3].
These two steps pronounce the construction of
CDIO, which highlight the best of talent cultivation
in engineering colleges.
Although researchers consistently have
investigating the theory of CDIO, the deployment of
this model is not scaled and the implementation is
beyond the college capacities. The investigation of
working principle of CDIO is the foundation of
application, which exploits the best of both by
defining engineering education strategies. The focus
on the study of CDIO principles in recent years has
called for corresponding support research in this
field. Effectiveness evaluation is one such field ,
with technical approaches exploring the potential of
CDIO for engineering education. Thus, in this study,
graduates of the mechanical departments of YS
University were selected as participants. The
Catastrophe Evaluation Method (CEM) was adopted
to comprehensively evaluate and verify the
effectiveness of the CDIO engineering education
model through empirical research. The paper is
arranged as follows. Section 2 introduces the basic
principle of catastrophe evaluation method. Section
3 presents a general overview of system for indicator
evaluation. Methodology of data collection as well
as the sample preprocessing approach is depicted in
section 4. Section 5 provides a modeling based on
the theory of CEM to examine the CDIO application
and the testing outcome is given. Section 6 shows
the discussion, conclusion and future steps to be
taken.
2 CATASTROPHE EVALUATION
METHOD
2.1 Fundamental Concept
Learning is a highly complex psychological activity.
The evaluation of students’ learning effectiveness is
inevitably provisional and largely depending on the
subjective experience as well as the specific
evaluation criteria. CEM(Catastrophe theory
evaluation method) is a comprehensive evaluation
method developed on the basis of catastrophe
theory. Traditional evaluation process considers
merely the relative importance of evaluation indices
without assigning weights to other parts.CEM can
effectively avoid errors in artificially determining
weights on quantitative evaluation. Therefore, the
evaluation results reduce subjectivity without
limiting overall robustness and are calculated in a
simple and convenient way[4-7]. This study adopted
CEM to assess each student’s achievement of the
expected learning.
CEM is an evaluation method considering the
purpose of an evaluation system, which constructs
the evaluation index system and breaks down its
contradictions in a multi-level manner consistent
with the mechanism of the system itself. The overall
indicator is gradually broken down into sub-
indicators, and the target units are presented as an
inverted tree. By determining the grade of the
underlying evaluation index,and the state variable is
normalized using the catastrophe fuzzy subordinate
function. Hereafter, the decision-making outcome is
evaluated[8-10].
2.2 Evaluation Method
The common catastrophe models contain the fold,
cusp, swallowtail, and butterfly catastrophe systems.
We employ the swallowtail and butterfly catastrophe
systems in this paper. Swallowtail catastrophe is
utilized for the condition of three control variables
while butterfly catastrophe for four. A potential
function f(x) can be used to express the state
variable x for each system[11-12].
The swallowtail catastrophe potential function is:
(1)
The butterfly catastrophe potential function is:
(2)
The coefficients a, b, c, and d indicate the control
variables of x.
The swallowtail catastrophe equilibrium surface
could be obtained by a first derivative of f(x), Again,
f^'(x) =0,thus:
The swallowtail catastrophe equilibrium surface
was:
(3)
The butterfly catastrophe equilibrium surface
was:
(4)
The singular point set of the equilibrium surface
could be obtained by a second derivative of f(x),
Again
, eliminating the state variable x
through simultaneous equations f^'(x) =0 and f^'' (x)
=0. Thus, a bifurcation equation of the catastrophe
system is obtained.
The swallowtail catastrophe bifurcation equation
was:
(5)
The butterfly catastrophe bifurcation equation
was:
(6)
The bifurcation equation showed that when the
control variables satisfy this equation, a mutation
occurs in the system and the effect of each control
variable upon the mutation would be obtained[13-
14].
A normalization equation was used to convert
the different qualitative states of the control
variables in the system into the same state and
calculate different x values for each control variable
of the same target. Normalization equation is the
basic computing formula that using catastrophe
theory to comprehensive analysis and evaluation
system, It carries out quantized recursive operation
for system, Therefore, the total catastrophe
subordinate function, of the system characterized is
obtained by the state characteristics of the system.
Thus:
The swallowtail catastrophe normalization
equation was:
(7)
The butterfly catastrophe normalization equation
was:
(8)
According to the fuzzy decision theory, the
complementary and non-complementary principles
should be adopted during evaluation, which depend
on the relationship between variables. The
complementary principle is utilized when there is a
certain relationship between the variables in the
system where the state variable x takes the average
value of the catastrophe level of each
variable
. In contrast, the non-complementary principle
is for all the variables irreplaceable where the state
variable x takes the minimum value of the
catastrophe level of each variable
.
As such, the minimax criterion was applied.
3. CONSTRUCTING THE
EVALUATION INDICATOR
SYSTEM
The aim of the CDIO engineering education model
is to nurture students in a modern, team-based
environment, enabling them to become engineers
who are proficient in applying CDIO in the context
of complex and value-added engineering products,
processes, and systems. The evaluation of
implementation effectiveness was determined by all
the relevant stakeholders in students’ learning. In
this study, we invite graduates, the direct recipients
of education, to be participants. Thereby, their
achievements of the expected learning are assessed
in relation to the CDIO engineering education model
by examining their personal development and
professional abilities[15-18]. The main influencing
factors determining the ability to achieve the
expected effectiveness were confirmed by applying
the evaluation standards for engineering personnel
as well as elements of the CDIO curriculum. The
main factors were broken down, one by one, into a
number of indicators to build a hierarchical structure
for the evaluation indicator system.
Based on the CDIO outline and on the relevant
requirements of educational evaluation, the authors
of this study divided the evaluation criteria into 4
two-level indexes from the four dimensions of
"subject knowledge" "personal skills and attitudes"
"non-technical skills""career competence and
development " , then16 three-level evaluation
indicators again.To ensure their validity, experts
were consulted to assess the indicators in terms of
substance and scope. In the first round, seven
domestic experts who promoted the CDIO
engineering education model were employed to
evaluate the 16 indicators. After repeated
consultation and deliberation, they finalized 14
evaluation indicators, which were included in the
first draft of the questionnaire design, as shown
below:
Figure1: Evaluation index system of the CDIO Education Model.
In the second round, 14 domestic experts with
experience of implementing the CDIO engineering
education model were invited to test the validity of
the questionnaire and measure its representativeness
using the content validity ratio.
Table1：Test of the content validity ratio.
Notes:
• V-A: Very Appropriate, A=10 points
• M-A: More Appropriate, B=8 points
• A: Appropriate,C= 6points
• N-A: Not Very Appropriate, D=4 points
• I: Inappropriate,F=0 points
The content validity of the indicators shown
above ranged between 8.0 and 8.5, which
demonstrates high content validity; the indicators
selected for evaluating the effectiveness of the CDIO
engineering education model, therefore, were
appropriate. The state variable x represented the
expected level of achievement when implementing
the CDIO model. The evaluation range of the 14
indicators included five levels: very satisfied,
satisfied, average, dissatisfied and very dissatisfied;
whose respective values were 5, 4, 3, 2, and 1.
According to CEM, the overall evaluation
indicator (A) is at the top of the evaluation system
for the achievement of expected learning in the
CDIO engineering education model. The middle
layer presents a butterfly catastrophe, with indicators
(B1), (B2), (B3), and (B4) corresponding to the
control variables
，
，
，
； of the
catastrophe system. The catastrophe systems
displayed from left to right are as follows:
swallowtail, butterfly, swallowtail, and butterfly.
Evaluation indicators 1, 2 and 3 are related to
“subject knowledge” and correspond to the control
variables
，
，
of the catastrophe system.
Evaluation indicators 4, 5, 6 and 7 are related to
“personal skills and attitudes” and correspond to the
control variables
，
，
，
of the
catastrophe system. Evaluation indicators 8, 9 and
10 are related to “non-technical skills” and
correspond to the control variables
，
，
of the catastrophe system. Evaluation indicators
11, 12, 13 and 14 are related to “career competence
and development” and correspond to the control
variables
，
，
，
of the catastrophe
system. As the study evaluated the achievement of
the expected effectiveness of the CDIO model, the
complementary principle applied to each catastrophe
model.
4. DATA SOURCES AND SAMPLE
DESCRIPTIONS
The participants in this study graduated from YS
University between 2011 and 2017. During their
studies, they were taught using a “project-based”
model based on the concept of CDIO. For the
survey, a total of 328 questionnaires were sent out,
of which 250 were returned, and valid
questionnaires 248.The data is composed as follows:
Table2: Questionnaire statistics.
4.1 Data Reliability Test
This study adopted the Cronbach’s α reliability
coefficient for evaluation, using this formula[19]:
α
(9)
where n is the total number of items in the scale,
is the intra-item variance of the score of the ith item,
and
is the variance of the total score of all the
items. The survey data into the formula (9), by
calculation, this questionnaire’s Cronbach’s
α=0.726, the “Subject knowledge” Cronbach’s
α=0.257, the “Personal skills and attitudes”
Cronbach’ s α=0.951, the “Non-technical skills”
Cronbach’s α=0.454, the “Career competence and
development” Cronbach’s α=0.342. The internal
consistencies of the four dimensions of the
questionnaire and the questionnaire as a whole were
above 0.70, so meet the requirements of exploratory
research.
4.2 Normalized Transformation
To obtain a better distribution of research data, a
normalized transformation was performed on the
original data by subtracting the minimum value from
the original data of each variable and dividing by the
range (the difference between the maximum and
minimum value of each variable), using this
formula[20-21]：
(10)
where i is the number of indicators , i=1,2，……，
and j is the number of units, j=1,2，…….
In this paper, the normalization transformation is
used to normalize the index system step by step.
Because of the complexity of the calculation
process, this paper only takes the evaluation data of
the 2011 graduates as an example to show the
deducing process of the implementation
effectiveness CDIO education model evaluation.
First, the original values of each indicator at the
lowest level were processed using equation(10), the
results are shown as table3:
Table3: The data of the 2011 graduates' evaluation of
normalization transformation.
After processing, the values were between 0 and
1. Next, the values were converted using formulas 2-
7 and 2-8：
：
According to the principle of complementarity：
According to the same method, the evaluation
data of the 2012-2017 graduates can also be
calculated，as shown in table 4:
Table4:Evaluation index value and result.
5. EVALUATION RESULTS
VERIFY AND ANALYSIS
5.1 The Evaluation Result of Principal
Component Analysis
CEM is used to evaluate the implementation
effectiveness of the CDIO education model, whether
this method is reliable or practical? Therefore, this
paper uses the evaluation index system as shown in
figure 1 and the evaluation data collected, using the
method of principal components analysis(PCA) to
verify this example. If the evaluation results of the
two methods are consistent, then there is a reason to
believe that the evaluation result of CME is true and
effective, so it can avoids the chance of consistency
of the evaluation results. PCA is a common method
in multivariate statistical analysis, the calculation
procedure is as follows:
First, suppose there are m evaluation objects and
n evaluation indexes, and the scores of each
principal component are calculated)：
(11)
isthe value of the kth principal component
about item the valuated object,
isthe normalized
value of the ith evaluating index about item the
valuated object,
is the load value of the k th
principal component about item I th of evaluation,
λ
is the characteristic value of the kth principal
component.
Second the proportion of the variance
contribution of each principal component is the
weight, the score of composite principal component
of each evaluated object was calculated.
(12)
is the comprehensive score of the tth evaluated
object,
is the weight of the score about the kth
principal component, s is the number of principal
components extracted( the number of principal
components extracted is determined by the
accumulated variance contribution rate of over
85%)[22-23].According to the above steps,
evaluation of the implementation effect of education
mode of CDIO project of 2011-2017 graduates, The
results are shown in table5.
Table5:Evaluation result of PCA.
5.2 Consistency Check of Two
Evaluation Results
The consistency of the evaluation results of the
mutation evaluation method and the principal
component analysis method was tested by the
Kendall cofactor test. The results are shown in
table6.
Table6: Test result of Kendall.
It can be seen that the coefficients of Kendall's
collaborative test for the evaluation results of the 7th
graduates are all between 0-1, and the probability p
all the 0.00. The evaluation results of the two
methods are verified by the Kendall’s, so the
evaluation results of the two methods are statistically
signify cant consistent.
The core of the CEM is to establish a recursive
algorithm for the multi-objective and multi-level
comprehensive evaluation problem by using the
normalized formula deduce the bifurcation equation
of the catastrophe theory. Its main advantage is that
it avoids the concept of direct use of "weights"
which are difficult to be determined and they are
subjective. At the same time, because the normalized
formula reflects the mechanism of the evaluation
index to a certain extent, the catastrophe evaluation
model can consider the importance of each
evaluation index reasonably and quantitatively.
5.3 Evaluation Results Analysis
The learning process of the traditional teaching
mode has been arranged by the predecessors
according to the optimal structure, and the students
only need to absorb and understand quickly. But the
CDIO engineering education mode with "project"
for the driver, around the "project" is completed, will
be this major should master the fragmentation of
knowledge and ability construction, it and the
emphasis of the traditional teaching mode has great
differences.
As the learning experience, knowledge base,
reading exposure, cognitive level, academic skills,
and practical abilities of each student are different,
inconsistent effectiveness was achieved in the
implementation of the CDIO engineering education
model. The model emphasizes student-centered
learning, requiring students to abandon the
traditional model of passive reception and to
embrace active participation in the learning process.
The results showed that the overall evaluation of the
students’ achievement in relation to “subject
knowledge” was high. Through implementing the
CDIO model, students better understood or
remembered abstract theories and concepts, and
were able to apply their theoretical knowledge in
practical engineering situations. They also formed
new cognitive frameworks relating to professional
learning in order to help strengthen their grasp of
fundamental knowledge.
The evaluation of “personal skills and attitudes”
focused on a comparison with traditional teaching
methods, and addressed the limitations of the
students’ knowledge of engineering technology and
methods in the context of multi-level, multi-area,
and large-scale complex systems. The
implementation of the CDIO education model made
students realize that there are unpredictable
challenges in learning and enhanced their individual
capabilities and attitudes by repeatedly strengthening
their skills of engineering reasoning, systematic
thinking and critical thinking, practical hands-on
training, and solving complex engineering problems.
According to the graduates’ feedback, the CDIO
model achieved relatively good levels of
effectiveness here.
The development of “non-technical skills”
among students mainly depended on the “projects”
offered during the implementation of the CDIO
model. A “project” is a broad concept and an
important factor worthy of further study. The
specific content of a project can effectively stimulate
enthusiasm for learning. During projects, factors
such as team composition and the nature of the
collaborative atmosphere among team members
directly influenced the frequency and intensity of
communication during the learning process, thereby
directly affecting the students’ development of non-
technical skills. The graduates rated this area as
average. This result will encourage schools to focus
more on the selection of project content, team
composition, and evaluation methods in order to
achieve higher teaching effectiveness when the
CDIO education model is implemented in the future.
“Career competence and development” focused
on the training of the ideal qualities necessary for
engineers. Limited teaching hours in institutions are
a huge constraint on the facilitation of career
development. The CDIO education model guides
students to form product-oriented values that enable
them to recognize the professional abilities of
engineers in visual thinking, effective
communication, social responsibility, and
accountability during their autonomous practice of
the CDIO model. This ensures that the students
demonstrate greater creativity in their future
engineering careers. The graduates rated this area as
relatively good.
Based on the standards of classroom teaching
effectiveness and on the characteristics of the CDIO
education model, the overall evaluation was
classified using five levels: excellent （ Result ≥
0.90 ） , good （ 0.80 ≤ Result ＜ 0.90 ） , average
（ 0.70≤ Result＜ 0.80） , pass（ 0.60≤ Result＜
0.70） and fail（Result＜0.60）.So it can be seen
that the evaluation of the graduates is good,
indicating that the graduates have a higher
recognition of the effect of the CDIO teaching
model.
Figure2: 2011-2017 Bar graph of evaluation results.
Students can learn in the same major, and the
group will not have the extreme ability difference.
Compared with the results of the seven graduates,
we can see that the overall trend of scores is getting
higher and higher. As for the reasons for the low
score of the 2017 graduates, the author once again
communicated with the graduates and learned that
the number of students in this class is more than that
of the previous grade. As a local university with
traditional engineering advantages, YS university is
limited by practical difficulties such as shortage of
resources for running schools.
Evaluation feedback from graduates has always
been an effective means for higher education
institutions to strengthen their relationships with
corporations and society, to obtain external
information, to reform personnel training models,
and to improve the quality of the teaching they offer.
By comparing the evaluation feedback of students
who graduated in different years, we found that
students who graduated earlier reflected more on
their experiences and achievements under the CDIO
model after their graduation. This shows that the
CDIO education model has indeed established a
solid professional foundation among students,
allowing them to accumulate qualities and abilities
that offer them more opportunities to perform well
as engineers in their working lives.
There is no fixed standard for the
implementation of the CDIO education model.
Higher education institutions adopt their own
appropriate methods of implementing the model to
ensure that students continuously accumulate and
practice innovation and teamwork, to ensure that
they develop practical and analytical abilities, and to
ensure that, during their participation, they gradually
build up a scientist’s pragmatic sense of judgment.
The evaluation results from several cohorts of YS
University graduates show that the CDIO model
achieved results that were good or excellent. The
students’ awareness of, and satisfaction with, the
CDIO engineering education philosophy were also
high. In future, higher institutions should continue to
review and improve the implementation of the CDIO
education model and strive to achieve better
expected results, ultimately promoting the model.
REFERENCES
1. Crawley, E.F., Malmqvist, J., Östlund, S., & Brodeur,
D.R.2012. Rethinking Engineering Education: The
CDIO Approach. M. (Gu, P., Shen, M., & Lu X. Trans).
Higher Education Press(Beijing,China 2012), 1-2.
2. Fan, Y., Zhang, X., Xie, X.2015.Design and
Development of a Course in Professionalism and
Ethics for CDIO Curriculum in China. J. Sci Eng
Ethics(No.21, 2015), 1381-1389.
3. Gu, P.,Bao N., etc. 2012.CDIO in China(Part I).
J.Research in Higher Education of Engineering,(Mar.
2012), 24-40.
4. Meng, H., Wang, Y. 2008. The comprehensive
Evaluation of Knowledge Innovation System of
Research University Based on CEM. J. Operations
Research and Management Science, (Vol17.3, 2008),
80-87.
5. Shi, Y.,Liu, Y., He, J. 2003. Further Study on Some
Questions of Catastrophe Evaluation Method. J.
Engineering Journal of Wuhan University, (Vol36.4,
2003),132-136.
6. E. Michael Staman., Research in Higher
Education:Catastrophe theory in higher education
research. M.Vol16(1), 41-53(1982)
7. Liu, C., Li, J. 2011. Application of Catastrophe
Progression Method Simulation in Gas Hazard
Assessment of Coal mine. J. Computer Engineering
and Applications, (Vol 47.6, 2011), 231-234.
8. Matthijs Koopmans,Dimitrios Stamovlasis.2016.
Complex Dynamical Systems in Education:
Concept,Methods and Applications.M. Springer
International Publishing, (Switzerland, 2016), 141-
175.
9. Qin, G. 2015. Research on the Evaluation of
Comprehensively Innovative Capacity in the Industrial
Transfer Demonstration Area of the Cities along the
Yangtze River Based on the Series Catastrophe
Theory. J. Science and Technology Management
Research, (Vol 16, 2015), 83-86.
10. Du,X. 1994. The application of Catastrophe Theory in
the economic field. M. University of Electronic
Science and Technology Press. (1994), 24-31.
11. Li, Y., Wu, J., The Catastrophe Series Approach to
Acquisition Performance of Listed Companies. J.
Journal of Guizhou College of Finance and
Economics .(Vol 4, 2014), 20-23.
0,8
0,81
0,82
0,83
0,84
0,85
0,86
0,87
Result
12. Su R. 2006. The Value of Mutation Theory in
Education Research Methodology. J. Journal of
Fujian Medical University (Social Science
Edition),(Vol.7.2, 2006), 38-41.
13. Chen, Y. 2016. Zhang, S., et. al., Comprehensive
assessment and hierarchical management of the
sustainable utilization of urban water resources based
on catastrophe theory. J.Journal of the Taiwan
Institute of Chemical Engineer. (Vol 60, 2016), 430-
437.
14. Shi, Y. 1997. Application of Catastrophe Evaluation
Method in Management Benefit Evaluation of Water
System. J.Water Conservancy Economy, (Vol 5, 1997),
52-55.
15. Gu, X. 2009. Connecting Abstrace Theories with
Concrete Engineering Skills in the CDIO Learning
Cycle.J. Research in Higher Education of Engineering,
(Vol 1, 2009), 11-23.
16. Edward F. Crawley. 2008. The CDIO Syllabus: A
Statement of Goals for Undergraduate Engineering
Education.(Online).http://cdio.org/cdio-syllabus-
rept/index.html.
17. Johan Malmqvist. 2006. Kristina Edström., Svante
Gunnarsson., Sören Östlund.,The Application of
CDIO Standards in The Evaluation of Swedish
Engineering Degree Programmes. M. World
Transaction on Engineering and Technology
Education .(Vol.5.2, 2006), 361-364.
18. Zheng, W. 2010. CDIO-Based Research and Practice
of Innovative Engineering and Technical Talent
Training Mode. D. Dalian University of Technology,
(2010)
19. What does Cronbach’s alpha mean? UCLA -
Available at: (Online )
http://www.ats.ucla.edu/stat/spss/faq/alpha.html.
(2014)
20. Jeffrey Penney. 2017. A self-referece problem in test
score normalization. J. Economics of Education
Review,(Vol 61.1, 2017)79-84.
21. Cheng, M., Zhang, L. 1996. Application of the
catastrophe evaluation method in the evaluation of
administering national tax education. J. Forecasting,
(Vol 4, 1996), 63-65.
22. Kuroda M., Mori Y., Lizuka M., et. al., Acceleration
of the Alternating Least Squares Algorithm for
Principal Components Analysis.J. Computational
Statistics and Data Analysis,(Vol 55.1, 2011), 143-153.
23. Liang.,X., Yang F., etc. 2011. Study on the multilayer
financial center system in China based on evaluation
of urban financial competitiveness.J. Systems
Engineering-Theory & Practice,(Vol.31.10, 2011),
1847-1857.
