Reflective ePortfolio System
Development and Assessment in Living Lab
Aleksandrs Gorbunovs
Distance Education Study Centre, Riga Technical University, Azenes Str. 16/20, Room 410, Riga, Latvia
Keywords: Information System, ePortfolio, Reflection, Living Lab, Assessment, Validation, Verification.
Abstract: Modern technologies, information systems, tools, methods and approaches give us new potentialities to
ensure better learning outcomes. One of such systems, which are kept high on the agenda, is ePortfolios.
ePortfolio systems are considered as an excellent tool to improve learners’ competence levels, critical
thinking and reflection. This paper shows an approbation of reflection stimulating ePortfolio system
developed by the Distance Education Study Centre, Riga Technical University. Introduced system merges a
scope of technological and educational aspects to facilitate system users’ better achievements. The author
underpins Living Lab research method which was used to approbate new system implementation.
Experimental part of the work is proved by verification of the ePortfolio system including necessary
statistical data analysis. Approbation results show that the developed algorithmic model ensures the
formation and functioning of the reflection stimulating ePortfolio system which has direct positive impact
on students’ competence development, achievements and learning outcomes.
1 INTRODUCTION
Nowadays technologies and information systems
bring new potentialities for both teaching staff and
learners. Innovative approaches together with
technologies and systems ought to ensure lifelong
learning demands. As a result, information system
developers have been facing the problems of
working up appropriate solutions to facilitate better
learning outcomes. One of such systems, which are
kept high on the agenda, is ePortfolios. More and
more educational organisations all over the world
embody ePortfolio systems in their curricula
(Timmins, 2008). ePortfolio is considered as an
excellent tool to improve information system users’
competence levels, critical thinking and reflection.
Usually educators and ePortfolio system
developers introduce their experience, approach,
methodology, educational tools and systems, which
display students’ achievements or allow some
interaction in the form of blogs, reflection of peers’
comments regarding particular tasks, etc. (Barrett,
2009, 2011). However, until this day there is still a
lack of comprehensive research activities and data
analysis related to measurement of ePortfolio
systems effectiveness. Experts in this field suggest
evaluating system efficiency by measuring activity
evidences of its users; characteristic features of the
reflection are underlined as the key components in
this case (Haig et al., 2007).
Pursuant to actuality of efficiency studies of
ePortfolios and a necessity to improve learning
outcomes, conformable purpose of the research was
formulated – to develop and approbate reflection
stimulating ePortfolio system which would merge a
scope of technological and educational aspects to
facilitate system users’ better achievements.
2 DEVELOPMENT AND
ASSESSMENT OF THE MODEL
2.1 Tools and Methods
After completed theoretical investigation
(Gorbunovs, 2011) the development of reflective
stimulating ePortfolio system was continued by
practical research activities. They included the
development of appropriate system algorithmic
model based on Enterprise Knowledge Development
(EKD) Methodology (Kirikova, Stecjuka, 2008),
simulation scenarios output (Quinn, 2005) and data
flow modelling.
Keeping in mind that Living Lab research
385
Gorbunovs A..
Reflective ePortfolio System - Development and Assessment in Living Lab.
DOI: 10.5220/0004934503850391
In Proceedings of the 6th International Conference on Computer Supported Education (CSEDU-2014), pages 385-391
ISBN: 978-989-758-020-8
Copyright
c
 2014 SCITEPRESS (Science and Technology Publications, Lda.)

method is defined as an environment where
information system users evaluate and validate novel
systems and technologies (Følstad, 2008), this
method was used to assess new ePortfolio
algorithmic model. Corresponding prototype was
created, validated and verified in Living Lab in
2011/12 and 2012/13 academic year. Statistical data
analysis was done by Excel 2010 and SPSS 21
software; Kolmogorov-Smirnov and Mann-Whitney
non-parametric tests, statistical hypothesis T-test,
and determination of correlations were used.
Additionally, positive feedback regarding system
impact was received from particular scientific field
experts at the international scientific conferences
and domestic seminars. Several inquiries were
organised also after each completed course to make
out students perceptions and their thoughts about
used information system’s effect on their reflection
and competence levels improvement.
2.2 Model Development
Based on an idea that humans do like teach others
rather learn themselves (Adler, 2013), as well the
statement that Living Labs involve users in the
innovation, knowledge sharing, exploration,
experimentation, assessment, and co-creation
process (Pallot, 2009), respective algorithmic model
was developed (Fig.1). It enables group formation,
self- and peer-assessment within groups,
responsibility for own attitude, activation of critical
thinking and reflection, and as a result –
improvement of learning outcomes.
First of all, it is necessary to specify that
university’s learning and content management
system (LCMS) and created ePortfolio system are
two independent information systems. Activities
within ePortfolio system are available when system
administrator or course tutor manually copies
fulfilled homeworks from the LCMS data base to
ePortfolio system.
Before ePortfolio system enables any activities,
students fulfil first assignments: take a test to assess
initial level of their competences, make self-
appraisal and submit first homework. All these data
go into university’s LCMS data base. After
submission of the homework at the first onset the
tutor inputs it to ePortfolio data base. Based on a
time sequence of submitted homeworks, ePortfolio
system forms groups of four students each (Fig.2).
Activities in ePortfolio system start with user’s
authentication and authorization. Login files data are
collected in ePortfolio data base. Students assess
other group members’ homeworks in a form of
suggestions and scores in the scale from 1 to 10,
where 1 is the lowest estimation and 10 – the highest
one.
The same procedure applies on self-assessment
of own homework. Both approaches stimulate
Selfassessment of
own homework
Assessment
within eP. group
Assessments
given by eP.
group members
Selfassessment
results of own
homework
Acquisition of next
themes
Improvement of
the homework,
based on
suggestions
Improved work
Must improve
no
ePortfolio data
base
University’s
LCMS data base
Reflection
Marks and critical
thinking notes
Group members’
given marks, critical
thinking notes
Analogical
procedures
to acquisition of
the 1st theme
End
Activities within
eP. system
Processing of the
1st selfassessment
1st selfassessment
questionnaire
Selfassessment
of own
competencies
1st test
Test run 1st test results
1st homework task
Start
Completion
1st homework
Data from
university’s IS
Input to eP. system
made by admin/tutor
Output from eP.
system made by
admin/tutor
Data from eP.
system
Sub-process
Formation of
eP. groups
yes
Login files
data
Figure 1: Reflection stimulating ePortfolio system’s algorithmic model.
CSEDU2014-6thInternationalConferenceonComputerSupportedEducation
386

The 1st
submission
Data 1
Data 2
Data 3
Data 4
Group
No.1
Data 5
Data 6
Data 7
Data 8
Group
No.2
Time
The 2nd
submission
The 3rd
submission
The 4th
submission
The 5th
submission
The 6th
submission
The 7th
submission
The 8th
submission
Figure 2: Group formation within ePortfolio system.
reflection and impact an improvement of previous
documents. Based on peers made evaluation the
student takes steps to improve own homework and
proceeds to the next course module, or, if he/she
decides that there is nothing to be improved in the
homework, peers remarks are taken into account and
the learner also proceeds to the next course module.
During 2011/12 and 2012/13 academic year all
improved papers were sent to university’s LCMS.
However, the algorithmic model allows developing
in future another prototype which would return all
improvements back to ePortfolio group collaborative
environment for reviewing.
2.3 Assessment of ePortfolio System in
Living Lab
2.3.1 Validation
To validate the first developed ePortfolio system
model, an appropriate prototype was built and
approbated in 2011/12 academic year at Riga
Technical University. Approbation results in Living
Lab show effectiveness of the system which
stimulates system users’ reflection and improves
particular to the course competence levels (Fig.3). It
was concluded that the improvement of learning
outcomes, i.e. competence levels (measured by exam
results), reflection (measured by a number of
improved papers) and accomplishments outside the
system (measured by a number of prepared papers),
were directly dependent on system users activities
within the system, i.e. fulfilled tasks in group-
working activities and login files.
Figure 3: Learning outcomes depending on activities
within ePortfolio system.
ePortfolio system users survey results also
demonstrate system’s importance on improvement
of learners competence levels (Fig.4) and reflection
(Fig.5). Considering that these surveys were
organised apart, the number of respondents vary.
In the first questionnaire students were asked to
mark in the scale from 1 to 10 (from the worst
answer to the best one, i.e., mark “1” meant that the
system had not an impact, mark “2” – the impact
was negligible, mark “10” – the system had the most
impact), how much ePortfolio system improved their
competence levels.
112 users participated in this survey. Majority of
them – 77 learners (or 68 per cent) had a strong
confidence about system’s (or 10 per cent) – held a
view that the system had a minor impact on their
competence improvement, and only 3 participants
(or less than 3 per cent) said that they did not notice
any system’s impact.
In the second questionnaire students were asked
how much ePortfolio system improved their
reflection abilities. From 116 users participated in
this survey majority – 103 respondents (or 88 per
cent) had a strong confidence about system’s
positive impact on their reflection, 12 students (or 11
per cent) ware rather satisfied, and only 1 learner (or
less than 1 per cent) admitted unsubstantial impact
of the system. There was nobody who would say
that the system did not improve his/her reflection
abilities.
0
2
4
6
8
10
12
14
16
18
20
Inactive
Low active
Less active
Fair active
Active
Very active
ReflectiveePortfolioSystem-DevelopmentandAssessmentinLivingLab
387

Figure 4: Users’ opinions related ePortfolio system’s
impact on their competence improvement.
Figure 5: Users’ opinions related ePortfolio system’s
impact on their reflection.
2.3.2 Verification
To find relationship between input and output
parameters of developed ePortfolio system,
representative sample of 145 students was ranked
discrete into two groups: non-users group (20
students) and experimental group – ePortfolio
system users with at least one login (125 students).
There was impossible to set up equal quantitative
structure of both groups due to a principle of
voluntary participation in research activities.
To find competence distribution in whole sample
and within groups, Kolmogorov-Smirnov test was
used to prove null hypothesis H0 - competence
distribution within group forms normal distribution,
and alternative one Ha - competence distribution
within group does not form normal distribution.
Table 1: K-S test for the whole sample.
The distribution of initial competences (initial
test) for whole sample forms normal distribution
(Table 1) – as asymptotic significance 0,247 > 0,05
and test value does not exceed critical values, the
hypothesis H
0
is affirmed with 95% level of
confidence. The distribution of final competences
(exam) at the end of the course does not form
normal distribution – as asymptotic significance 0,00
< 0,05, the hypothesis H
0
is not affirmed.
Wherewith, it confirms the impact of created
ePortfolio system on users learning outcomes.
Table 2: K-S test for experimental group.
During Kolmogorov-Smirnov test for
experimental group (Table 2) it is found that as
asymptotic significance of initial test 0,502 > 0,05,
and test value does not exceed the critical one, the
null hypothesis H
0
is affirmed with 95% level of
confidence. It means that the distribution of initial
competences for experimental group forms normal
distribution. The distribution of final competences
does not form normal distribution – as asymptotic
significance 0,00 < 0,05, the null hypothesis H
0
is
rejected. Analysis for each separate login file was
not made due to excessive number of login files
(from 1 to 38) and insufficient number of users in
each. Consequently, the distribution of achieved
competence levels at the end of the course does not
form normal distribution in active users’
experimental group. However, as soon as we start
analysing users activity within the system,
exponential distribution appears. We can conclude
that the more the user logins the better results he/she
achieves.
3%
4%
6%
6%
13%
10%
22%
16%
13%
7%
1
2
3
4
5
6
7
8
9
10
CSEDU2014-6thInternationalConferenceonComputerSupportedEducation
388

For comparing mark distribution in two populations
(non-users and experimental ePortfolio system users
group) Mann-Whitney nonparametric test was used
(Table 3) to test null hypothesis H
0
that competence
distribution in both populations have identical
distribution functions against the alternative
hypothesis H
a
that competence distribution in two
distribution functions differs.
Table 3: Mann-Whitney test.
During Mann-Whitney test it was found that: as
asymptotic significance of initial test 0,374 > 0,05,
the null hypothesis H
0
is affirmed with 95% level of
confidence. It means that distribution of initial
competence levels in both non-users and users’
groups is identical. As asymptotic significance of
exam (i.e. achieved competence levels at the end of
the course) 0,00 < 0,05, the null hypothesis H
0
is
rejected, and alternative hypothesis H
a
is affirmed –
distribution of final competence levels in two groups
differs. It could be concluded that experimental
group, which took part in ePortfolio activities,
achieved better results than non-users group.
For comparing significant difference of
arithmetic means between two groups (experimental
and non-users ones) the T-test was used (Table 4).
As the T-test value -1,304 does not exceed critical
values, with 95% level of confidence we can
conclude that both groups have the same on average
initial competence level (average initial competence
level value-judgement of non-users is 7,29 and users
– 7,55). But, as the T-test value -7,112 exceeds
critical values, with 95% level of confidence we can
conclude that there is a difference in achieved
competence levels at the end of the course between
non-users and users groups (average final
competence level value-judgement of non-users is
5,65 and users – 8,43). Namely, ePortfolio system
users achieve better learning outcomes (i.e.
competence levels) than non-users.
To find relationships, their strength and way
between input and output parameters, the correlation
coefficients were calculated. It was found that:
- There is a moderate positive correlation
between activities within ePortfolio system and the
number of improved homeworks – the main
parameter of reflection (correlation coefficient r=
Table 4: T-test.
0,492, correlation is significant at the 0.01 level (ɑ=
0,01));
- There is a moderate positive correlation
between activities within ePortfolio system and
achieved competence levels at the end of the course
(r= 0,475, ɑ= 0,01);
- There is a moderate positive correlation
between activities within ePortfolio system and
fulfilled external tasks (r= 0,613, ɑ= 0,01);
- There is a moderate positive correlation
between activities within ePortfolio system and
login files (r= 0,454, ɑ= 0,01). At the same time
there is weak correlation between activities within
ePortfolio system and initial test results (r= 0,169,
ɑ= 0,05). As a result, it could be concluded that
approbated ePortfolio system has considerable
impact on learners’ activities apart from initial
competence levels;
- There is also a positive correlation between the
number of login files and the number of improved
homeworks (r= 0,356, ɑ= 0,01), as well achieved
competence levels at the end of the course (r= 0,269,
ɑ= 0,01);
- There is a weak positive correlation between
initial test results and the number of improved
homeworks (r= 0,129), as well exam results (r=
0,258, ɑ= 0,01). It could be concluded that
ePortfolio system impacts its users’ competence
development and reflection improvement apart from
initial competence levels.
2.3.3 Assessment of Modified Model
In contradistinction to the first version of developed
ePortfolio system, where group composition
remained unchanged from initial ePortfolio activity
till the end of the course, its modified version
ReflectiveePortfolioSystem-DevelopmentandAssessmentinLivingLab
389

provides group formation anew for each activity
module.
Due to the fact that the university’s LCMS and
ePortfolio system are two independent information
systems, partial automation was made. Namely,
course instructor regularly downloaded all students’
papers into one directory (c:\ePortfolio). The system
divided these files into groups – registered submitted
papers (files) and group numbers in ePortfolio
system data base. New approach and implementation
of automation tool ensured regular group completing
and permanence of quantitative structure, although
the amount of all groups was decreased.
To validate modified ePortfolio system model,
an appropriate prototype was built and approbated in
2012/13 academic year at Riga Technical
University. Approbation results in Living Lab show
again effectiveness of the system.
Like in previous year, survey results regarding
modified system’s impact on users’ competence
improvement and reflection development mainly
displayed students’ confidence about system’s
positive impact on their reflection and competence
improvement.
At the model’s verification stage to find
relationship between input and output parameters of
modified ePortfolio system, representative sample of
99 students was discrete ranked into two groups:
non-users group (18 students) and experimental
group – ePortfolio system users with at least one
login (91 students).
After completion of Kolmogorov-Smirnov,
Mann-Whitney and T-tests, as well determination of
possible correlations, it was observed that tests’
results are similar to previous ones made for the first
prototype in 2011/12 academic year. However,
comparison and analysis of correlation coefficients
in both cases gave some suggestions regarding
efficiency of developed systems. Thus:
- There is a moderate positive correlation
between activities within modified ePortfolio system
and the number of improved homeworks – the main
parameter of reflection (r= 0,446, ɑ= 0,01);
- There is a moderate positive correlation
between activities within modified ePortfolio system
and achieved competence levels at the end of the
course (r= 0,565, ɑ= 0,01);
- There is a moderate positive correlation
between activities within ePortfolio system and
fulfilled external tasks (r= 0,493, ɑ= 0,01);
- There is also positive correlation between the
number of login files and the number of improved
homeworks (r= 0,304, ɑ= 0,01), as well achieved
competence levels at the end of the course (r=
0,393, ɑ= 0,01);
- There is no correlation between initial test
results and the number of improved homeworks (r=
0,023), as well exam results (r= 0,070). It could be
said that ePortfolio system impacts its users’
competence development and reflection
improvement apart from initial competence levels.
3 CONCLUSIONS
Approbated ePortfolio system encourages learners’
reflection, which is the part of critical thinking
process, and is realised through feedback links,
enables competence levels improvement, enhance
learning outcomes, and stimulates activities also
outside the system.
This model ensures system users’ active
availability for work and participation in group-
working activities to develop own reflection and
competence levels. This could be done through
collaboration with peers within ePortfolio groups by
assessing group members’ papers and suggesting
them necessary improvements, as well by thinking
critically on the own accomplishments. Reflective
approach brings new attitudes and better learning
outcomes.
Based on test results it might be concluded that
modified ePortfolio system has greater impact on its
users’ competence levels improvement than previous
system (accordingly, r= 0,565 against r= 0,475). On
the other side, the first developed ePortfolio system
has better results than modified system version in
facilitation of users’ reflection (accordingly, r=
0,492 against r= 0,475) and fulfilled external tasks
outside the system (accordingly, r= 0,613 against r=
0,493).
Developed ePortfolio system algorithmic model
could be used as a base for creation of further
modifications of reflection and competence
enhancement information systems.
Living Lab research method ought to be
considered as an excellent approach to validate and
verify information system models in educational
environment.
ACKNOWLEDGEMENTS
This work has been supported by the European
Social Fund within the project «Support for the
implementation of doctoral studies at Riga Technical
University».
CSEDU2014-6thInternationalConferenceonComputerSupportedEducation
390

The author is very grateful to Dr. Sarma Cakula, the
professor of Vidzeme University of Applied
Sciences, for provided constructive
recommendations and comments regarding possible
improvements of this paper.
REFERENCES
Adler, S., 2013. Six Steps to an Information Strategy.
Keynote at the IBM Forum, Riga, 25.09.2013.
Barrett, H., C., 2009. Balancing the Two Faces of
ePortfolios. Retrieved 11.11.2012 from:
http://electronicportfolios.org/balance/balance.pdf.
Barrett, H., C., 2012. Categories of ePortfolio Tools.
Retrieved 07.03.2012, from http://
electronicportfolios.org/categories.html.
Følstad, A., 2008. Living Labs for Innovation and
Development of Information and Communication
Technology: a Literature Review. The Electronic
Journal for Virtual Organizations and Networks, Vol.
10 (Special Issue on Living Labs), pp.99-131.
Gorbunovs, A., 2011. New Insight in Actuality of e-
Portfolio Systems, Communication & Cognition, Vol.
44, No.2, pp.95-110.
Haig, A., Beggs, K., Cadzow, A., Colthart, I., Hesketh, A.,
Peacock, H., Tochel, C., 2007. Efficacy of e-
Portfolios: a Systematic Review of the Evidence. The
5th ePortfolio and Identity Conference „ePIC 2007”,
Maastricht, pp.20-22.
Kirikova, M., Stecjuka, J., 2008. Report on EKD
methodology (EKD metodologijas parskats), pp.1-30.
Retrieved 10.10.2013, from http://stpk.cs.rtu.lv/sites/
all/files/stpk/materiali/sazi/EKD_parskats.pdf.
Pallot, M., 2009. Engaging Users into Research and
Innovation: The Living Lab Approach as a User
Centred Open Innovation Ecosystem. Webergence
Blog. Retrieved 25.08.2012 from: http://www.cwe-
projects.eu/pub/bscw.cgi/1760838?id=715404_176083
8.
Quinn, C. N., 2005. Engaging Learning: Designing e-
Learning Simulation Games. San Francisco: John
Wiley & Sons, Inc., pp.210.
Timmins, F., 2008. Making Sense of Portfolios. A Guide
for Nursing Students. Glasgow: Open University
Press, pp.167.
ReflectiveePortfolioSystem-DevelopmentandAssessmentinLivingLab
391