ASSISTING LEARNER’S IN MONITORING THEIR
CONCEPTUAL DEVELOPMENT
On the Design of a Computer-based Service to Support Conceptual Development
Adriana J. Berlanga, Howard Spoelstra, Kamakshi Rajagopal
Centre for Learning Sciences and Technologies (CELSTEC), Open University of The Netherlands, Netherlands
Alisdair Smithies, Isobel Braidman
Medical School, University of Manchester, U.K.
Fridolin Wild
Open University, Milton Keynes, U.K.
Keywords: Conceptual Development, Design, Service, Latent Semantic Analysis, Computer-based Service.
Abstract: This paper elaborates on the design of a computer-based service that supports conceptual development. Our
ambition is provide learners a way to compare their conceptual development against different reference
models, so they recognize the limits of their expertise. These models are (semi) automatically generated
from learning materials and learner text inputs using Latent Semantic Analysis, a technique that identifies in
input text materials the concepts and their relations. The paper explains the envisioned service presenting a
scenario that illustrates how it could be used in formal and informal learning context. After, the paper
elaborates the theoretical background behind the design of the service and, finally, it draws conclusions and
outlines future work.
1 INTRODUCTION
Modern educational approaches stress the
importance of activities such as problem based
learning, joint presentations, discussions,
collaborative knowledge co-construction and so on.
These activities often are assessed on the joint
group‘s performance, instead of on the individual
learner‘s performance. This makes it difficult for
individual learners to recognise their personal
understanding and knowledge of the topic of study.
For that, learners need to receive formative feedback
to identify the boundaries of their knowledge. Tutors
will no always be able to provide that feedback due
to workload. On the other hand, tutors require
reliable means of analysing the progress of learners
in order to provide appropriate guidance and
feedback to each individual. A means of providing
learners and tutors with a clear understanding of the
group‘ and the individual learners‘ conceptual
development, which is also economical with tutor‘s
time, is therefore required.
This paper presents the design of a computer-
based service aimed at supporting learners‘
conceptual development. The service is envisaged to
communicate information to learners intended to
engender the formation of an accurate, (targeted)
conceptualization of a particular topic. The
information should also allow learners to improve
their understanding of a topic without the immediate
need for a tutor. The design of the service is
theoretically grounded in research on expertise
development, a knowledge building process that
comprises both cognitive and social approaches.
The service is envisaged to process learner‘s
textual inputs (i.e., knowledge evidences) and to
return a graphical representation that reflects how a
learner conceptualizes a topic in terms of concepts
and their relations. Learners can then compare their
topic representations against a group reference
model, and/or a pre-defined reference model. The
294
Berlanga A., Spoelstra H., Rajagopal K., Smithies A., Braidman I. and Wild F. (2010).
ASSISTING LEARNER’S IN MONITORING THEIR CONCEPTUAL DEVELOPMENT - On the Design of a Computer-based Service to Support
Conceptual Development.
In Proceedings of the 2nd International Conference on Computer Supported Education, pages 294-299
Copyright
c
 SciTePress

group reference model is a representation of how
peers conceptualize the topic, while the ―pre-defined
reference model‖ is a representation of how in
learning materials (or tutor notes) the topic is
conceptualized.
The service explained in this paper goes beyond
existing approaches on measuring conceptual
development (Clariana & Wallace, 2007; Jeong,
2008; Shute, 2008) as we attempt to derive the
reference models and topic representations
(semi)automatically. To this end Latent Semantic
Analysis (Landauer et al., 2007) will be explored in
order to analyze (raw) text and extracting terms and
relationships with respect to their relatedness in
meaning, thereby enabling the generation of
conceptualisation models. Afterwards, these models
will be contrasted to obtain meaningful information
on conceptual development.
The rest of this paper is structured as follows.
First it presents a scenario to elucidate further the
need to monitor one‘s conceptual development and
outlines how the proposed service will work, which
is illustrated with a working prototype of the service.
Next, the paper presents the theoretical
underpinnings for the design of the service. Finally,
it discusses related work and indicates opportunities
for future work.
2 SCENARIO
Marion is a Medical Student in her third year of
study. This week she is working together with a
group of peers on a problem based case about
―cervical dysplasia‖. They have to collect related
information, and discuss and agree on the diagnosis
on the case. At the end of the activity, they have to
present their results to their peers. Learners are also
asked to keep a learning diary in the shape of a blog
to reflect on their learning. The learning activity
goes well, but Marion is not sure that she grasps all
the notions and concepts of the topic, and if her
understanding of the topic corresponds to the level
she is supposed to have reached at this point in her
learning career.
She then decides to use the conceptual
development monitoring service, which is a freely
available widget that can be included in her Personal
Learning Environment. Marion finds the topic space
―Oncology- 3
rd
year‖, created before by her tutor Dr.
Moon. She then submits the blog entry she wrote
about cervical cancer.
After processing Marion‘s blog entry, the service
displays a topic representation graph that includes
the concepts the blog entry contains and how these
concepts are related. The graph uses colours to
identify also different themes (i.e., clusters of
concepts). Figure 1 shows an example of a
representation graph. There Marion can see that in
her blog entry she is relating, for instance, the
concept of ―Cancer‖ with ―Prostate‖ and ―Breast‖.
But also that she relates the theme ‗Cancer‖ to the
theme ―Research‖.
Figure 1: Example topic representation graph.
Marion can also compare her topic representation
graph with other topic representation graphs. These
representations can be, for instance, a group
reference model (a graph that consists of all topic
representations of her peers) or a predefined
reference model, which represents the intended
learning outcomes (a topic representation her tutor
created using learning materials). For instance,
Figure 2 shows the graph Marion sees when she
compares her topic representation (in blue) with the
tutor‘s intended outcomes of the case about ―cervical
dysplasia‖ she was studying with her peers (in
green). There it becomes evident to her that in her
blog she is not mentioning topics related to cancer,
such as the ―Care‖ aspect (showed in the left top
corner of the graph) and the ―Keeping up to date‖
aspect (shown as ‗knowledge‘ in the middle of the
graph).
If Marion decides to ask Dr. Moon for feedback,
she will make her topic representation public, so Dr.
Moon can see it and provide feedback. If this is the
case, Dr. Moon might explain to her that she should
be more aware of the ―Care‖ aspect, which includes
―Diet‖, but also ―Cancer pharmacology‘. She
recommends Marion to read a book chapter as well
as two journal articles so Marion improves her
understanding.
Marion can also use the service to compare her topic
representation graph to that of any particular peer (of
ASSISTING LEARNER'S IN MONITORING THEIR CONCEPTUAL DEVELOPMENT - On the Design of a
Computer-based Service to Support Conceptual Development
295

the peers that have also made their representations
public). The service also keeps a record of Marion‘s
topic representation graphs, so she can compare her
representation graphs over time. This allows her to
gain insight into her progress in understanding the
topic. Figure 3 shows how Marion uses her topic
representation graphs. In this view she can make
graphs public and select which graphs she would
like to compare.
Figure 2: Example comparison topic representation vs.
predefined reference model.
Figure 3: View of topic representations.
Marion likes the service, so she decides to
introduce it in an informal learning context as well:
the Latin American literature group she is part of. In
this context she acts as tutor (―initiator‖) and creates
a topic space for ―magical realism‖. Her friends
include the service in their Personal Learning
Environments, join the topic space Marion created,
and use the service to submit their knowledge
evidences. Some of them submit a blog entry, while
others decide to submit an essay they wrote about
the topic. They work with the service to get topic
personal representation graphs of their
understanding side by side with their friend‘s
representation graphs of the topic. As the service can
create a topic representation graph that is based on
all their joint submissions, they can, when they meet
face-to-face, use that representation (the group
reference model) to see and discuss their shared
representation graph of the topic. They also have
been using well-known literature about the topic to
create a pre-defined reference model. This allows
them to compare and discuss about the differences
and similarities between the different models,
namely their personal topic reference models, the
group reference model, and the predefined reference
model.
3 THEORETICAL
BACKGROUND
The design of the service described above is
underpinned by the idea that learners develop their
expertise taking part in a knowledge building cycle,
which comprises cognitive and social processes.
Research on expertise has shown differences in
the knowledge base development between novice to
expert (Boshuizen & Schmidt, 1992). Experts and
novices differ in their knowledge usage, information
processing, and on how their knowledge structures
are organized (Arts et al., 2006). Findings in Law
(Nievelstein et al., 2008), Physics (Dufresne et al.,
1992), Management (Arts et al., 2006), and
Medicine (van de Wiel et al., 2000) have shown that
knowledge, with increasing expertise, is more
hierarchically structured than novices‘ knowledge,
which appears to be highly fragmented with
concepts loosely connected.
Learners develop their expertise taking part in a
knowledge building cycle, which comprises
cognitive and social processes. The cognitive
process focuses on perception, memory and
meaning; it assumes the memory is an active
processor of information, and knowledge, as a
commodity plays an important role in learning. The
social process assumes that learning is a social
activity that occurs in interaction with others. This
process takes into account both the learner and the
environment, where learners are pro-active
producers of the environment in which they operate.
Consequently, the service is designed to assist
learners in the development of their expertise from
both a cognitive and social perspective. It provides
CSEDU 2010 - 2nd International Conference on Computer Supported Education
296

learners with diverse ways of comparing their
understanding against different models, mainly
(Berlanga et al., 2009a):
(1) Predefined reference model, considering
indented learning outcomes described in, for
instance, course material, tutor notes, relevant
papers.
(2) Group reference model, considering the
concepts and the relations a group of people
(e.g., peers, participants, co-workers, etc.) used
the most.
The result is that, from a cognitive point of view,
the service provides learners with information that
contrasts their understanding of the topic against the
intended learning outcomes. From a social point of
view, the service provides information to learners so
they recognize the differences in how they
conceptualize a topic with respect to how others do.
Furthermore the service provides cultural and
cognitive artifacts to support the knowledge building
process. In this respect we base our work on Stahl‘s
knowledge building cycle (Stahl, 2006). Following a
social epistemological perspective (Brown &
Duguid, 1991; Lave & Wenger, 1991), Stahl models
the learning process as a mutual construction of the
individual and the social knowledge building. In his
view knowledge is a socially mediated product.
Individuals generate personal beliefs from their own
perspectives, but they do so on the basis of socio-
cultural knowledge, shared language and external
representations. These beliefs become knowledge
through social interaction, communication,
discussion, clarification and negotiation. Learners,
therefore, build knowledge both personally and
collaboratively.
Figure 4: Cycle of knowledge building (Stahl, 2006).
Figure 4 shows Stahl's cycle of knowledge
building. The diagram depicts how the personal and
the collaborative knowing building cycles interact.
The lower left corner shows the cycle of personal
understanding, which might start with a tacit pre-
understanding influenced by personal knowing. This
understanding may change if we explicate the
implications of that understanding and resolve
conflicts or fill gaps—by reinterpreting our meaning
structures—to arrive at a new comprehension. This
typically involves some feedback from e.g., our
experience with artifacts such as our tools and
symbolic representations. New comprehension
gradually settles in to become our new tacit
understanding and provides the starting point for
future understanding and further learning. If we
cannot resolve the problematic character of our
personal understanding alone, which happens mostly
when it is provoked by other people, then we need to
enter into an explicitly social process and create new
meaning collaboratively. To do this, we typically
articulate our initial belief in words and express
ourselves in public statements, and we enter into the
cycle of social knowledge building.
The right part of the diagram depicts how the
social process of interaction with people and with
our shared culture influences the individual‘s
understanding. This process is an interchange of
arguments that provides rationales for different
points of view, which eventually may converge on a
shared understanding.
Our service aims at supporting both knowledge
building cycles. On the left hand side of the cycle, it
provides a cognitive artifact (i.e., a graph
representing learner‘s topic representation) that can
help learners to understand and resolve conflicts or
fill in gaps in their knowledge. If this is not possible,
learners enter into the cycle of social knowledge
building. In this cycle, the service provides a
‗cultural artifact‘ (i.e. a graph that contains the
intended learning outcomes or a single graph that is
based on all peers graphs) that can help to foster
understanding.
Regarding how the service can be deployed in an
educational context, if a cognitive or a social
perspective should be followed, it is important to
stress that many educational practices start by
providing learners with explicit knowledge, and only
after learners have gathered what is considered a
critical mass of that knowledge, they allow learners
to acquire implicit, experiential, applied knowledge.
Likewise, to develop stimulating and suitable
instructional strategies, the instructional approach
needs to take into account whether learners are
novice or experts,. Researchers on instructional
design (Ertmer & Newby, 1993; Jonassen et al.,
1993) do not advocate a single theory of learning,
but emphasise that the instructional strategy and the
content addressed depend on the learner‘s expertise
level. They claim, therefore, that behavioural
ASSISTING LEARNER'S IN MONITORING THEIR CONCEPTUAL DEVELOPMENT - On the Design of a
Computer-based Service to Support Conceptual Development
297

strategies can facilitate mastery of the content of a
profession (knowing what); that cognitive strategies
are useful for acquiring procedural knowledge
(knowing how); and that constructivist strategies are
appropriate when dealing with ill-defined problems,
as summarized in Figure 5.
Figure 5: The Continuum of Knowledge Acquisition
Model (Jonassen et al., 1993).
4 CONCLUSIONS AND FUTURE
WORK
In this paper we introduced a computer-based
service aiming to help learners to monitor their
conceptual development. Our ambition is to
implement a service that processes –using Latent
Semantic Analysis– learners‘ learning evidences and
learning materials in order to identify concepts and
their relations to generate different reference models,
which then can be compared. We have elaborated on
a use case that explains how the service will work,
and we have explained the theoretical foundations
behind the design of the service. Particularly, we
discussed how the design of the service is grounded
in findings in the expertise development area and on
a knowledge building model. We also elaborated on
how the service can be used in educational contexts.
It is important to stress that a lifelong learning
perspective was also considered on the design of the
service. That is to say the service is designed in a
way that can be used only for personal use, or in
formal or informal learning situations. A personal
use of the service, learners will not share their
representations, but still will get information on how
they conceptualize a topic, create reference models,
and compare them. In a formal learning context,
tutors can create reference models and the service
can provide information to both tutors and learners.
In informal learning situations, the service can be
used by a group of people, not guided by a tutor, to
share their knowledge and reach a common
understanding.
Up to now, existing tools and software that
identify and approximate learner‘s conceptual
development have been explored, and a proof-of-
concept has been conducted to demonstrate the
generation of reference models (Berlanga et al.,
2009b).
Undoubtedly, more research is needed to
establish how learners would benefit the most from
comparing their conceptual development with the
proposed models (pre-defined reference model or
group model): whether it is good strategy for
learners to see comparisons with both models or,
whether, depending on their level of expertise,
comparisons with different models will be made
available. The type of reference model used may
depend on the level of learner development. The
group reference model, which is based on concepts
and their interrelationships, generated by peers,
would most likely be of use for an individual learner
at a novice level, as at this stage it would correspond
to his/her Zone of Proximal Development
(Vygotsky, 1978). As expertise develops, the group
reference models may still be appropriate, depending
on the development stage of the group as a whole,
but pre-defined reference models may be more
suitable to more advanced learners.
To ensure quality and applicability of our
service, we use a scenario-based design
methodology (Hensgens et al., 2009) which requires
conceptual validation with stakeholders and
formative evaluation of the service. To this end a
validation with stakeholders from the Medical
School, Manchester University will be conducted.
The feedback received will be adopted to design the
first version of the service. This version will be then
evaluated, using primarily qualitative methods, and
the results will be considered to develop a new
release of the service.
ACKNOWLEDGEMENTS
The work presented in this paper was carried out as
part of the LTfLL project, which is funded by the
European Commission (IST-2007-212578).
REFERENCES
Arts, A. J., Gijselaers, W. H., & Boshuijzen, H. (2006).
Understanding managerial problem-solving,
knowledge use and information processing:
Investigating stages from school to the workplace.
CSEDU 2010 - 2nd International Conference on Computer Supported Education
298

Contemporary Educational Psychology, 31(3), 387-
410.
Berlanga, A. J., Brouns, F., Van Rosmalen, P., Rajagopal,
K., Kalz, M., & Stoyanov, S. (2009a). Making Use of
Language Technologies to Provide Formative
Feedback. Proceedings of AIED 2009 Workshops
Proceedings Volume 10. Natural Language
Processing in Support of Learning: Metrics, Feedback
and Connectivity. Proceedings of the 14th
International Conference in Artificial Intelligence in
Education, Workshops Proceedings (pp. 1-8). July, 6-
7, 2009, Brighton, United Kingdom
Berlanga, A. J., Kalz, M., Stoyanov, S., Van Rosmalen, P.,
Smithies, A., & Braidman, I. (2009b). Using
Language Technologies to Diagnose Learner’s
Conceptual Development. Paper presented at the
conference Proceedings of the 9th IEEE International
Conference on Advanced Learning Technologies
(ICALT2009) July, 14-18, 2009, Riga, Latvia: IEEE
Boshuizen, H. P. A., & Schmidt, H. G. (1992). On the role
of biomedical knowledge in clinical reasoning by
experts, intermediates and novices. Cognitive Science,
16, 153-184.
Brown, J. S., & Duguid, P. (1991). Organizational
Learning and Communities-of-Practice: Toward a
Unified View of Working, Learning, and Innovation.
Organization Science, 2(1), 40-57.
Clariana, R., & Wallace, P. (2007). A Computer-Based
Approach for Deriving and Measuring Individual and
Team Knowledge Structure from Essay Questions.
Journal of Educational Computing Research, 37(3),
211 - 227.
Dufresne, R. J., Gerace, W. J., Thibodeau-Hardiman, P., &
Mestre, J. P. (1992). Constraining novices to perform
expertlike problem analysis: Effects on schema
acquisition. The Journal of the Learning Sciences, 2,
307-331.
Ertmer, P. A., & Newby, T. J. (1993). Behaviorism,
cognitivism, constructivism: Comparing critical
features from an instructional design perspective.
Performance Improvement Quarterly, 6(4), 50-70.
Hensgens, J., Rusman, E., Van Bruggen, J., Armitt, G.,
Osenova, P., & Simov, K. (2009). LTfLL.D-3.2:
Designing the LTfLL services: guidelines, scenarios
and communalities., from
http://hdl.handle.net/1820/2038
Jeong, A. (2008). Assessing Skills in Scientific Inquiry,
Argumentation, and Causal Modeling. Paper presented
at the conference Assessment Symposium, Florida
State University
Jonassen, D. H., McAleese, T. M. R., & Duffy, T. M.
(1993). A Manifesto for a constructivist approach to
technology in higher education. In T. M. Duffy, J.
Lowyck & D. H. Jonassen (Eds.), Designing
environments for constructive learning: Implications
for instructional design and the use of technology.
Heidelburg, FRG: Springer-Verlag.
Landauer, T. K., McNamara, D. S., Dennis, S., & Kintsch,
W. (2007). Handbook of Latent Semantic Analysis.
Mahwah, New Jersey: Lawrence Erlbaum Associates.
Lave, J., & Wenger, E. (1991). Situated learning:
Legitimate peripheral participation. Cambridge, MA:
Cambridge University Press.
Nievelstein, F., Van Gog, T., Boshuizen, H. P. A., &
Prins, F. J. (2008). Expertise-related differences in
ontological and conceptual knowledge development in
the legal domain. European Journal of Cognitive
Psychology, 20(6), 1043-1064.
Shute, V. J. (2008). Focus on Formative Feedback. Review
of Educational Research, 78(1), 153-189.
Stahl, G. (2006). Group Cognition: Computer Support for
Building Collaborative Knowledge. Cambridge: MIT
Press.
van de Wiel, M. W. J., Schmidt, H. G., & Boshuizen, H. P.
A. (2000). Knowledge Restructuring in Expertise
Development: Evidence From Pathophysiological
Representations of Clinical Cases by Students and
Physicians. European Journal of Cognitive
Psychology, 12, 323-355.
Vygotsky, L. S. (1978). Mind in Society. Cambridge, MA:
Harvard University Press.
ASSISTING LEARNER'S IN MONITORING THEIR CONCEPTUAL DEVELOPMENT - On the Design of a
Computer-based Service to Support Conceptual Development
299