![](bg4.png)
Sonak, 1999). Similarly, scaffolding strategies, are
often used to help learners in concept mapping activ-
ities (e.g, Chen et al., 2012) and they might ease the
assessment because the learners follow a well-defined
structure. However, they might hinder learners’ cre-
ativity and advancement (e.g, Amadieu et al., 2009).
B3: There Is Less Consensus on Measures and
Heuristics of “what is a good concept map”. Well-
known heuristics were proposed by Novak, which
relate to the validity and significance of (1) propo-
sitions, (2) hierarchies, (3) cross-links, (4) exam-
ples, and (5) comparison (to experts’ maps) (Novak
and Gowin, 1984). Although Novak’s heuristics are
widely used, they have subtle limitations. They do
not capture attributes of the artifacts that communi-
cate meaning (e.g., similarities, relatedness, order-
ing, prominence, adjacency, proximity, etc.). Addi-
tionally, Novak heuristics focus mainly a hierarchi-
cal (top-down) concept maps. As the supplemen-
tary demonstrates
1
, we found that students use vary-
ing ways to structure concept maps (e.g., networks,
mind maps, grids, etc.). And finally, existing heuris-
tics do not capture critical relational qualities, such
as (i) fluency, i.e., ease in generating concepts, rela-
tions, and relations’ types; (ii) originality, i.e., unique-
ness, rarity, the relevance of concepts and associa-
tions; and (iii) flexibility, i.e., conceptual categories,
themes, depth/breadth of thinking underlying a con-
cept map.
3.3 A New Method for Assessments
To overcome the challenges mentioned earlier, we de-
cided to instantiate the practice of concept mapping
as a cognitive and creative activity of externaliza-
tion of concepts and associations (Crilly et al., 2006).
Here, a concept map can be seen as technique of
brainstorming (Al-Samarraie and Hurmuzan, 2018).
Brainstorming is an act of externalization of ideas and
associations that leads to the production of spatial, vi-
sual, and conceptual artifacts (e.g., ideas, concepts,
designs, diagrams, writings, etc. (Crilly et al., 2006)).
Research into brainstorming as a tool for problem-
solving, creativity, and concept generation, has
yielded measures to evaluate the results of a brain-
storming activity. Primary measures involve the quan-
tity of ideas, quality of ideas, novelty of ideas, re-
source utilization (e.g., initial ideas), redundancy of
ideas, and categorization of ideas, among others (Al-
Samarraie and Hurmuzan, 2018). Quantity of ideas,
also known as fluency, represents the degree of ease
in processing inputs, such as understanding a prob-
lem, or the degree of ease in producing outputs, such
as generating ideas, concepts, or solutions (Thomp-
son et al., 2013). Fluency is widely quantified as the
number of ideas generated for a given situation.
Additionally, ideas can have several qualitative at-
tributes. One quality is originality, which refers to the
pertinence, novelty, and rarity of ideas (Puccio and
Cabra, 2012). Originality can be essential to quantify
unique, clever, and less frequent ideas but still valu-
able and appropriate for the subject. Another qual-
ity is flexibility, which refers to the conceptual cat-
egories and shifts in thinking underlying ideas, and
indicates heuristics and strategies adopted when re-
solving a problem or a challenge (Puccio and Cabra,
2012). Qualities of flexibility are primarily the results
of thematic analysis of the content. Thus, flexibility
can be an umbrella for concept maps’ qualities. Qual-
ities can be conceptual, relational, structural, or visu-
ospatial, which can be framed depending on the con-
text. In this view, we can define qualitative and quan-
titative measures for fluency, originality, and flexibil-
ity of concept maps.
• Fluency Measures. We quantify three fluency mea-
sures for concept maps. Concept fluency (CFlue):
the number of generated concepts. Relation fluency
(RFlue) the number of generated relations. Relation-
type fluency (RTFlue): number of generate relations’
types (i.e., unique relations’ labels). Fluency mea-
sures are quantitatively quantified.
• Originality Measures. We quantify five originality
measures for concept maps. We do so in two ways.
First, we qualitatively quantify originality through
novelty, uniqueness, or rarity of ideas. Thus we quan-
tify concept originality (COrig): the number of orig-
inal concepts, relation originality (ROrig): the num-
ber of original relations, and relation-type originality
(RTOrig): number of original relations’ types (i.e.,
unique relations’ labels). Second, we use Natural
Language Processing (NLP) approaches to quantita-
tively quantify the rarity scores of ideas. We quan-
tify the rarity score of ideas as the sum of the fre-
quency of each idea’s stem words. After cleaning up
misspellings and abbreviations, we tokenize each idea
using 1-gram (one word). We remove stop-words. In
NLP, stop words are common words of a language,
such as articles and prepositions. We generate the
stem of each word using a dictionary of stems. Stem-
ming unifies the wording used for all ideas, which is
appropriate for computing the frequency of words in
a corpus of ideas. And finally, we compute the rar-
ity score of each idea as the sum of the frequency
scores of its stem words. A lower rarity score means
that the words used for ideas are unique or less fre-
quent. Using this approach, we quantify concept-stem
originality (CSOrig): the rarity score of concepts and
relation-stem originality (RSOrig): the rarity score of
DotWrangler: A Method for Assessing Fluency, Originality, and Flexibility of Concept Maps and Diagrams at Scale
445