T-MASTER

A Tool for Assessing Students’ Reading Abilities

Erik Kanebrant

, Katarina Heimann M

uhlenbock

, Soﬁe Johansson Kokkinakis

, Arne J

onsson

Caroline Liberg

Asa af Geijerstam

, Jenny Wiksten Folkeryd

and Johan Falkenjack

Department of Computer and Information Science, Link

oping University, Link

oping, Sweden,

Department of Swedish, G

oteborg University, G

oteborg, Sweden,

Department of Education, Uppsala University, Uppsala, Sweden

Keywords:

Reading Assessment, Vaocabulary Assessment, Teacher and Student Feedback.

Abstract:

We present T-MASTER, a tool for assessing students’ reading skills on a variety of dimensions. T-MASTER

uses sophisticated measures for assessing a student’s reading comprehension and vocabulary understanding.

Texts are selected based on their difﬁculty using novel readability measures and tests are created based on the

texts. The results are analyzed in T-MASTER, and the numerical results are mapped to textual descriptions

that describe the student’s reading abilities on the dimensions being analysed. These results are presented to

the teacher in a form that is easily comprehensible, and lends itself to inspection of each individual student’s

results.

1 INTRODUCTION

The importance and teaching effect of ﬁnding the

right reading level for each student, i.e. to ﬁnd the

”zone of proximal development” (ZPD), is a well-

known fact from both a theoretical and an empiri-

cal perspective (Vygotsky, 1976). The task of ﬁnd-

ing appropriate texts for different groups of students

gets, however, more demanding from grade 4 and on-

wards. To begin with, the texts become longer and

more complex in their structure. Students who have

a reading ability solely adjusted to more simple texts

will get into problem. A little higher up in school

around grade 7, the texts become more and more sub-

ject speciﬁc, which is especially visible in the vocab-

ulary. Text structure and vocabulary are, thus, two

very important aspects that can cause problems for

students who are not so experienced readers.

Our long term goal is to support reading for ten to

ﬁfteen year old Swedish students. The means for this

is a tool that assesses each individual student’s read-

ing ability, presents the results to the teacher to facil-

itate individual student feedback, and automatically

ﬁnds appropriate texts that are suitable and individu-

ally adapted to each student’s reading ability.

In this paper we present a fully functional tool for

Swedish that presents a proﬁle of a student’s reading

comprehension and vocabulary understanding based

on sophisticated measures. These measures are trans-

formed to values on known criteria like vocabulary,

grammatical ﬂuency and so forth. Four main aspects

of the reading process are in focus in this study; the

student’s reading comprehension, his/her vocabulary

understanding, levels of text complexity, and the sub-

ject area the text deals with. We will ﬁrst in Sec-

tion 2 present the models of reading upon which the

tests are based, how the tests are constructed and how

text complexity is measured. In Section 3 we present

the experiments that have been conducted this far.

Section 4 presents the actual toolkit and Section 5

presents some conclusions and future work.

2 MODELS OF READING

Common to models of reading in an individual-

psychological perspective is that reading consists

of two components: comprehension and decoding,

e.g. Adams (1990). Traditionally the focus has been

on decoding aspects, but in later years research with a

focus on comprehension has increased rapidly. Some

studies of comprehension concern experiments where

different aspects of the texts have been manipulated in

order to understand the signiﬁcance of these aspects.

In other studies interviews with individuals or

group discussions are arranged in order to study how

a text is perceived and responded to and how the

reader moves within the text, e.g. Liberg et al. (2011);

220

Kanebrant E., Heimann Mühlenbock K., Johansson Kokkinakis S., Jönsson A., Liberg C., af Geijerstam Å., Wiksten Folkeryd J. and Falkenjack J..

T-MASTER - A Tool for Assessing Students’ Reading Abilities.

DOI: 10.5220/0005410902200227

In Proceedings of the 7th International Conference on Computer Supported Education (CSEDU-2015), pages 220-227

ISBN: 978-989-758-107-6

 2015 SCITEPRESS (Science and Technology Publications, Lda.)

Langer (2011). This last type of studies is very often

based on a socio-interactionistic perspective. What

is considered to be reading is, thus, extended also to

include how you talk about a text when not being

completely controlled by test items. In such a per-

spective it has been shown how students build their

envisionments or mental text worlds when reading

by being out and stepping into an envisionment of

the text content, being in and moving through such

an envisionment, stepping out and rethinking what

you know, stepping out and objectifying the expe-

rience, and leaving an envisionment and going be-

yond (Langer, 2011, p. 22-23). These so called

stances are not linear and for a more developed reader

they occur at various times in different patterns dur-

ing the interaction between the reader and the text,

i.e. the reader switches between reading e.g. ”on the

lines”, ”between the lines”, and ”outwards based on

the lines”.

In a socio-cultural perspective the focus is made

even wider and reading is perceived as situated social

practices. The term situated pinpoints that a person’s

reading ability varies in different situations and with

different text types and topics. A model of reading

as social practice is proposed by Luke and Freebody

(1999). They map four quite broad reading practices

that they consider to be necessary and desirable for

members of a Western contemporary society: coding

practices, text-meaning practices, pragmatic practices

and critical practices. The ﬁrst two practices could

be compared to what above is discussed as decod-

ing, comprehension and reader-response. The last

two practices on the other hand point to the conse-

quences of the actual reading act, which at the same

time is the raison d’etre of reading: there can be no

reading without having a wider purpose than to read

and comprehend. These practices concern on the one

hand how to ”use texts functionally” and on the other

hand to ”critically analyse and transform texts by act-

ing on knowledge that they represent particular points

of views and that their designs and discourses can be

critiqued and redesigned in novel and hybrid ways”

(ibid p 5-6). A person with a very developed reading

ability embraces all these practices and can move be-

tween them without any problem. He/she is not only

able to decode and comprehend the text but also able

to use what has been generated from the text and to

take a critical stance, all this in order to extend his/her

knowledge sphere. All these perspectives taken to-

gether give both a very deep and a very broad under-

standing of the concept of reading. In order to mark

this shift from a narrower to a much more widened

concept, the term reading literacy is often preferred

over reading, see e.g. OECD (2009, p. 23).

When assessing students reading ability the types

of texts and reading practices tested have thus a much

broader scope today than earlier. It facilitates a more

delicate differentiation between levels of reading abil-

ity. Two well-tested and established studies of read-

ing ability in the age span focused here are the inter-

national studies of ten year old students (PIRLS) and

ﬁfteen year old students (PISA). Both these studies

are based on a broad theoretical view of reading, i.e.

reading literacy. The frameworks of PIRLS and PISA

concerning both the design of tests and the interpre-

tation of results in reading ability levels will there-

fore be important sources and resources for construct-

ing students’ reading ability proﬁle in this study, see

e.g. Mullis et al. (2009); OECD (2009).

2.1 Testing Reading Comprehension

The test of students’ reading comprehension in this

study will include, in accordance with a broad view,

different text types of different degrees of linguistic

difﬁculty, where the students are tested for various

reading practices within different topic areas. In the

construction of this test at least three degrees of lin-

guistics difﬁculty will be used. Accordingly at least

three prototypical texts will be chosen per school sub-

ject area. Each of these texts also includes testing dif-

ferent aspects of vocabulary knowledge.

Items testing the following reading processes are

furthermore constructed for each of these three texts,

cf. Langer (2011); Luke and Freebody (1999); Mullis

et al. (2009); OECD (2009): 1) retrieve explicitly

stated information, 2) make straightforward infer-

ences, 3) interpret and integrate ideas and informa-

tion, and 4) reﬂect on, examine and evaluate content,

language, and textual elements. In accordance with

the procedure used in the international study PIRLS,

these four reading processes are collapsed into two

groups in order to reach a critical amount of data to

base the results on, see below.

2.2 Vocabulary Tests

In assessing the vocabulary knowledge of students,

we have focused on the receptive knowledge of sub-

ject and domain neutral lexical items in Swedish nov-

els. The tests intend to cover a part of the lexical

knowledge divided into ages 10-15 years, in 7 sepa-

rate levels. Each level is represented by tests covering

reading comprehension and vocabulary knowledge.

We have used a reliable approach in creating vo-

cabulary tests by compiling text corpora at each level

adapted to the age of the students. These corpora are

then used to generate seven separate frequency based

T-MASTER-AToolforAssessingStudents'ReadingAbilities

221

Table 1: A description of subject speciﬁc and subject neutral content words.

Word categories Explanation Example (from tests for 10-14 year olds)

Subject neutral words

1. Most frequent The most common words, which could

appear in any text.

Person, sun, run, talk, nice, fast

2. Middle-less frequent Less frequent words above the 2000

most frequent words in age adapted

texts.

Disappointment, realize, decide, desperate,

barely

3. Genre typical (aca-

demic, news etc.)

Academic words in school context,

newspaper genre, descriptive texts

Explanation, development, consider, consist

of, type of, particularly

Subject/domain speciﬁc words

4. Every day words

(homonyms)

Words which have a common every day

meaning but also a subject or domain

speciﬁc meaning.

Mouth (of a person), mouth (of a river), bank

(of savings), bank (of sand), arm (a body

part), arm (a weapon)

5. Subject typical Words common to a type of text, cf.

natural science.

Motion, radiation, pollution

6. Subject speciﬁc Often only appears in one type of text

as unique words, a text in a biology text

book.

Assimilation, digestion, photosynthesis

vocabulary lists.

When studying and analysing vocabulary in texts,

words may be divided into subject-neutral and

subject-speciﬁc words. Subject speciﬁc words are

known as those words which would typically appear

in texts regarding speciﬁc subjects, school subjects or

domains in general. The categories are described in

a further developed category scheme partly inspired

by previous research by Nation (2014); Lindberg

and Kokkinakis (2007); Kokkinakis and Fr

andberg

(2013). Each of these groups can then be divided into

several sub-categories, as described in Table 1.

Words represented in each category are content

words, such as nouns, verbs, adjectives and adverbs.

These words are selected since they represent im-

portant parts of the knowledge as opposed to func-

tion words which are used to connect words into sen-

tences. Function words tend not to vary too much and

are normally not perceived as the most difﬁcult words

since they are quite common.

To select test items for the vocabulary test, the fre-

quency based vocabulary list from each level of age is

used as a point of departure. The 2000 most common

tokens, category 1 in Table 1, are deleted from the list

by comparing with other frequency based vocabulary

lists. Then, 50% nouns, 25% verbs, 25% adjectives

and adverbs are selected from that list, starting out

with the most frequent of category 2 in Table 1.

A synonym list of equivalent words and deﬁni-

tions is used to select 15 appropriate test items accom-

panied with 1 correct answer and 3 possible distrac-

tors. Distractors are words other than the correct an-

swer. As many as possible of the distractors are words

similar to the test item orthographically (in writing)

or phonologically (in sound). Previous research has

proven that test takers tend to select distractors with

any of the two similarities when not knowing the cor-

rect answer. Test items are ﬁnally composed of the

vocabulary test in context. This means that a sentence

in the text corpus is used to give a better example on

how a test item is used.

2.3 Text Complexity

The traditional way of measuring text complexity and

expected readability is to consider the Flesch-Kincaid

or Flesch Reading Ease measures (Flesch, 1948), or

for Swedish the LIX value (Bj

ornsson, 1968). The

underlying very simpliﬁed hypothesis is that short

sentences indicate an uncomplicated syntax, and that

short words tend to be more common and conse-

quently easier to understand. The measures men-

tioned are found to have bearing on entire texts, as

they are based on average counts, but do not suite an

inspection of isolated sentences or on a more scientif-

ically grounded analysis of texts.

The SVIT model (Heimann M

uhlenbock, 2013)

on the other hand includes global language measures

built upon lexical, morpho-syntactic and syntactic

features of a given text. It takes into account linguis-

tic features appearing at the surface in terms of raw

text, but also at deeper language levels. The ﬁrst level

includes surface properties such as average word

and sentence length and word token variation calcu-

lated with the OVIX (word variation) formula (Hult-

man and Westman, 1977). At the second level we

ﬁnd the vocabulary properties which are analysed in

terms of word lemma variation and the proportion of

words belonging to SweVoc — a Swedish base vocab-

ulary (Heimann M

uhlenbock and Johansson Kokki-

nakis, 2012). The third, syntactic level, is inspected

by measuring the mean distance between items in the

CSEDU2015-7thInternationalConferenceonComputerSupportedEducation

222

syntactically parsed trees, mean parse tree heights,

and the proportions of subordinate clauses and modi-

ﬁers. Finally, the fourth level, idea density present in

the texts, is calculated in terms of average number of

propositions, nominal ratio and noun/pronoun ratio.

All features taken into account when analysing the

texts in T-MASTER are listed in Table 2. Some of

them are quite straightforward, while others need an

explanation.

The Lemma variation index is a better way to

count the vocabulary variation in a text, as compared

to the OVIX formula. With the LVIX formula, all

word forms of the same lemma are tied together into

one item. For Swedish, being an inﬂected language,

there are for instance 8 inﬂected forms of a noun. It is

likely that if a student knows the meaning of the base

form of a given word, the meaning of most of its reg-

ular inﬂected forms can be deduced. Moreover, it is

calculated with a formula that minimizes the impact

of hapax legomenon, i.e. words occurring only once

in a corpus.

Words considered as Difﬁcult words are those not

present in category (C), (D) and (H) in the SweVoc

wordlist. In category (C) we ﬁnd 2,200 word lem-

mas belonging to the core vocabulary. Category (D)

contains word lemmas referring to everyday objects

and actions. Category (H), ﬁnally, holds word lem-

mas highly frequent in written text.

The syntactic features MDD, UA, AT, ET and PT

refer to properties in the dependency parsed sentences

in the text.

The Nominal ratio is achieved by calculating the

proportion of nouns, prepositions and participles in

relation to verbs, pronouns and adverbs.

Table 2: SVIT features.

Level Feature Abbrev

Surface

Mean sentence length MSL

Mean word length MWL

Vocabulary

Lemma variation index LVIX

Difﬁcult words DW

Syntactic

Mean dependency distance MDD

Subordinate clauses UA

Prenominal modiﬁer AT

Postnominal modiﬁer ET

Parse tree height PT

Idea density

Nominal ratio NR

Noun/pronoun ratio NPN

Propositional density Pr

3 TESTS

We have developed a ﬁrst series of reading tests with

texts and questions measuring reading comprehension

and vocabulary knowledge. The tests comprise ﬁction

texts and are expected to match three different levels

of reading proﬁciency in the 4th, 6th and 8th school

grades respectively. (The same test is used for the

hardest grade i test and the easiest grade i+ 1 test giv-

ing seven levels in total.)

The tests were carried out in a total of 74 schools

and more than 4000 students in three different cities

in Sweden. The size of a class is around 20 students,

but differs enormously, from as low as 5 students to

more than 40 in one class. Each student did a series of

three tests, with texts and vocabulary on three levels

of difﬁculty. The tests were conducted in the grade

order 6, 4 and 8.

3.1 Initial Text Selection

Twenty-two texts from the L

aSBarT corpus (Heimann

uhlenbock, 2013) and 31 texts from a bank of Na-

tional Reading Tests were examined both qualita-

tively and quantitatively. They were all manually

checked with regard to subjects and choice of words,

and texts that could be considered offensive or obso-

lete were discarded. The ambition was to ﬁnd suit-

able portions of narrative texts depicting a sequence

of events that would allow construction of test ques-

tions.

After the initial ﬁltering, 6 texts from the L

aSBarT

corpus were selected for the 6th grade, and 8 texts

from a bank of national reading tests for grades 4

and 8. The texts varied in length between 450 and

1166 words and were graded into levels of difﬁ-

culty after multivariate analysis with the SVIT model.

Earlier experiments showed that for the task of dis-

criminating between ordinary and easy-to-read chil-

dren’s ﬁction, linguistic feature values at the vocabu-

lary and idea density levels had the highest impact on

the results of automatic text classiﬁcation (Heimann

uhlenbock, 2013). We therefore chose to reward

the features mirroring vocabulary diversity and difﬁ-

culty, in addition to idea density, when the metrics did

not unambiguously pointed towards signiﬁcant differ-

ences at the syntactic level. Based on the SVIT model

we selected 7 texts for use in the ﬁrst tests that were

later reduced into 6, see below.

3.2 Some Findings

We will not present all results from the experiments in

this paper, only some ﬁndings that assess the quality

of the tests.

We ﬁnd that students perform better on simpler

than on more difﬁcult text, which corroborates the

SVIT model. For the tests conducted in grade 6,

T-MASTER-AToolforAssessingStudents'ReadingAbilities

223

many students acquired top scores and therefore two

of the texts from grade 6 were also used for grade

4 providing a stronger correlation between the stu-

dents’ results and the text’s difﬁculty. We saw an

even stronger correlation between text complexity, in-

dicated by SVIT, and response rates of the weakest

students, i.e. those whose overall test results were <

2 standard deviations below the average. This ob-

servation held for all three school levels. Given that

the SVIT measures were used as benchmark in the

initial levelling phase, we believe that our ﬁndings

strongly support the hypothesis that these measures

are able to grade a text’s complexity and hence read-

ability (M

uhlenbock et al., 2014).

There is, further, a statistically signiﬁcant corre-

lation between the students’ results on the vocabu-

lary tests and the reading tests for all six levels. This

shows that the tools and theories used to develop the

tests are applicable. Note that the vocabulary test

comprises domain neutral every day words from the

same corpus as the readability texts. The purpose is

to assess a general vocabulary competence.

4 T-MASTER

To conduct the tests and provide feedback we have

developed a tool for teachers that has been distributed

to all teachers with students that did the tests (Kane-

brant, 2014). The tool has only been used once, for

ﬁction texts, but is currently used for social science

texts and will be used for natural science texts after

which we hope to make it publicly available. It allows

teachers to get results on reading ability for each indi-

vidual student. The tool is password protected to en-

sure that results only can be accessed by the teacher.

The response texts intend to describe the reading com-

prehension competencies and vocabulary knowledge.

There are other assessment systems available for

English

. LetsGoLearn

is one interactive tool shar-

ing our idea of having students read texts and answer

questions and use that as a basis for the assessment.

The main differences are in how the tests are devel-

oped and texts selected. T-MASTER, for instance,

uses tests that consider contextual issues in more de-

tail and select texts using novel, and complex, read-

ability measures.

The current version of the system supports two

kinds of users, students and teachers. The system

is implemented as a Java web application and is ac-

cessed as a webpage using a web browser. All tests,

http://usny.nysed.gov/rttt/teachers-leaders/assessments/

approved-list.html gives one overview

http://www.letsgolearn.com

responses and analyses are stored in a database that

allows for easy modiﬁcation of tests, additional anal-

yses and facilitates security. To gain access to the sys-

tem one must be given login credentials beforehand.

In creating these credentials the user is also assigned a

certain role within the system, either as a student or a

teacher. Every user will use the same login page and

is after submitting their login information redirected

to the part of the system relevant to their role.

The students can use the system to take reading

comprehension and vocabulary tests in order to as-

sess their reading level concerning several different

factors. Each test consists of a text of a certain subject

accompanied by a reading comprehension test and a

vocabulary test. The test forms are presented on the

screen while the text can be supplied on paper or on

the screen. The tests are not supposed to be memory

tests, it is, thus, important to ensure that the students

can read the text over and over again, even if the text

is on the screen.

The teacher can log in to the system to receive

feedback in the form of a generated text describing

the reading level of their students for the speciﬁc texts

they have been tested on.

T-MASTER has been used to give feedback to all

teachers that participated in the ﬁrst test series, which

was done only on paper. Out of 126 teachers 74

(60%), have been using the tool to get student feed-

back. It is not that surprising that not all teachers have

used the tool. Although we do not have data on indi-

vidual teachers, students in grade 6 will not have the

same teacher in grade 7, i.e. the teachers can not use

the result to help a student. Teachers in grade 8 will

only have the students one more year. Teachers in

grade 4, on the other hand, will have the same stu-

dents for another 2 years and we believe that more or

less all of those teachers have used the tool. We also

included a questionnaire asking teachers if they liked

the tool. Unfortunately, only four teachers responded

and needless to say, they liked the tool and found it

easy to use. Currently another 4500 grade 4, 6 and 8

students use it to do the tests in order to automatically

create individual reading proﬁles.

4.1 Student Perspective

For each subject and school year there are 2-3 difﬁ-

culty levels. For each difﬁculty level the student will

read a text and take both a reading comprehension test

and a vocabulary test. Furthermore there are differ-

ent sets of texts for each school year, so student A

will read texts from a different set than student B. The

texts of one set correspond to the same difﬁculty level

as the texts from another set. At the top of the screen

CSEDU2015-7thInternationalConferenceonComputerSupportedEducation

224

Figure 1: Student test page for the reading test. The ﬁrst

item is an example to illustrate how the students are sup-

posed to ﬁll in the form. The ﬁrst item, termed Fr

aga 1 can

be translated as: ”Question 1: Whom decides which penalty

to be imposed on someone in society” and the fours possi-

ble answers are (from left to right) ”A police”, ”A judge”,

”A prosecutor”, ”A victim”.

there is a progress bar showing the students progres-

sion in the test set he or she has been assigned.

Each test consists of 10-15 multiple choice ques-

tions where the student can mark one answer, see Fig-

ure 1. When submitting a test the student will receive

a message asking the student whether he or she wants

to have a second look at the answers or to submit

the test. To facilitate receiving complete test sets the

system will prompt the students that there are unan-

swered questions; You have X unanswered questions.

After submitting the ﬁrst test for a text, the reading

comprehension test, the student will be moved on to

the vocabulary test of that text. After having submit-

ted both tests the student is given the choice to either

log out to continue at a later point or move on to the

next test. This procedure will continue until the stu-

dent has submitted the tests for each of the difﬁculty

levels he or she is assigned in the current subject. If

the browser for whatever reason is closed during a test

the student will after logging in again be forwarded to

the test after the last submitted one in the series.

4.2 Teacher Perspective

When logging on to the system the teacher will see a

list of all the students of his or her class that have re-

sults stored in the system, see Figure 2. Privacy is im-

portant, especially allowing researchers access to the

results without revealing individual students ID, but

at the same time allowing teachers to monitor indi-

vidual students. Therefore, the list comprises student

IDs only. The teacher will have a list not accessible in

the system where all the IDs are connected to speciﬁc

students thus keeping the students’ data anonymous

Figure 2: Teacher start up page. The text in the box can

be translated as: ”The students are organised according to

the student ID’s that the teacher gave them when they did

the tests. Only the teacher that can identify an individual

student.” ElevID can be translated to ”Student ID” and visa

resultat can be translated to ”show results”.

in the system.

Clicking on a student’s ID will generate a list of

tests the student has taken. The teacher can then ei-

ther access each result individually or all at once. The

result is presented as a text describing the student’s

performance viewed from several aspects of reading

comprehension and vocabulary capacity, as presented

above.

Each item in the reading comprehension test is as-

signed a certain type of reading process. The system

recognizes the four types of processes used. They are

collapsed into two groups, termed local (retrieve ex-

plicitly stated information and make straightforward

inferences) and global (interpret and integrate ideas

and information and reﬂect on, examine and evalu-

ate content, language, and textual elements), as de-

scribed above. The reading comprehension scores are

collapsed into ﬁve levels. The vocabulary test is sim-

ply scored based on how many correct answers the

student supplied and grouped into three levels.

Thus, our tests can extract ﬁve levels of read-

ing comprehension and three levels of word under-

standing, i.e. all in all ﬁfteen different levels for each

subject giving the teacher a very detailed descrip-

tion of a student’s reading strengths and weaknesses.

The ﬁfteen levels of achievement are collapsed into

four achievement levels

: 1) (L)ow reading ability, 2)

(Av)erage reading ability, 3) (H)igh reading ability,

and 4) (Ad)vanced reading ability, see Table 3.

The generated feedback consists of two main

parts, see Figure 3. The upper half is a descrip-

tion of the text’s readability level and style as pre-

There is a ﬁfth group, D, including those students that de-

viates from what can be expected. They are very few, to-

gether D1 and D2 account for less than 4%.

T-MASTER-AToolforAssessingStudents'ReadingAbilities

225

Table 3: Types of reading comprehension and vocabulary understanding.

Reading comprehension

Vocabulary understanding

Low Medium High

A few questions correct: both local and global L L D1

Half of the questions correct: local > global L Av Av

Half of the questions correct: global > local L Av Av

Most questions correct: local > global D2 H Ad

All global correct and most local D2 H Ad

Figure 3: Excerpt from feedback text.

sented above, Section 2. The lower half, after ”Re-

sultat” comprise three parts: a descriptive text on how

the student handles questions concerning their abil-

ity to retrieve explicitly stated information and make

straightforward inferences, another similar text on the

students comprehension of questions that are oriented

towards interpreting and integrating ideas and infor-

mation and reﬂect on, examine and evaluate content,

language, and textual elements and ﬁnally a text gen-

erated from the vocabulary test score that describes

the students vocabulary. The text in the lower part in

Figure 3 can be translated as:

Results

The text read by the student is long and has a

complex sentence structure. The word variation

and amount of difﬁcult words is high. When the

student has read this text he/she shows a reading

proﬁle where the most prominent features are that

the student answered a large amount of the ques-

tions correctly but manages the text based ques-

tions slightly more often than the interpreting and

reﬂecting ones. Thus, the student works with the

text more on the surface by mostly ﬁnding details

and obvious connections between closely related

aspects in the text. Less often does the student man-

age to go deeper and integrate aspects from other

parts of the text and examine parts of it or the text

as a whole. The student has good knowledge on the

tested words. The vocabulary of the student corre-

sponds to his/her age. The student can many dif-

ferent words as well as different meanings of single

words. The student seems to know enough words to

understand the content in a text adapted for his/her

age.

This text is generated based on the scores obtained

by the questions, grouped according to which aspect

they cover, giving one of the ﬁfteen levels discussed

above.

In the second part of this study, pedagogical rec-

ommendations are also included, adjusted to individ-

ual students’ results (see e.g. Deshler et al. (2007);

McKeown et al. (2009); Palincsar and Herrenkohl

(2002); Shanahan (2014)): For students with low

reading ability, it is suggested that the teacher mod-

els reciprocal teaching. For students with an aver-

age reading ability reciprocal teaching is suggested,

where students successively take more responsibility

in talking about the text. For students with high read-

ing ability, structured talk about a text in groups of

students is suggested. For students with advanced

reading ability, it is suggested that they are given more

challenging texts and therefore may need to take part

in one or all of the above described types of teaching

situations.

Examples of different reading techniques are

given in these recommendations. The teacher is also

urged to observe students’ reading behavior in other

reading situations, and to look for to what degree

students seem to integrate appropriate reading tech-

niques into their reading, and use them as strate-

gies (Goodman, 2014).

5 CONCLUSIONS

We have presented T-MASTER, a tool for automatic

grading of students’ reading abilities along a vari-

ety of reading didactic dimensions. T-MASTER fa-

cilitates teachers’ ability to give individually adapted

support for students by providing a reﬁned picture of

their reading ability along four dimensions: the sub-

ject treated by the text, a text’s difﬁculty, the student’s

vocabulary understanding and the student’s ability to

engage in various reading processes.

T-MASTER has been used to present reading abil-

ity results to teachers for more than 4000 students

CSEDU2015-7thInternationalConferenceonComputerSupportedEducation

226

having read ﬁction texts and is currently used to con-

duct readability tests for another 4000 students on so-

cial sciences texts.

The toolkit is developed for Swedish and at

present we have no plans on adapting it to other lan-

guages. Given that language speciﬁc resources are

readily available, this can easily be done. There are,

for instance, plenty of resources for English making it

easier to analyse language complexity.

Future research includes developing a module that

analyses texts in a subject area according to the SVIT

measures and suggests texts that have a reading difﬁ-

culty suitable for an individual student based on their

results on the test.

REFERENCES

Marilyn Jager Adams. Beginning to Read. Thinking and

Learning about Print. Cambridge, Massachusets &

London, England: MIT Press., 1990.

Carl Hugo Bj

ornsson. L

asbarhet. Liber, Stockholm, 1968.

Donald D. Deshler, Annemarie Sullivan Palincsar, Gina

Biancarosa, and Marnie Nair. Informed Choices

for Struggling Adolescent Readers: A Research-

Based Guide to Instructional Programs and Prac-

tices. Newark DEK: International Reading Associa-

tion, 2007.

Rudolph Flesch. A new readibility yardstick. Journal of

Applied Psychology, 32(3):221–233, June 1948.

Yetta M. Goodman. Retrospective miscue analysis: Illumi-

nating the voice of the reader. In Kenneth S. Good-

man and Yetta M. Goodman, editors, Making Sense

of Learners, Making Sense of Written Language: The

selected works of Kenneth S. Goodman and Yetta M.

Goodman, pages 205–221. NY: Routledge, 2014.

Katarina Heimann M

uhlenbock. I see what you mean. As-

sessing readability for speciﬁc target groups. Disser-

tation, Spr

akbanken, Dept of Swedish, University of

Gothenburg, 2013. http://hdl.handle.net/2077/32472.

Katarina Heimann M

uhlenbock and Soﬁe Johans-

son Kokkinakis. SweVoc - a Swedish vocabulary

resource for CALL. In Proceedings of the SLTC 2012

workshop on NLP for CALL, pages 28–34, Lund,

2012. Link

oping University Electronic Press.

Tor G. Hultman and Margareta Westman. Gymnasistsven-

ska. LiberL

aromedel, 1977.

Erik Kanebrant. Automaster: Design, implementation

och utv

ardering av ett l

aroverktyg. Bachelors thesis,

Link

oping University, 2014.

Soﬁe Johansson Kokkinakis and Birgitta Fr

andberg.

ogstadieelevers anv

andning av naturvetenskapligt

spr

akbruk i kemi

amnet i timss. [language use in nat-

ural science in chemistry in timss.]. Utbildning &

Demokrati, 3, 2013.

Judith A. Langer. Envisioning Knowledge. Building Liter-

acy in the Academic Disciplines. New York: Teach-

ers’ College Press., 2011.

Caroline Liberg,

Asa af Geijerstam, and Jenny Wiksten

Folkeryd. Scientiﬁc literacy and students’ movabil-

ity in science texts. In C. Linder, L.

Ostman, D.A.

Roberts, P-O. Wickman, G. Erickson, and A. MacK-

innon, editors, Exploring the Landscape of Scientiﬁc

Literacy, pages 74–89. New York: Routledge, 2011.

Inger Lindberg and Soﬁe Johansson Kokkinakis. Ordil - en

korpusbaserad kartl

aggning av ordf

orr

adet i l

aromedel

or grundskolans senare

ar [ordil - a corpus based

study of the vocabulary in text books in secondary

school]. Technical report, Institutet f

or svenska som

andraspr

ak, G

oteborgs universitet, 2007.

Allan Luke and Peter Freebody. Further notes

on the four resources model. Reading On-

line. http://www.readingonline.org/research/ lukefree-

body.html, 1999.

Margaret G. McKeown, Isabel L. Beck, and Ronette G.K.

Blake. Rethinking reading comprehension instruction:

A comparison of instruction for strategies and content

approaches. Reading Research Quarterly, 44(3):218–

253, 2009.

Katarina Heimann M

uhlenbock, Erik Kanebrant, Soﬁe Jo-

hansson Kokkinakis, Caroline Liberg Arne J

onsson,

Asa af Geijerstam, Johan Falkenjack, and Jenny Wik-

sten Folkeryd. Studies on automatic assessment of

students’ reading ability. In Proceedings of the Fifth

Swedish Language Technology Conference (SLTC-

14), Uppsala, Sweden, 2014.

Ina V.S. Mullis, Michael O. Martin, Ann M. Kennedy, Kath-

leen L. Trong, and Marian Sainsbury. PIRLS 2011

Assessment Framework. PIRLS 2011 Assessment

Framework., 2009.

Paul Nation. Learning Vocabulary in Another Language.

Cambridge University Press, 2014.

Joakim Nivre, Johan Hall, Jens Nilsson, Atanas Chanev,

uls¸en Eryigit, Sandra K

ubler, Svetoslav Mari-

nov, and Erwin Marsi. MaltParser: A language-

independent system for data-driven dependency pars-

ing. Natural Language Engineering, 13(2):95–135,

2007.

OECD. Pisa 2009 Assessment Framework. Key Compe-

tencies in reading, mathematics and science. Paris:

OECD, 2009.

Annemarie Sullivan Palincsar and Leslie Rupert Her-

renkohl. Designing collaborative learning contexts.

Theory into Practice, 41(1):26–32, 2002.

Timothy Shanahan. To teach comprehen-

sion strategies or not to teach them.

http://www.shanahanonliteracy.com/2014/08/to-

teach-comprehension-strategies-or.html., 2014.

Lev S. Vygotsky. Tænkning og sprog. Volym I & II.

openhamn: Hans Reitzel, 1976.

T-MASTER-AToolforAssessingStudents'ReadingAbilities

227