Yixue Adaptive Learning System and Its Promise on Improving

Student Learning

Haoyang Li

, Wei Cui

, Zhaohui Xu

, Zhenyue Zhu

and Mingyu Feng

Shanghai YiXue Educational Technology Inc., 10 Jianguozhonglu #5110, Shanghai, China

Department of Physics, University of California, Irvine, CA 92697, U.S.A.

Center for Technology in Learning, SRI International, 333 Ravenswood Ave, Menlo Park, CA 94025, U.S.A.

Keywords: Adaptive Learning System, Promise of Learning System, Evaluation.

Abstract: Adaptive learning systems offer personalized learning experience to students’ characteristics and abilities.

Studies have shown these systems can be effective learning tools. Many schools in the United States have

adopted adaptive learning systems. Yet development of such systems is still in the early stage in China, and

little empirical evidence exists on their efficacy. This paper describes an adaptive learning system, YiXue,

and presents two studies that were conducted in China to establish the promise of YiXue adaptive learning

platform and two comparison learning platforms used in an after-school English language arts course. The

results were promising: student learning efficiency was significantly higher with YiXue than with the

comparison platforms. Survey responses suggested that students in the treatment group felt more positive

about their learning experience.

1 INTRODUCTION

Through machine learning algorithms and data

analytics techniques, adaptive learning systems offer

personalized learning experience to students’

characteristics and abilities. The intent is to determine

what a student really knows and to accurately,

logically move the student through a sequential path

to prescribed learning outcomes and skill mastery.

Many learning products with adaptive features have

been developed, such as Cognitive Tutor, i-Ready,

DreamBox Learning, Achieve3000, and ALEKS.

Such systems constantly collect and analyze students’

learning and behaviour data and update learner

profiles. As students spend more time in it, the system

knows their ability better, and can personalize the

course to best fit their talents (Triantallou,

Pomportsis, & Demetriadis, 2003; van Seters et al.,

2012).

In schools, adaptive systems help close

performance gaps, introduce variety into the

classroom, provide real-time data on individual

students’ needs, and free instructors’ time for

individualized intervention. Studies have shown these

systems can be effective learning tools (VanLehn,

2011) and can promote student engagement. An

analysis of learning data from 6,400 courses, 1,600 of

which were adaptive, revealed that the adaptive

courses were more effective in improving student

performance than the 4,800 non-adaptive courses

(Bomash & Kish, 2015).

Given these advantages, many schools in the

United States have adopted adaptive learning

systems. Yet development of such systems is still in

the early stage in China, and little empirical evidence

exists on their promise. This paper presents two

studies that were conducted in China to establish the

promise and evaluate the efficacy of an adaptive

learning platform YiXue when being used in an after-

school English language arts (ELA) course. The

results were promising: Student learning gain was

marginally significantly higher when YiXue system

was used and student learning efficiency was

significantly higher with YiXue than with the

comparison platforms. Survey responses suggested

that students in the treatment group felt more positive

about their learning experience than their peers in the

comparison group.

Li, H., Cui, W., Xu, Z., Zhu, Z. and Feng, M.

Yixue Adaptive Learning System and Its Promise on Improving Student Learning.

DOI: 10.5220/0006689800450052

In Proceedings of the 10th International Conference on Computer Supported Education (CSEDU 2018), pages 45-52

ISBN: 978-989-758-291-2

2 YIXUE ADAPTIVE LEARNING

SYSTEM

Online education has developed rapidly in China in

recent years. According to the China Internet

Network Information Center (2017), the number of

online education users in China had reached 138

million, accounting for 19% of total Internet users, by

December 2016. YiXue Inc. was one of the first

organizations to develop an adaptive learning system

in China. Its system provides instructions and

supports for K–12 students and has the following

features:

 Fine-grained knowledge map in which

knowledge components were organized

hierarchically based on learning progression

relationship;

 Adaptive diagnostic pre-assessment;

 Automated differentiated instruction;

 Rich, high-quality learning repository of

various types of learning content;

 Immediate feedback and explanations to

students;

 In-class support and intervention by

teachers.

Supported by psychometric measurement models

and artificial intelligence, the system implements

mastery-based learning and tracks students’ mastery

of knowledge over time. In mastery learning, students

only advance to a new learning objective when they

demonstrate proficiency with the current one.

Mastery learning programs lead to higher student

achievement than more traditional forms of teaching

(Anderson, 2000; Gusky & Gates, 1986; Koedinger

& Aleven, 2007). A meta-analysis (Kulik, Kulik, &

Bangert-Drowns, 1990) of 103 studies of mastery

learning reported a mean effect size of 0.52 standard

deviation. When students are learning in YiXue, they

are first given a pre-assessment that diagnose which

knowledge components the students have mastered,

and which ones they haven’t, according to the

predefined hierarchical knowledge structure map.

Then the students will enter a learning-by-doing stage.

The knowledge that they demonstrate mastery on

during the pre-assessment are skipped during the

instructional phrase, while weak knowledge was

arranged in an optimal order for learning. The most

suitable content such as instructional videos, lecture

notes, worked examples, embedded practice

problems, and tests is delivered based on the student’s

ability which is estimated in real time by an item

response theory (IRT) model (van der Linden, 2016).

Fundamental to the YiXue design is provision of

feedback and reports to students and teachers.

Research has shown that frequent feedback increases

student learning (Hattie & Timperley, 2007; Kluger

& deNisi, 1996). After students enter solution to

problems, YiXue system provides immediate

feedback, and presents step-by-step explanations of

how the correct answer is obtained.

The system automatically collects student

responses to questions. At the end of each session, the

system presents students with reports on how they

performed. The system uses student response data in

real time to update estimates of student ability on

related knowledge components and adjusts

subsequent contents accordingly. In YiXue, data and

feedback are continuously available to teachers to use

in making appropriate instructional decisions, hence

enabling formative assessment, another practice with

a strong research basis (Black & Wiliam, 1998a,

1998b; Roediger & Karpicke, 2006; Speece, Molloy,

& Case, 2003; Heritage & Popham, 2013).

Further, YiXue implemented a differentiated

instruction model in which students who progress at

different rates receive different supports (Subban,

2006; Tomlinson & McTighe, 2006). Students who

progress quickly through the basic level of a topic are

given more challenging problems to accelerate their

preparation for advanced topics. A student who

experiences difficulties with a topic first gets a

computer-based tutorial, recommendations on review

of prerequisites, and finally help from teachers.

A YiXue classroom appears different from a

typical classroom because students are expected to

spend much of their time independently solving

problems on a computer, with frequent feedback,

instructional support, and remote tutoring, and using

features unique to the delivery of instruction on a

computer (e.g., animations, %correct) that are

designed to keep their level of engagement high

throughout the instructional period. In addition,

teachers have important and defined roles in the

Yixue classroom including the traditional role of

leading periodic classroom discussions on key subject

topics. However, teachers also receive easy-to-read

reports on classrooms’ and students’ individual

progress, and they are expected to use the data to

identify individual students who might be struggling

and what they are struggling, and provide them with

targeted support during class time. When teaching

with YiXue, teachers are expected to spend 1/6 of a

session to work with students face-to-face for

individualized instructions while the others continue

working with YiXue digital content. In this way,

YiXue supports a synergistic blending of the

CSEDU 2018 - 10th International Conference on Computer Supported Education

Figure 1: Illustration of a math problem in YiXue and its associated explanations.

teacher’s and technology’s role in delivering

instruction and supports differentiated instruction—

characteristics that are unlikely to be found in the

average “business-as-usual” classroom.

YiXue Inc. has developed instructional materials

for middle school mathematics, English, Physics,

Chinese, and Chemistry and is working on expanding

content coverage to the whole spectrum of K-12

education settings. In the year of 2017, Yixue has

been used by approximately 2,200 students in 17

cities in China, representing a broad range of student

populations with respect to socioeconomic status,

urbanicity, and performance levels.

Fig. 1 illustrated a math learning-by-doing

problem in YiXue system. At the center of the screen

is a math problem that the student needs to solve. The

top left corner of the screen shows the focal

knowledge that the current problem addresses. The

top right corner shows measures of practice and

progress of the student, including how long a student

has been learning within the system, percent correct

so far, and, of all the knowledge components that the

student is weak on, and what % of them that the

student has mastered through learning in Yixue. On

the right side of the screen, there are a few buttons

that when clicked, will bring up resources the student

may refer to if he/she has difficulty solving the

problem. The student may request to watch a short

video (usually 3-5 minutes) in which a teacher

explains the knowledge component and demonstrates

how to solve similar problems, or request for step-by-

step explanation of the problem. The bottom half of

Fig. 1 shows 3 messages. The first one provides a

high-level problem-solving plan overall, but doesn’t

Yixue Adaptive Learning System and Its Promise on Improving Student Learning

scaffold the problems or show single steps. The

purpose of presenting the plan at first is to prompt the

students to come up with their own solutions given

the overview. If the student still has trouble moving

on, he/she can click “next” to request for the

explanation 2, which shows the first step to solve the

problem, on top of explanation 1. Similarly,

explanation is only available upon request by the

student, and the student can go back making an

attempt at the problem at any moment. In the end, the

student can click “show me the answer” button to

request the system to show the correct answer. When

doing so, the student will be prompted that this

problem will be evaluated as “incorrect” if he/she

decides to continue.

It is worth mentioning that in order to increase

learning efficiency and prevent students from wasting

time over-attempting (aka. students take a guess-and-

check approach and repeatedly enter incorrect

answers), the system will automatically bring up the

explanations after 3 failed attempts. Note that even a

student provides the correct answer to the problem on

the first attempt, the system will still present the full

explanation, including explanation #1 to #3, plus all

forms of correct answers to prompt the student to

compare his/her solution to the one(s) provided by the

system to reinforce student’s understanding of the

knowledge.

3 METHODS

We conducted two small-scale randomized controlled

studies to examine the efficacy of YiXue Learning

compared with two popular online learning systems,

New Oriental Online (www.koolearn.com) and

Magic Grid (www.mofangge.com) and to examine

the promise of the YiXue platform on improving

student learning. These systems were selected

because they have similar learning content, format,

and intensity as YiXue. They have been popular

options for ELA learning in China. In New Oriental

Online, an experienced teacher first explains the

concepts thoroughly in an online video, and then

students practice the in-class quiz problems online.

Students check their understanding through a subject

test. Magic Grid helps students improve their

knowledge of English through problem practicing,

redoing problems that they got wrong, and carefully

reviewing lecture notes. Neither systems implement

adaptation during the learning phase.

4 EXPERIMENT 1: YIXUE AND

NEW ORIENTAL ONLINE

Sample and Random Assignment. In the first

experiment, 41 eighth-grade students (ranging from

13 to 15 years old) were recruited from one middle

school (School Y) in Shanghai. A pre-study survey

was used to collect students’ school grades in English

and other background information (such as gender,

family socioeconomic status, parent educational

level). We then used stratified block randomization

(Trochim, Donnelly, & Arora, 2016, p229) to

randomly assign the students to either control group

or treatment group. Students were first put in blocks

of four according to their prior achievement level, and

then two students in each block were randomly

assigned to the treatment group and the others to the

control group.

Research Procedure. On the first day of the study,

students in both groups took a paper-based pre-test on

the topic they would study, “Passive Tense,” which

had not been covered in school. Research staff trained

the students on using the technologies and navigating

through the systems before the learning sessions

began. Students in both groups studied the topic

online during two classes periods in one week for a

maximum of 100 minutes. The 21 students in the

treatment group used YiXue, and the 20 students in

the comparison group used New Oriental Online. The

experiment was conducted without teachers’ support,

and no other instructions outside the systems were

provided to the students. The learning schedule for

both groups was identical, including the break time

between online classes and the number of breaks. At

the end of the learning sessions, a paper-based post-

test was administrated to both groups. Students were

given 25 minutes to finish the pre- and post-tests. All

students finished the tests within in the given time,

and all 41 participating students completed all steps

(pre-test, learning, and post-test) of the experiment.

4.1 Experiment 2: YiXue and Magic

Grid

For the second experiment, we recruited a different

group of 87 eighth-grade students from School Y in

the same age range and with the same English skill

and background as those in Experiment 1. Students

were randomly assigned to conditions through a

similar randomization procedure as in Experiment 1,

with 44 students assigned to the control group and 43

to the treatment group. Experiment 2 followed the

same procedure as Experiment 1 and students studied

CSEDU 2018 - 10th International Conference on Computer Supported Education

for a maximum of 100 minutes except that in this

experiment, the comparison group used Magic Grid,

instead of New Oriental Online, and the learning topic,

“Adjectives and Adverbs,” had been introduced to the

students during regular classes. Thus, the online

learning sessions were to review this knowledge.

Students learning with Magic Grid got exposed to at

least 80 problems and their associated solutions and

explanations. All but 2 students in the comparison

group studied for 100 minutes. Students in the YiXue

group learned at their own pace and the amount of

instructional video watched and the number of

practice problems finished varied, based on student’s

proficient level on knowledge points within the

designated learning topic. 11 students in the treatment

group spent less or equal to 75 minutes on YiXue

system while the other 32 students used up all 100

minutes. All students completed all steps, namely,

pre-test, learning, and post-test, of the experiment.

4.2 Data Sources

Pre- and Post-tests. The pre- and post-tests were

developed for the experiments. The problems in the

tests were constructed by an experienced English

teacher in the local school and covered the focal

learning subjects during the experiments. Two

independent, experienced subject matter experts

reviewed the pre- and post-tests to ensure that they

were comparable in their coverage, overall difficulty,

types of items, and alignment with the local middle

school English learning standards. The experts also

checked to make sure that the test items are not over-

aligned with YiXue learning content, or the content

being taught by the comparison systems. Each test

included 45 multiple-choice, fill-in-the-blank, or

sentence transformation questions and the total score

on the test was 100.

Student Surveys. Two student surveys were

administered during each experiment. Students were

asked to complete an information survey about

themselves and their families’ educational

background on the first day of the study and to

respond to a post-study survey of their online learning

experience after the post-test was finished. In the

post-study survey, students were asked to reflect on

their online learning experience from three aspects:

learning efficiency, ease of use, and satisfaction with

the system. Each aspect included a few Likert scale

questions ranking from from completely disagree (1

point) to neutral (3 points) to completely agree (5

points).

5 RESULTS

5.1 Learning Outcome

The first step we did was to check baseline

equivalence of the control and treatment groups in

their prior knowledge. t-test results showed that for

both experiments, as expected given the randomiza

tion featured in the research design, analyses found

no difference between the treatment and control

groups in their pre-test scores (p > .05).

However, in Experiment 1 the average post-test

score of the control group was lower than the average

pre-test score, and scores for a few outlier students

dropped significantly from pre-test to post-test. We

are still examining the possible reasons for this

unexpected drop. Thus, we report learning results

only from Experiment 2.

We used multiple linear regression in R (lm function)

to model the mean differences in post-test scores

between students in the treatment and comparison

groups, controlling for their pre-test scores. The result

showed that after controlling for their pre-test scores,

students who used YiXue scored 3.8 points higher on

average on the post-test, comparing to students who

used Magic Grid and the difference is marginally

significant (F(2, 84) = 104.6, p = .09, r

= 0.71).

We then calculated learning efficiency, which was

defined as the gain score divided by the total number

of minutes that a student spent on online learning

(Table 1). We noticed that for Experiment 2, student’s

pre- and post-test scores was highly correlated (r =

.84). Across all students in both conditions, pre-test

average score was 52 points, and post-test average

score was 61 points. We first calculated gain score for

each individual student by subtracting his/her pre-test

score from the post-test score, and then divided each

student’s gain score by the amount of time they spent

learning in the systems (YiXue or Magic Grid

depending on their conditions) to compute learning

efficiency for each student. Results from linear

regression modelling suggested that learning

efficiency was significantly higher in the YiXue

group (p = .01).

5.2 Survey Results

32 students in Experiment 1 and 76 students in

Experiment 2 responded to the post-study survey.

Both groups had positive feedback about their online

learning experience (scores higher than 3 points)

compared with traditional in-class learning. Students

felt that they were more active and had more

flexibility.

Yixue Adaptive Learning System and Its Promise on Improving Student Learning

Table 1: Learning Efficiency Results from Experiment 2.

Condition

Mean gain score from

pre-test to post-test

Mean time spent in

system (minutes)

Mean learning efficiency

(gain score per minute)

YiXue

10.56

90.19

0.13

Magic Grid

6.5

98.86

0.06

Figure 2: Survey Results from Experiment 2.

Scores for YiXue were higher than for New

Oriental Online on 15 of the 17 survey questions,

especially regarding “ease of use” and “learning

efficiency.” Students in the treatment group felt that

learning, along with practice quizzes and

explanations of common mistakes, was very effective

because they were always aware of what knowledge

they had mastered and what they needed to strengthen.

The control group students felt that most of the time

they were only watching online videos repeatedly,

without practicing.

Moving from Experiment 1 to Experiment 2, we

removed 3 redundant questions in the post-survey as

students’ responses to the questions were not

differentiated from to their responses to other similar

questions. Thus, the survey was reduced to 14

questions. As shown in Figure 2, in the Experiment 2,

scores for YiXue Learning were higher than for

Magic Grid on all 14 questions and significantly

higher on questions on “learning efficiency” and

“satisfaction with the system.” During the post-study

interview, students in the control group stated that

they felt bored because they kept “practicing

problems without knowing why” and that they “never

understood why they made mistakes when solving

problems.”

6 CONCLUSION

In this paper, we introduce an adaptive learning

system, YiXue, and its features, and described two

CSEDU 2018 - 10th International Conference on Computer Supported Education

small-scale randomized controlled experiments that

aimed at establishing the promise of the YiXue

system. Overall, the results of the experiments

suggest that the YiXue adaptive learning system has

promise for improving student learning outcomes

effectively and efficiently. The studies did have

limitations; the sample sizes were small, the duration

was short, the focus was on only selected ELA topics,

and we were not able to use an external standardized

outcome measure. Thus, further research is warranted

to examine the efficacy of the YiXue adaptive

learning system. We are in the process of examining

the features of the systems used in the studies and

student’s learning activities during the experiment to

better understand what might have led to the

difference in learning outcomes and how the learning

outcome differs across students of different incoming

knowledge, or students of different self-efficacy in

math (a question in the student survey). We are

analysing student learning log data to investigate the

relationship between learning process and the

learning outcome, as well as how YiXue can be

improved to better facilitate learning. For instance,

analysis is being conducted to see if higher learning

gains in YiXue is associated with longer learning time,

better performance within the system, or longer

engagement time with videos. We are also planning

on more randomized controlled experiments with

larger sample size, longer duration to further evaluate

the efficacy of YiXue in other subjects, including

mathematics and physics. In the meantime, the

development team at YiXue is focusing on enhancing

the system’s adaptivity and effectiveness through

profiling students (Bouchet et al., 2013), attending to

student engagement level (Baker & Ocumpaugh,

2015) and cognitive styles (Yang et al., 2013), and

more accurately tracking student’s progress on fine-

grained knowledge points using state-of-art

algorithms and data-intensive modelling approaches.

As stated above, many randomized trials and

other sound studies of adaptive learning systems have

been conducted in the United States, but very few

rigorous experimental studies have been done in

China. With many schools in China introducing

online learning systems, there is broad interest in how

to select and use such systems and whether they lead

to improvement. With these studies, we have the

opportunity to contribute to much-needed knowledge

about online learning and adaptive learning in K–12

instruction, esp. in China. We expect the results of the

studies reported here to be meaningful to teachers,

educators, and parents.

REFERENCES

Anderson, J. R., 2000. Learning and memory: An

integrated approach (2nd ed.). New York: John Wiley

and Sons, Inc.

Baker, R.S.J.d., Ocumpaugh, J., 2015. Interaction-Based

Affect Detection in Educational Software. In R.A.

Calvo, S.K. D’Mello, J. Gratch, A. Kappas (Eds.),

Handbook of Affective Computing. Oxford, UK: Oxford

University Press.

Black, P., & Wiliam, D., 1998a. Assessment and classroom

learning. Assessment in Education: Principles, Policy

and Practice, 5, 7–74.

Black, P., & Wiliam, D., 1998b. Inside the black box:

Raising standards through classroom assessment. Phi

Delta Kappan, 80(2), 139–149.

Bomash, I., & Kish, C., 2015. The improvement index:

Evaluating academic gains in college students using

adaptive lessons. New York, NY: Knewton.

Bouchet, F., Harley, J.m, Trevors, G.J., & Azevedo, R.

(2013). Clustering and Profiling Students According to

their Interactions with an Intelligent Tutoring System

Fostering Self-Regulated Learning. Journal of

Educational Data Mining, v(5), Issue 1, April 2013.

China Internet Network Information Center., 2017.

Statistical survey report on Internet development in

China. Retrieved March 15, 2017 from http://www.

cnnic.cn/hlwfzyj/hlwxzbg/hlwtjbg/201701/P02017012

3364672657408.pdf

Guskey, T. R., & Gates, S., 1986. Synthesis of research on

the effects of mastery learning in elementary and

secondary classrooms. Educational Leadership, 43,

73–80.

Hattie, J., & Timperley, H., 2007. The power of feedback.

Review of Educational Research, 77, 88–112.

Heritage, M., & Popham, W. J., 2013. Formative

assessment in practice: A process of inquiry and action.

Cambridge, MA: Harvard Education Press.

Kluger, A. N., & deNisi, A., 1996. The effects of feedback

interventions on performance: A historical review, a

meta-analysis, and a preliminary feedback intervention

theory. Psychological Bulletin, 119(2), 254–284.

Koedinger, K., & Aleven, V., 2007. Exploring the

assistance dilemma in experiments with Cognitive

Tutors. Educational Psychology Review, 19, 239–264.

Kulik, C., Kulik, J., & Bangert-Drowns, R., 1990.

Effectiveness of mastery learning programs: A meta-

analysis. Review of Educational Research, 60(2), 265–

306.

Speece, D. L., Molloy, D. E., & Case, L. P., 2003.

Responsiveness to general education instruction as the

first gate to learning disabilities identification. Learning

Disabilities: Research & Practice, 18(3), 147–156.

Roediger, H., III., & Karpicke, J. D., 2006. Test-enhanced

learning: Taking memory tests improves long-term

retention, Psychological Science, 17(3), 249–255.

Subban, P., 2006. Differentiated instruction: A research

basis. International Education Journal, 7(7), 935–947.

Tomlinson, C. A., & McTighe, J., 2006. Integrating

differentiated instruction and understanding by design.

Yixue Adaptive Learning System and Its Promise on Improving Student Learning

Alexandria, VA: Association for Supervision &

Curriculum Development.

Triantallou, E., Pomportsis, A., & Demetriadis, S., 2003.

The design and the formative evaluation of an adaptive

educational system based on cognitive style. Computers

& Education, 41, 87–103.

Trochim, W., Donnelly, J., Arora, K., 2016. Research

Methods: The Essential Knowledge Base. Boston, MA:

CENGAGE Learning.

vanLehn, K., 2011. The relative effectiveness of human

tutoring, intelligent tutoring systems, and other tutoring

systems. Educational Psychologist, 46(4), 197–221.

van der Linden, W., 2016. Handbook of Item Response

Theory. Boca Raton, FL: Chapman and Hall/CRC.

van Seters, J. R., Ossevoort, M. A., Tramper, J., & Goedhart,

M. J., 2012. The influence of student characteristics on

the use of adaptive e-learning material. Computers &

Education, 58, 942–952.

Yang, T.-C., Hwang, G.-J., & Yang, S. J.-H., 2013.

Development of an adaptive learning system with

multiple perspectives based on students’ learning styles

and cognitive styles. Educational Technology & Society,

16 (4), 185–200.

CSEDU 2018 - 10th International Conference on Computer Supported Education