![](bg3.png)
in the assignment descriptor, testing procedure and an
XML file. After the evaluator has finished the submis-
sion for a particular assignment, the system checks
the validity of the XML file against an XML Schema.
If the result of XML validation is negative, the sys-
tem halts any further processing of the file and the
errors are returned to the evaluator. The evaluator has
to correct the errors and then resubmit the files. If the
validation of the XML file against the XML Schema
is positive, then the assignment description becomes
available for students to download or view and subse-
quently submit their solutions.
These submissions are stored in a database. The
system accepts the submissions only if they are sub-
mitted within the valid dates specified by the evalua-
tor. The next criteria that should be passed for com-
pletion of submission is that proper file names are
used, as specified by the evaluator in the assignment
description. The above operations are performed by
the File Verification module. After its operation is
complete the next module which comes into the pic-
ture is the Makefile Prep/Compilation module with
the task of preparing a makefile and then using this
makefile to compile the code. This module gener-
ates feedback based on the result of the compilation.
I.e. if there are any compilation errors, then the sys-
tem rejects the submission and returns the compila-
tion errors to the student. Thus the submission is com-
plete only if the student is able to pass the operations
of both file verification module and the compilation
module. After the submissions are complete the next
module executed is the testing module which includes
both dynamic testing as well as static testing. The dy-
namic testing module accepts students’ submissions
as input, generates random numbers using the random
number generator module
1
(not shown in figure). It
then uses the testing procedures provided by the eval-
uator to test a program on the randomly generated
inputs. Sometimes it is not enough to test the pro-
gram on randomly generated inputs (e.g. where the
test fails for some boundary conditions). By using the
randomly generated inputs we cannot be sure that the
program undergoes specific boundary tests on which
the test may fail.
To overcome this drawback, facilities have been
provided by which the evaluator can specify his/her
own test cases. The Evaluators have the freedom to
specify the user defined inputs either directly in the
XML file or he may also specify the user defined in-
puts in a text file (if the number of inputs is signif-
icant). This is an optional facility and the evaluator
may or may not be using it based on the requirements
of the assignment. If the results of testing the pro-
gram on random input as well as user defined inputs
1
This module generate the random numbers required for
testing the programs
are positive, then the next phase of testing that be-
gins is the correctness of the program on execution
time as well as space complexity. During this test-
ing, the execution time and the space usage of the
user program are compared against the execution time
and the space usage of the model solutions submitted
by the evaluator. During this testing, the execution
time of the student program as well as the evalua-
tor’s program are determined for a large number of
inputs. The execution time of the student’s program
and the evaluator’s program are compared and ana-
lyzed to determine whether the execution time is ac-
ceptable or not. If the test for space and time complex-
ity goes well then the next phase is the Static Analysis
or the Style analysis. Static testing involves measur-
ing some of the characteristics of the program such as,
average number of characters per line, percentage of
blank lines, percentage of comments included in the
program, total program length, total number of condi-
tional statements, total number of goto, continue and
break statements, total number of looping statements,
etc. All these characteristics are measured and com-
pared with the model values specified by the evaluator
in the XML file. Testing and Grading modules works
hand in hand as testing and grading are done simulta-
neously. For example, after testing a program on ran-
dom inputs as well as user defined inputs is over, the
grading for that part is done at that moment, without
waiting until all testing is complete. If at any point
of time during testing, the programs are rejected or
aborted, then the grading that has been done up to that
point is discarded (but feedback is not discarded).
Security is a non-trivial concern in our system
because automatic evaluation, almost invariably, in-
volves executing potentially unsafe programs. This
system is designed under the assumption that pro-
grams may be unsafe and executes programs within
a ’sand-box’ using regular system restrictions.
5 AN EXAMPLE
Let us start with an example that is very common for
a data structures course, for example: the Mergesort
Program. As the regular structure of a C program con-
sists of a main function and a number of other func-
tions performing different activities, we have decided
to break down the Mergesort Program into two func-
tions (1) Mergesort function - This function performs
the sorting by recursively calling itself and the merge
function. (2) Merge function - This function will ac-
cept two sorted arrays as input and then merge them
into a third array which is also sorted. As a White
Box testing strategy is followed, testing and award-
ing marks on correct output generated by the program
will be of no use because this will not distinguish be-
WEBIST 2006 - E-LEARNING
198