Requirements for an Online Integrated Development Environment for

Automated Programming Assessment Systems

Eduard Frankford

, Daniel Crazzolara

, Clemens Sauerwein

, Michael Vierhauser

and Ruth Breu

University of Innsbruck, Institute of Computer Science, Technikerstraße 21a, 6020 Innsbruck, Austria

Keywords:

Programming Education, Automated Programming Assessment Systems, Integrated Development

Environments, Requirements, Online IDE.

Abstract:

More and more introductory programming courses are being held online, using Automated Programming

Assessment Systems (APASs). Some APASs provide online editors where students can solve and submit their

exercises, because some course teachers want to avoid that students have to set up an Integrated Development

Environment (IDE) on their PCs, fearing that this could increase the entry barrier to the course. However,

most of the available online editors for APASs only provide rudimentary programming support, making it

harder to write code and, therefore, have the undesired side effect of increasing the entry barrier to learning

programming. To gain a better understanding of the requirements of an online IDE for APASs, we conducted

a survey asking 48 APASs users to give their opinions on the importance of different online IDE features.

Based on their responses we derived a set of main requirements for an online IDE within APASs. The survey

was complemented by a tool review of state-of-the-art online IDEs, to see whether existing online IDEs meet

the requirements. Finally, we evaluated whether the online IDEs can be integrated within APASs. This

study establishes a framework for online IDEs tailored to APASs, creating the foundation for subsequent

improvements.

1 INTRODUCTION

Automated assessment of exercises in programming

courses is becoming more and more important (Barra

et al., 2020). Particularly, in large introductory pro-

gramming courses, the number of participants can

exceed several hundred or even thousands of stu-

dents, making it infeasible to grade all exercises by

hand. Furthermore, recent developments related to

the COVID-19 pandemic, or the popularity of massive

open online courses (MOOCs) have resulted in a shift

towards virtual teaching and learning (Mekterovi

et al., 2023). As a result, APASs have emerged to sup-

port course instructors in handling these large number

of students (Mekterovi

c et al., 2023). APASs allow

students to submit their exercises online, which are

then automatically evaluated, using a build pipeline

https://orcid.org/0009-0005-5959-4936

https://orcid.org/0009-0007-3548-4665

https://orcid.org/0009-0009-9464-5080

https://orcid.org/0000-0003-2672-9230

https://orcid.org/0000-0001-7093-4341

to execute predeﬁned test cases on the students’ code

(Krusche and Seitz, 2018). Afterwards, the stu-

dents receive the test results as feedback, which they

can use to further improve their submissions. Most

APASs use version control systems combined with

continuous integration servers to handle submissions

(Chen et al., 2020). This introduces an entry bar-

rier to novice programmers, as most beginners are not

yet familiar with version control systems. As a re-

sult, some APASs already provide the ability for stu-

dents to solve the exercises in a built-in online editor

(Krusche and Seitz, 2018) (Mekterovi

c et al., 2023).

However, currently, most of these editors lack im-

portant IDE features, such as Syntax Highlighting,

Auto-Completion, or Key Shortcuts, to name but a

few. This makes programming in APASs unneces-

sarily difﬁcult and puts additional burden on students

yet again raising the entry barrier. To establish the

theoretical foundations to evaluate and improve on-

line IDEs, we seek to address the following research

questions:

• RQ1. What are the most important requirements

for online IDEs as perceived by students?

Frankford, E., Crazzolara, D., Sauerwein, C., Vierhauser, M. and Breu, R.

Requirements for an Online Integrated Development Environment for Automated Programming Assessment Systems.

DOI: 10.5220/0012556400003693

Paper published under CC license (CC BY-NC-ND 4.0)

In Proceedings of the 16th International Conference on Computer Supported Education (CSEDU 2024) - Volume 1, pages 305-313

ISBN: 978-989-758-697-2; ISSN: 2184-5026

305

• RQ2. To what extent do existing online IDEs

meet these requirements?

• RQ3. To what extend, and with what effort can

existing online IDEs be integrated within APASs?

Based on these research questions, our initial step

involved conducting a comprehensive analysis of fea-

tures provided by various well known desktop IDEs.

Using this foundational analysis, we surveyed APASs

users to gain insights about their preferences regard-

ing different IDE features. Based on these derived

preferences, we evaluated existing online IDEs to cre-

ate an extensive list showcasing which requirements

are fulﬁlled and which programming languages are

supported by these IDEs. In a third step, we evalu-

ated whether these tools are available as open-source

or if they are of a proprietary nature. This helps to get

a complete picture of the online IDE landscape.

The remainder of this paper is structured as fol-

lows: Section 2 provides an overview of related work.

Section 3 elaborates on the research methodology.

Section 4 presents the main ﬁndings of this study,

which are further discussed in Section 5. Section

6 outlines potential constraints of this study, and we

conclude in Section 7, summarizing the main insights

and reﬂecting on the broader implications of this re-

search.

2 RELATED WORK

(Kusumaningtyas et al., 2020) compared several on-

line IDEs, speciﬁcally evaluating their internet data

usage and the availability of supporting libraries.

Eleven online IDEs that support Python program-

ming, offer free access, and do not require to login

were examined. The analysis revealed that Replit,

Codechef, and Ideone have the most comprehensive

library support. Notably, Replit was found to require

the least data transfer, offering a user-friendly inter-

face that effectively organizes code and output, and

is free from distracting advertisements. Thus, Replit

is recommended for users with limited internet re-

sources, especially for learning computer program-

ming during the pandemic.

(Sinanaj et al., 2022) compared ten online com-

pilers, assessing their service offerings and execu-

tion performance. The following service offerings

were analyzed: (1) Open Source, (2), Personal Ac-

count, (3), Cloud Storage, (4) Size Cloud Storage, (5)

File Size, (6) Programming Language Supported, (7)

Choice Compiler Version, (8) Intellisense, (9) Collab-

orative Real-Time, (10) Permalink Share Code and

(11) Debugging Code. Findings showed that On-

lineGDB, JDoodle, Replit, and Paiza.IO offered the

most services, with JDoodle, CodeInterview, Ideone,

WandBox, OnlineGDB, Judge0, and Replit leading in

performance. Based on a combined assessment of ser-

vices, performance, and user-friendliness, JDoodle,

OnlineGDB, and Replit were recommended as the top

choices for beginners, students, and teachers.

(Satav et al., 2011) presents a comparative study

of various desktop IDEs for C, C++, and Java, with

the goal of identifying key speciﬁcations for develop-

ing a comprehensive and optimal IDE. The ﬁndings

indicate that more popular IDEs tend to offer better

user support and provide many features to assist the

creation of source code.

(Hausladen et al., 2014) developed an online IDE

for embedded systems and deﬁned the following

functional requirements: (1) Project and version man-

agement capabilities, (2) Control over the build pro-

cess including conﬁguration of compilers, (3) Abil-

ity to download or ﬂash the binary to the target hard-

ware connected to the client’s computer, (4) Debug-

ging capabilities, such as controlling the ﬂow of exe-

cution with breakpoints, and inspecting variables and

memory locations, and (5) Standard source code edit-

ing features like syntax highlighting and code inden-

tation. Additionally, they deﬁned the following non

functional requirements: (1) Stability and responsive-

ness, (2) Intuitive user interface and (3) Extensibility

regarding plugins.

(Tran et al., 2013) developed Ideol, an IDE that

signiﬁcantly focuses on collaborative programming

and learning features. Key features include a real-

time interactive editor for simultaneous multi-user

editing and instant change viewing, a real-time dis-

cussion board with enhanced tagging and comment-

ing capabilities. Furthermore, it allows debugging,

along with an interactive execution tool to provide

support for C/C++ programming. Designed for a

wide range of users, from beginners to experts, Ideol

offers a user-friendly interface and is compatible with

HTML5-supported browsers on various platforms, in-

cluding partial tablet support.

The related work shows that few scientiﬁc re-

search has been conducted regarding the deﬁnition

of requirements of online IDEs. We found that some

developers of online IDEs deﬁned speciﬁc functional

and non functional requirements for their developed

IDEs. Additionally, online IDEs have only been com-

pared regarding their internet data usage, availabil-

ity of libraries, performance and speciﬁc offered ser-

vices. However, a scientiﬁc requirements deﬁnition

and a detailed comparison of IDE features focusing

on supporting novice programmers is missing. We

seek to address this gap by ﬁrst creating a require-

ments checklist for online IDEs including common

CSEDU 2024 - 16th International Conference on Computer Supported Education

306

IDE features and then conducting a tool review to an-

alyze what existing tools fulﬁll which requirements.

3 METHODOLOGY

To address the aforementioned research questions (cf.

Section 1), we employed a two-stage approach: ﬁrst,

we collected requirements from APASs users, and

second, we conducted a systematic tool review. In the

following, we provide a detailed description of each

step.

3.1 Requirements Speciﬁcation

In order to gain an overview of the main requirements

for an online IDE in context of APASs, we ﬁrst con-

ducted a survey asking APASs users to rate the im-

portance of common (non-online) IDE features and

to mention whether they are missing some features.

The survey was distributed to APAS users from the

following universities: University of Innsbruck, Jo-

hannes Kepler University Linz and Paris Lodron Uni-

versity Salzburg. To reach as many participants as

possible, the survey was carried out online, and was

promoted directly on the APAS Artemis hosted at the

University of Innsbruck

. The survey was designed

to be completely anonymous using the LimeSurvey

tool

3.1.1 Survey

The survey was divided into three parts. First, we

asked participants about their familiarity and experi-

ences in using APASs (P). Second, we asked them to

rate IDE features with regards to their perceived im-

portance (F), and ﬁnally, two open-ended questions

concluded the questionnaire (O).

For the ﬁrst part, we asked participants the follow-

ing questions, with the main intention to collect initial

information about the participants of the survey

• P1: In which semester are you currently in?

• P2: In how many university courses (including

current ones) did you use Artemis?

• P3: What did you use to solve the programming

exercises in Artemis?

(Single choice regarding ﬁve options ranging

from “online editor only” to “local IDE only”)

In the second part, the survey included a list of

IDE features and participants were able to rate these

https://github.com/ls1intum/Artemis

https://www.limesurvey.org

regarding their perceived importance. The partici-

pants were asked to rate each of the listed features us-

ing a 5-point Likert scale ranging from “Very Impor-

tant” (1) to “Very Unimportant” (5). Each feature in-

cluded a brief description and screenshot to assist the

participants in understanding its functionality. Such a

description is depicted in Figure 1.

The following list shows all the features we eval-

uated and the corresponding descriptions displayed in

the survey:

• F0: Syntax Highlighting - distinguishes various

parts of source code using colors.

• F1: Debugger / Breakpoints - provides the ability

to pause code execution at a given line of code

and inspect the initialized variables.

• F2: Auto-completion / Code Suggestions - dis-

playing potential code completion / suggestions

based on the currently typed code.

• F3: Syntax Error Highlighting - underlining po-

tentially incorrect code leading to syntax errors.

• F4: Brace Matching Highlighting - highlights the

matching brace in code to improve the visibility of

its scope.

• F5: Key Shortcuts - integrated basic shortcuts

(e.g. copy, paste, cut, ﬁnd etc.

• F6: Compiler / Interpreter (Run Button) - possi-

bility to compile/run the code without executing

related tests.

• F7: Dedicated Shell Console - access to a con-

sole where code can be run manually (e.g. using

custom arguments or properties).

Finally, in the last part of the survey, we asked

two open-ended questions to collect additional fea-

tures and requirements we might have missed.

• O1: Do you have any further suggestions on how

the editor could be improved?

• O2: Are there any other highly important features

a code editor should have, in order to support you

in writing code?

3.2 Systematic Tool Review

Complementing the survey, to better understand the

state-of-the-art online IDEs, we conducted a system-

atic tool review. For this purpose, we searched three

digital libraries: IEEE Explore

, ACM Digital Li-

brary

and Springer

http://ieeexplore.ieee.org

http://dl.acm.org

https://link.springer.com/

Requirements for an Online Integrated Development Environment for Automated Programming Assessment Systems

307

Figure 1: Example of a feature description and screenshot provided to the survey participants.

The scope covered all publications from 2013 un-

til 2023 using the following search term: “online code

editor” OR “online IDE” OR “web-based IDE” OR

“browser-based IDE” OR “cloud IDE” OR “online

compiler”.

The search term was carefully chosen to include a

wide variety of IDE synonyms used in research. Be-

fore ﬁnalizing the search term, we conducted a pilot

search. This initial search helped to test the adequacy

and appropriateness of the chosen search string. Like

this we were able to verify that the used terms were

resulting in relevant literature. Additionally, this pi-

lot search resulted in insights into the volume and na-

ture of the available literature, which helped in deﬁn-

ing the scope of the review, including deﬁning which

databases to search and what time frame to consider.

In total we found 39 results in IEEE Explore, 243

results in the ACM Digital Library and 122 results in

the Springer database. On these results, we used the

following inclusion and exclusion criteria:

Table 1: Inclusion and Exclusion criteria.

Inclusion Criteria Exclusion Criteria

Papers Accessible in Full text Non-english articles

Presenting a new online IDE Not presenting an online IDE

Focusing on supporting programming Not focusing on code

Available to test online or open source Not available to test.

Based on the results we created a list of tools, and

prepared a checklist that includes the feature set as

proposed by the survey respondents. Following this,

each tool was tested, and a comprehensive list of the

provided features was extracted. In addition to the

provided features, the supported programming lan-

guages, as well as the integrated editor on which the

tool relied on, were noted down in the same docu-

ment. This review was conducted to ﬁnd commonly

offered and practically implementable features. It

also enabled a comparative evaluation of integrated

editors across various platforms, providing a broad

view of their strengths and weaknesses.

4 RESULTS

In the following, we present the results of the con-

ducted survey and tool analysis analysis.

4.1 Requirements for an Online IDE

For the survey that has been displayed in the APAS

Artemis we received 48 valid responses of mainly ﬁrst

year computer science students. In the following the

results will be presented.

4.1.1 Demographics

The preliminary questions revealed that the APAS is

primarily utilized by students in the initial stages of

their university journey, with a signiﬁcant concentra-

tion of users in their ﬁrst semester. The results fur-

ther show that around 60% of participants mostly use

GIT combined with a local IDE when solving pro-

gramming exercises. Only, approximately 20% of the

respondents use mainly the online editor and the re-

maining 40% use both equally. The exact distribution

can be seen in Figure 2. The substantial reliance on

external tools, like local IDEs and GIT, suggests the

need for improvement of the online code editor used

in the APAS, as this would help beginner program-

mers get started without having to learn GIT and in-

stall an IDE ﬁrst. This validates the necessity of this

study, aiming to understand the requirements for on-

line IDEs for APASs in general.

Figure 2: Preferred tools for solving programming exer-

cises.

CSEDU 2024 - 16th International Conference on Computer Supported Education

308

4.1.2 Features’ Importance Questions

Following the methodology outlined in Section 3, par-

ticipants were asked about the signiﬁcance they place

on various features offered by online IDEs.

The prioritization of IDE features according to the

participants’ responses is as follows:

• Syntax Highlighting: This feature has received the

highest number of “Very Important” ratings, mak-

ing it the most valued feature among the surveyed

participants. Syntax highlighting is crucial be-

cause it allows developers to quickly distinguish

elements of the code, such as variables and func-

tions, which can enhance readability and reduce

the likelihood of errors.

• Syntax Error Highlighting: Similarly to syntax

highlighting, syntax error highlighting is consid-

ered very important. It provides immediate feed-

back on errors, without the need to compile or ex-

ecute the code, which is essential for learning and

improving code.

• Compiler / Interpreter (Run Button): The feature

is considered important, reﬂecting its role in the

actual running and testing of the code. This in-

dicates that a seamless transition from coding to

execution is valued by users.

• Auto-completion / Code Suggestions: Rated as

very important, this feature suggests that users

value assistance in coding for speed and efﬁ-

ciency. It helps in reducing typos and understand-

ing available functions and methods.

• Debugger / Breakpoints: This feature is highly

rated, but already more on the lower end of re-

garding the importance compared to other fea-

tures. Indicating that beginner programmers

might not directly realize the importance of de-

bugging to understanding code ﬂow and identify-

ing bugs.

• Brace Matching Highlighting: This feature is im-

portant but has a slightly lower rating compared

to other features. It helps users in navigating com-

plex code blocks, which is useful in understanding

and maintaining code structure.

• Key Shortcuts: This feature has the most neutral

ratings. It indicates that while some users ﬁnd

shortcuts essential for faster coding, others may

not rely on them as much, which could depend

on the user’s familiarity with the IDE or personal

coding habits.

• Dedicated Shell Console: Although it has re-

ceived a signiﬁcant number of negative ratings, it

still has enough positive responses to suggest that

for certain tasks or users, a dedicated shell con-

sole within the IDE is an essential feature. This

might be more important for those who work with

command-line interfaces or scripting.

The aggregate data suggests that while some fea-

tures are universally acknowledged as critical (like

syntax highlighting and error detection), others are

subject to more personal preference (like key short-

cuts and shell consoles). This emphasizes the need

for customizable online IDEs that allow users to tai-

lor their environment to their personal workﬂow and

the tasks at hand. In Figure 3 we ordered the results

with the most important feature being Syntax High-

lighting and the least important being Dedicated Shell

Console using a weighted sum.

For the open questions, students frequently high-

lighted the need for enhanced compilation perfor-

mance in the Artemis platform. Currently, Artemis

does not support direct code compilation. Instead,

upon submission, the Artemis Platform runs all prede-

ﬁned test cases in the CI environment to validate the

solutions, leading to longer wait periods for students

to see whether their solution compiles or not. This

concern was raised four times, with students speciﬁ-

cally suggesting the implementation of a “Run-only”

button to allow code compilation without executing

tests. Additionally, there was a strong desire, men-

tioned ﬁve times, for increased editor pane size or the

introduction of a presentation mode to improve the

user interface. The improvement of the editor’s intel-

lisense, particularly for code completion and ﬁle ex-

plorer functionality, was highlighted twice. Further-

more, there were individual requests for the integra-

tion of a debugger and enhancements to the feedback

mechanism provided by the APAS.

Main Findings for RQ 1

Based on the survey responses we could derive the

following ranking for requirements of an online

IDE: Syntax Highlighting, Syntax Error Highlight-

ing, Compiler and Interpreter, Auto-completion

and Code Suggestions, Debugger and Breakpoints,

Brace Matching Highlighting, Key Shortcuts and a

Dedicated Shell Console .

The answers to the open ended questions indicated

a strong demand for improved compilation speed

in Artemis, with students proposing a “Run-only”

button to bypass test runs during compilation. Ad-

ditionally, improving the user interface via larger

editor panes or a presentation mode was a notable

request.

Requirements for an Online Integrated Development Environment for Automated Programming Assessment Systems

309

Figure 3: The detailed feelings of students about different IDE features.

4.2 State of the Art Solutions

The results of the systematic tool review are shown

in Table 2, showcasing a total of 15 analyzed online

IDEs, along with their features, supported program-

ming languages, used editors and whether they are

available open source or not.

From this analysis, it is evident that three promi-

nent editors dominate as a basis for online IDEs: (1)

Monaco (2) ACE and (3) CodeMirror. The Monaco

editor stands out as being the basis for more than 50%

of the tools. Furthermore, we found that the sup-

port for various programming languages is extensive

across the board, with most tools catering to a vast

majority of the selected languages. Tools utilizing the

Theia framework, which builds on the ’Monaco’ ed-

itor, tend to offer the most comprehensive IDE fea-

ture set compared to other editors. Notably, the fea-

tures, syntax error highlighting and auto-completion

are only found in tools using the Monaco editor, with

the exception of ’Online-ide’ that uses the ’ACE’ ed-

itor and provides auto-completion for a subset of the

languages.

Last but not least, Table 3 also shows that

Codespaces and Gitpod, fulﬁll all the requirements

we have deﬁned in this study. Proving that it is possi-

ble to create a complete online IDE.

Main Findings for RQ 2

Based on the systematic tool review we found that

the online IDEs Gitpod and Codespaces already

fulﬁll all of the deﬁned requirements. Proving that

it is possible to create a functional online IDE.

4.3 Online Ides’ Usability Within

APASs

For an online IDE to be usable within APASs it has

to be either open source or has to offer an API to

which to connect to. However, connecting an APAS

via an API to an external online IDE is not recom-

mended, because it introduces complexities in main-

taining consistent and secure data exchange, poten-

tial latency issues and can signiﬁcantly increase de-

pendency on the external service’s uptime and relia-

bility, which usually does not align with the APAS’s

operational requirements and standards. Therefore,

we mainly concentrated on whether the source code

of an online IDE is available as open source during

this review. Based on this we found that only three

tools were available as open source: (1) Artemis,

(2) Coderunner and (3) Exercism. Inspecting these,

we found that they offer important features like: (1)

Syntax highlighting, (2) Compiler / Interpreter, (3)

Brace matching highlighting, (4) Line numbers, (5)

Key shortcuts, (6) File navigation tools and (8) Line

numbers. However, important features are missing.

Among others, the following: (1) Debugger, (2) Auto-

completion / suggestions, (3) Syntax error highlight-

ing and (4) Dedicated shell console.

Main Findings for RQ 3

There are open source online IDEs like the built-

in editor of the APAS Artemis or Coderunner’s

and Exercism’s online IDEs that can be easily in-

tegrated within APASs. However, all only provide

partial feature support. In order to create an on-

line IDE that fulﬁlls all the requirements deﬁned in

this study it seems that improving Artemis’ editor

should be the recommended path to follow because

the editor is already integrated into an open source

automated assessment system.

CSEDU 2024 - 16th International Conference on Computer Supported Education

310

Table 2: Systematic Tools Review.

Tool name Provided features (ta-

ble 3)

Supported lan-

guages (table 3)

Base Editor

used

Open Source

Artemis 1,2,6,8 1,3,4,5 Monaco Yes

Codeboard 1,2,3,7,8,(9) 1,2,3,4,5 ACE No

Coderunner 1,3,7,8,(9) 1,2,3,4,6 ACE Yes

Codebyte-editor 1,2,3,7,8,9 1,2,3,4,6 Monaco No

Exercism 1,2,3,7,8,9 1,2,3,4,5,6 CodeMirror Yes

Sololearn 1,3,7,8,9 1,2,3,4,6 Monaco No

Replit 1,2,3,4,(5),6,7,8,9,10 1,2,3,4,5,6 Monaco No

Codesandbox 1,2,5,7,8,9,(10) 6 Monaco No

W3schools 1,3,(9) 1,2,3,4,6 CodeMirror No

Codewars 1,3,7,(9) 1,2,3,4,5,6 CodeMirror No

Online-ide 1,2,3,(5),7,8,9 1,2,3,4 ACE No

Onlinegdb 1,(2),3,4,7,8,(9) 1,2,3,4,6 ACE No

Codespaces 1,2,3,4,5,6,7,8,9,10 1,2,3,4,5,6 Theia

(Monaco)

Gitpod 1,2,3,4,5,6,7,8,9,10 1,2,3,4,5,6 Theia

(Monaco)

Codeanywhere 1,2,3,4,5,6,7,8,9,10 1,2,3,4,6 Theia

(Monaco)

(#) means partially supported

Table 3: Reviewed features and languages.

# Features # Programming languages

1 Syntax highlighting 1 C

2 File navigation tools 2 C++

3 Compiler / Interpreter 3 Java

4 Debugger 4 Python

5 Auto-completion / suggestions 5 Haskell

6 Syntax error highlighting 6 JavaScript

7 Brace matching highlighting

8 Line numbers

9 Key shortcuts

10 Dedicated shell console

5 DISCUSSION

In this study we found that ﬁrst-semester computer

science students prioritize online IDE features as fol-

lows: (1) Syntax Highlighting, (2) Syntax Error High-

lighting, (3) Compiler / Interpreter (Run Button), (4)

Auto-completion / Code Suggestions, (5) Debugger

/ Breakpoints, (6) Brace Matching Highlighting, (7)

Key Shortcuts, and (8) Dedicated Shell Console. The

preference for syntax highlighting and error high-

lighting aligns with the expectations for beginner pro-

grammers, who often have problems with understand-

ing the syntax and structure of code. These fea-

tures provide immediate visual cues that can help be-

ginners identify and correct errors more efﬁciently,

thereby enhancing their learning experience and re-

ducing frustration. The preference for a compiler or

interpreter with a simple ’Run’ button reﬂects the de-

sire for straightforward, immediate feedback on code

execution, which is particularly appealing for begin-

ners not yet comfortable with command-line inter-

faces or complex build processes. The appreciation

for auto-completion and code suggestions further in-

dicates the students’ need towards features that pro-

vide direct assistance and reduce the cognitive load

of remembering syntax and function names. By re-

ducing the cognitive load associated with understand-

ing syntax and executing programs, these features can

help mitigate early frustrations and enhance the over-

all learning experience. This is particularly relevant in

the context of APASs, where the immediate feedback

loop and the opportunity for iterative improvement

are key. The ability of an online IDE to provide clear,

instant feedback aligns with the key requirements of

Requirements for an Online Integrated Development Environment for Automated Programming Assessment Systems

311

APASs, facilitating a learning environment where stu-

dents can rapidly adapt and reﬁne their code based on

automated assessments.

The mid-tier ranking of debugging tools might

initially seem surprising, given their importance in

the software development process. However, this

can be understood in the context of novice program-

mers’ limited exposure to and understanding of sys-

tematic debugging strategies. As students’ program-

ming skills evolve, they are likely to recognize the

value of these tools in diagnosing and resolving issues

more effectively.

Beside this, feedback from open-ended questions

in the survey brought to light additional desired fea-

tures, such as a presentation mode and a large edi-

tor pane, which are particularly relevant in an educa-

tional context where students often present their work.

The need for enhanced compilation performance, as

speciﬁcally mentioned in relation to the Artemis plat-

form, points to a broader requirement for more efﬁ-

cient code execution within online IDEs. Students’

suggestions for a run button, to bypass full test exe-

cution for quicker compilation feedback, illustrates a

critical area for improvement in APASs.

The systematic review of online IDEs, empha-

sizing the robustness of editors like Monaco and

the comprehensive capabilities of Theia, provides a

roadmap for the development of APASs. However,

the challenge of balancing comprehensive feature sets

with resource efﬁciency remains vital, especially in

educational settings where scalability is key. Addi-

tionally, the systematic tool review allows educators

and alike to quickly decide on which tool to use de-

pending on the programming language and feature

support needed.

In summary, this study not only identiﬁes the on-

line IDE features most valued by beginner program-

mers but also highlights the critical role these features

play in supporting the objectives of APAS by helping

to create an environment that reduces initial barriers

to learning programming.

6 LIMITATIONS

This study largely leaned on a survey and a system-

atic tool analysis. Regarding the survey, it might be

the case that we overlooked important online IDE fea-

tures. However, this threat to validity should be min-

imized by allowing the students to add missing fea-

tures via open-ended feedback.

In addition, the over representation of ﬁrst-year

students to judge the most important features for an

IDE presents a potential limitation. These students,

due to their limited experience, might prioritize fea-

tures that offer immediate convenience and are easy

to understand over those that could provide signiﬁ-

cant long-term beneﬁts in learning and coding proﬁ-

ciency. For instance, syntax highlighting, being an in-

tuitive feature, is easily appreciated by beginners for

its ability to improve code readability and reduce er-

rors. This direct and immediate beneﬁt might lead

students to rate such features as highly important. On

the other hand, the utility of a debugging tool, which

is more complex and requires a deeper understanding

of programming concepts to use effectively, might not

be as readily apparent to them. Even-though, over

time, as students’ programming skills improve, the

ability to systematically diagnose and ﬁx errors using

a debugger signiﬁcantly enhances their productivity

and learning outcomes. Future work could involve

conducting a similar study with programming edu-

cators and more experienced programmers to gain a

broader perspective on the importance of various IDE

features. However, starting with students as the pri-

mary respondents was a deliberate choice, consider-

ing they are the main users of online IDEs in APASs

and as a results understanding their preferences and

immediate needs can provide valuable insights into

designing more accessible and user-friendly program-

ming environments.

Last but not least, in the systematic tool analysis,

we might have missed important online IDEs. How-

ever, the search string was developed with input from

two subject matter experts, aiming to encompass most

signiﬁcant online IDEs. Despite this, the review pro-

cess could still be affected by subjective interpreta-

tions, even-though we used a systematic data extrac-

tion methodology to mitigate such biases.

7 CONCLUSION

This research aimed to identify and evaluate the

essential features required for an effective online IDE

in the context of APASs. Through a comprehensive

survey conducted among APASs users, primarily

composed of ﬁrst-semester computer science stu-

dents, we gained valuable insights into the features

that are most valued in an online IDE. The ranking

of these features, led by Syntax Highlighting and

Syntax Error Highlighting, reﬂects the speciﬁc needs

and preferences of beginners in programming. This

ranking, complemented by the additional require-

ments identiﬁed through open-ended questions,

provides a clear direction for future development and

enhancement of online IDEs in educational contexts.

CSEDU 2024 - 16th International Conference on Computer Supported Education

312

The systematic review of existing online IDEs un-

derscored the prominence of the Monaco editor and

revealed the comprehensive feature set offered by the

Theia IDE. However, the resource-intensive nature of

Theia points to a need for further investigation into its

feasibility for widespread use. This study’s ﬁndings

also highlight a gap in the availability of open-source

online IDEs with only Artemis, Coderunner and Ex-

ercism, offering partial feature support.

Future research should on the one hand expand

the survey’s population to also include experienced

programmers and educators and on the other hand

explore the resource demands of comprehensive on-

line IDEs, like Theia, and investigate the feasibility

of developing open-source solutions that fully meet

the identiﬁed requirements.

In conclusion, this study contributes to the under-

standing of what constitutes an effective online IDE

for APASs. By prioritizing features according to user

preferences and exploring the current state of online

IDEs, we lay the basis for future advancements in this

area, ultimately aiming to improve the learning expe-

rience of programming students and the efﬁciency of

APASs.

ACKNOWLEDGMENTS

The CodeAbility Austria project has been funded by

the Austrian Federal Ministry of Education, Science

and Research (BMBWF).

REFERENCES

Barra, E., Lopez-Pernas, S., Alonso, A., Sanchez-Rada,

J. F., Gordillo, A., and Quemada, J. (2020). Auto-

mated assessment in programming courses: A case

study during the COVID-19 era. Sustainability,

12(18):7451.

Chen, H.-M., Nguyen, B.-A., Yan, Y.-X., and Dow, C.-R.

(2020). Analysis of learning behavior in an automated

programming assessment environment: A code qual-

ity perspective. IEEE Access, 8:167341–167354.

Hausladen, J., Pohn, B., and Horauer, M. (2014). A cloud-

based integrated development environment for em-

bedded systems. In Proc. of the 2014 IEEE/ASME

10th Int’l Conference on Mechatronic and Embedded

Systems and Applications, pages 1–5. IEEE.

Krusche, S. and Seitz, A. (2018). Artemis: An automatic

assessment management system for interactive learn-

ing. In Proc. of the 49th ACM Technical Symposium

on Computer Science Education, pages 284–289.

Kusumaningtyas, K., Nugroho, E. D., and Priadana, A.

(2020). Online integrated development environment

(ide) in supporting computer programming learning

process during COVID-19 pandemic: A comparative

analysis. International Journal on Informatics for De-

velopment, 9(2):66–71.

Mekterovi

c, I., Brki

c, L., and Horvat, M. (2023). Scaling

automated programming assessment systems. Elec-

tronics, 12(4):942.

Satav, S. K., Satpathy, S., and Satao, K. (2011). A compar-

ative study and critical analysis of various integrated

development environments of C, C++, and Java lan-

guages for optimum development. Universal Journal

of Applied Computer Science and Technology, 1:9–15.

Sinanaj, L., Ajdari, J., Hamiti, M., and Zenuni, X. (2022). A

comparison between online compilers: A case study.

In Proc. of the 11th Mediterranean Conference on

Embedded Computing, pages 1–6.

Tran, H. T., Dang, H. H., Do, K. N., Tran, T. D., and

Nguyen, V. (2013). An interactive web-based IDE to-

wards teaching and learning in programming courses.

In Proc. of 2013 IEEE Int’l Conference on Teach-

ing, Assessment and Learning for Engineering, pages

439–444. IEEE.

Requirements for an Online Integrated Development Environment for Automated Programming Assessment Systems

313