Let’s Choose STEM: An Overview on Study Program Guiding Online

Self-Assessments and Future Directions

Vivien Landgrebe

1 a

, Sarah Aragon-Hahner

1,2 b

and Sven Strickroth

1 c

LMU Munich, Munich, Germany

TU Bergakademie Freiberg, Freiberg, Germany

ﬁ

Keywords:

Online Self-Assessments, Study Program Choice, Decision-Making, Online Career Guidance,

Career Decision-Support Systems.

Abstract:

Online Self-Assessments (OSAs) are common tools for university orientation. They assist high school students

in linking their vocational interests with matching ﬁelds of study or checking their suitability for a certain

study program by considering personal interests, skills, and vocational expectations. We are the ﬁrst to provide

a review and classiﬁcation of N=12 OSAs currently offered in Germany in the area of Science, Technology,

Engineering, and Mathematics (STEM). Our results show that OSAs differ considerably in their content

and duration. While the analyzed tools offer time and location independence, they still lack ﬂexibility and

adaptability with respect to the individual user. We discuss these ﬁndings and provide future directions for

research as well as for OSA suppliers. The approaches we identiﬁed as promising offer immersive experiences,

adaptive content, and empower students to make well-reﬂected decisions.

1 MOTIVATION

Study program choice can be a complex task for high

school graduates given the plethora of options. The

website https://www.studycheck.de (last accessed

2024–04–11) lists about 21,000 degree programs in

Germany. Even after determining a broad vocational

direction, prospective students have difﬁculties eval-

uating the concrete options and are uncertain about

the potential outcomes of their decision (Germeijs and

De Boeck, 2003). To assist students in making better

career decisions, many universities have introduced

Online Self-Assessments (OSA) for study orientation.

These tools rely on the students’ individual evalua-

tion of their interests, skills, and personality traits.

Based on these characteristics, OSAs calculate match-

ing ﬁelds of study or assess the user’s aptitude for a

certain degree program. Such tests can help students

understand their strengths and interests and link them

to relevant career objectives (Hasenberg and Schmidt-

Atzert, 2014). By providing information on different

degree programs and aligning expectations, OSAs can

be a resource-efﬁcient complement to academic advis-

https://orcid.org/0009-0002-1878-7182

https://orcid.org/0000-0001-7587-080X

https://orcid.org/0000-0002-9647-300X

ing, inﬂuence the external perception of the university,

and manage access to degree programs (Stoll, 2019).

Therefore, both prospective students and providers can

beneﬁt from well-designed OSAs. Although many

such tests exist, there is only little literature on the

creation and validation of OSAs (cf. (Stoll and Weiss,

2022)). Accordingly, OSAs vary widely and there is

no overview of the current state of the art.

We are the ﬁrst to provide a systematic review on

German OSAs. To give a diversiﬁed overview, we look

at a cross-section of the Science, Technology, Engi-

neering, and Mathematics (STEM) ﬁeld. We therefore

classify a set of N=12 OSAs, reveal advantages and

disadvantages of state-of-the-art systems and discuss

future directions for research as well as for providers.

Our results show that, on the one hand, OSAs convince

with their time and location independence as they of-

fer ubiquitous access through the internet. In contrast

to personal counseling, they can preserve the user’s

anonymity while answering personal questions. On

the other hand, there is a huge inconsistency regarding

their duration and content, which can make it challeng-

ing for students to trust the test results. To overcome

these drawbacks, future research should develop gen-

eral guidelines for OSAs, leverage adaptive content,

and explore the use of further technologies, such as

virtual reality.

Landgrebe, V., Aragon-Hahner, S. and Strickroth, S.

Let’s Choose STEM: An Overview on Study Program Guiding Online Self-Assessments and Future Directions.

DOI: 10.5220/0012632200003693

Paper published under CC license (CC BY-NC-ND 4.0)

In Proceedings of the 16th International Conference on Computer Supported Education (CSEDU 2024) - Volume 1, pages 135-145

ISBN: 978-989-758-697-2; ISSN: 2184-5026

135

In the following, we ﬁrst present the theoretical

background of career decision making, related chal-

lenges, and how these are addressed by online career

guidance systems. We then explain the methodology

of our systematic review, present the resulting clas-

siﬁcation, and discuss the beneﬁts and drawbacks of

current OSAs based on their features. Finally, we sum-

marize our ﬁndings and present our vision for future

OSAs.

2 BACKGROUND

Career choice is one of the most important yet complex

life decisions for young adults. Given the multitude

of inﬂuencing factors, the process can be particularly

challenging for high school graduates in comparison to

experienced professionals (Galliott, 2017). In a recent

literature review, Gati and Kulcs

ar elaborate on the

process of career decision-making and discuss related

issues and challenges of the 21st century (Gati and

Kulcs

ar, 2021). They present the career choice process

as a three-stage model comprising (1) the prescreening

phase, (2) the in-depth exploration phase, and (3) the

choice phase. First, people broadly explore the envi-

ronment of career opportunities and identify promising

alternatives according to their interests. Second, they

narrow down the different options ﬁtting their person-

ality and skills. Finally, they choose the most suitable

alternative. The OSAs presented in this work are either

designed to be used in the prescreening phase (OSAs

for general study program orientation) or the in-depth

exploration phase (OSAs on certain degree programs).

During these different stages, various problems can oc-

cur, e. g., lack of readiness, lack of information (about

oneself or about professions), or internal and external

conﬂicts (Gati and Kulcs

ar, 2021).

The idea of using technology to support this chal-

lenging process dates back to the late 1960s, when

the ﬁrst computer-assisted career guidance systems

(CACGS) were implemented to complement personal

career counseling (Harris-Bowlsbey, 2013). While

traditional CACGS were mostly digital versions of

paper-based assessments, online tools offer a lot more

opportunities: they have the advantages of reducing

costs, extending access to job information, and offer-

ing new possibilities for interactive content (Vigurs

et al., 2017), such as automatic and immediate feed-

back (Kleiman and Gati, 2004). While paper-based

assessments or print media can only provide static

career information, OSAs can potentially present up-

to-date information and customize content for each

individual user. German universities and government

institutions, such as the German Federal Employment

Agency

, draw on these beneﬁts when developing and

providing OSAs. However, scientiﬁc research on this

widely used instrument is limited. Gati et al. empha-

size the need for career counselors and researchers to

work together in the development of OSAs, to ensure

their quality and usefulness (Gati and Saka, 2001; Gati

and Asulin-Peretz, 2011).

Galliott performed a user study to investigate

which functionalities OSAs need to have in order

to eliminate students’ career uncertainties (Galliott,

2017). The study showed that students are often not

aware of the available services for career orientation.

The authors emphasize that important information on

career orientation must always be kept up-to-date. This

applies not only to the content of OSAs and general ca-

reers websites, but also to the sources in which they are

advertised. Additionally, the information should ap-

peal to different target groups: Having in mind that stu-

dents are often inexperienced in making far-reaching

life decisions, Galliott recommends that OSAs should

not only be promoted to students, but also to their par-

ents, teachers, as well as career counseling institutions,

in order for them to assist the students in the career

choice process (Galliott, 2017). Related work also

highlights the need for considering culture in career

guidance (Akosah-Twumasi et al., 2018).

Well-known research from psychology suggests

that career guidance systems need to support students

in ﬁnding their professional interests to make a satisfy-

ing decision on a course of study (e. g., (Holland, 1997;

Tracey, 2010)). An established taxonomy that links

personal interests with occupational preferences is the

RIASEC model (Holland, 1997). It is also known as

Holland’s hexagon model, as it consists of six areas

of interest, as shown in Figure 1. The model com-

prises (1) the realistic type referring to a person with

technical and practical interests, (2) the investigative

type describing theory-led and observational interests,

(3) the artistic type representing creative and cultural

interests, (4) the social type referring to helping and

caring individuals, (5) the enterprising type related to

leadership and organization, and (6) the conventional

type including administrative interests. Holland sug-

gests to determine the level of interest for these six

dimensions to ﬁnd matching vocational ﬁelds. The

result can then for example be compared with the qual-

ities that a degree program requires of its students. Our

systematic review (see Section 4) revealed that some

of the German OSAs (EI, SIT) are based on Holland’s

RIASEC taxonomy.

https://www.arbeitsagentur.de, last acc. 2024–04–11

https://www.researchgate.net/figure/Hollands-hexag

on-model-for-vocational-interests fig4 224971173, last

accessed 2024–04–11

CSEDU 2024 - 16th International Conference on Computer Supported Education

136

Figure 1: RIASEC model by Holland

Other works further emphasize the importance of

personal interests in study orientation. For example,

Tracey observed that the greater a student’s interest in a

subject, the better they perform and the more satisﬁed

they are with their decision (Tracey, 2010). Another

study found that students, who choose their degree

program according to their interests, are more likely

to widen their knowledge in the ﬁeld of the program

and thus improve their professional identity (Smitina,

2010). In this context, a pronounced professional iden-

tity means that persons know their vocational strengths,

why they chose previous career pathways, and what

they want to achieve in the future. Those who have a

low professional identity tend to randomly decide on a

study program and thus have to face more difﬁculties

in their further career (Smitina, 2010). The presented

publications are just a small selection of well-known

examples of the theoretical basis of the career choice

process. Developers of OSAs can draw on a variety of

proven concepts and theoretical models on vocational

decision-making. However, the way in which these

tests are developed in practice remains unclear. To

the best of our knowledge, there is no overview of the

current state of the art. We ﬁll this research gap by

providing a systematic review of up-to-date German

OSAs in the STEM ﬁeld and derive recommendations

for research and providers.

3 METHODOLOGY

This section describes the selection process and classi-

ﬁcation categories we found and used for our analysis.

3.1 Selection Process

We started the research for relevant online tools with

two overview websites that independently list sev-

eral OSAs developed by universities in the German-

speaking area: OSA Portal

and “Komm, mach

MINT”

. The ﬁrst website contains OSAs for various

degree programs as well as multidisciplinary OSAs.

Here we selected three tools provided by different uni-

versities: BTU, GAU, and LMU. The second website

gives an overview of OSAs for the STEM area, where

we chose six more tools: EAH, HC, HTW, SIT, PUM,

and UF. We selected common courses of study from

the STEM area like mechanical engineering, biology,

or computer science. For each speciﬁc study program,

we chose only one OSA to examine a cross-section of

this area. Moreover, every assessment is provided by

a different university to draw better comparisons.

We then visited the websites of various universities

to ﬁnd out if they offered aptitude tests for speciﬁc

degree programs or a general test for all majors. In

this process, we discovered ST – a complex OSA, that

does not only take vocational interests into account,

but also creates a personality proﬁle of the user accord-

ing to their input. This sets this tool apart from others

that deal with general vocational interests, which is

why we considered it in our work. To increase the

number of such general interest tests, we checked the

websites of several German institutions that deal with

career orientation like the German Federal Employ-

ment Agency. As a result, we identiﬁed ﬁve additional

instruments relevant to our evaluation: CU, EI, HCU,

MBW, and UN. Our ﬁrst preselection included 15

OSAs (seven general tools and eight tools on certain

study programs), as shown in Table 1. We tested all

of the preselected tools by answering the questions

honestly and from our point of view. Our goal was to

gain an impression of the content, the duration and the

results. Afterwards, we removed three general OSAs

(HCU, MBW, UN) from our list, as they appeared to be

similar or nearly identical to other ones and including

them would not add to the classiﬁcation. Our search

for relevant OSAs ﬁnished when we did not ﬁnd any

more tools that notably differed from the already tested

ones. Our ﬁnal data set consists of twelve OSAs: four

general and eight speciﬁc tools (cf. Table 3).

To get more insights into the conception of the tests,

we looked up the developers’ contact information, if

available, and reached out to them via email. We

asked them who was involved in the conception, how

the content was produced and what was their targeted

duration of the OSA. We also wanted to know if the

OSA was originally planned as an online tool or if an

ofﬂine version was adapted, on which platforms it was

available, which technologies were used and if the tool

was scientiﬁcally evaluated.

http://www.osa-portal.de, last accessed 2024–04–11

https://www.komm-mach-mint.de/schuelerinnen/teste

-dich-selbst/self-assessments, last accessed 2024–04–11

Let’s Choose STEM: An Overview on Study Program Guiding Online Self-Assessments and Future Directions

137

Table 1: URLs and providers of the 15 OSAs considered for the classiﬁcation. The OSAs that are not included in the

classiﬁcation are shown in parentheses. All websites last accessed 2023–12–01.

ID Provider – Description URL

BTU

Brandenburg University of Technology –

Online self-assessment for the mathemat-

ics degree program

https://www.b-tu.de/elearning/college/mod/quiz/vi

ew.php?id=250

Check-U by the German Federal Employ-

ment Agency – Exploration tool for appren-

ticeships and studies

https://www.arbeitsagentur.de/bildung/welche-ausbi

ldung-welches-studium-passt

EAH

University of Applied Sciences Jena – On-

line self-assessment for the mechanical en-

gineering degree program

https://selfassessment.eah-jena.de/osa.php?id=13

EI Einstieg GmbH – Vocational interest test https://interessencheck.einstieg.com

GAU

University of G

ottingen – Virtual study ori-

entation of the Faculty of Chemistry

https://www.studienorientierung.uni-goettingen.de/n

avigator/chemie/index.php?pid=1000

Coburg University of Applied Sciences and

Arts – Orientation test for study programs

in the STEM area

https://www.studiengangstest.de/test/index.php/392

996

(HCU)

HafenCity University Hamburg – Voca-

tional interest test

https://hcu-studienorientierung.cyquest.net/navigator

/interessentest/

HTW

Berlin University of Applied Sciences – On-

line self-assessment of study programs in

the power engineering & automation area

https://studienwahl.htw-berlin.de/index.php?t=TGQ

LMU

Ludwig-Maximilians-Universit

at M

unchen

– Online self-assessment for the physics de-

gree program

https://www.self-assessment.lmu.de/course/view.php

?id=10

(MBW)

Ministry of Science, Research and

Arts Baden-W

urttemberg – Online self-

assessment for study orientation

https://www.was-studiere-ich.de

PUM

University of Marburg – Online self-

assessment of the Bachelor’s degree pro-

gram in Biology

https://survey.online.uni-marburg.de/qss/index.php/

235912

SIT Hochschulkompass – Study interest test

https://www.hochschulkompass.de/studium-interesse

ntest.html

University of T

ubingen – General study pro-

gram choice test

https://www.studienwahltest.uni-tuebingen.de

University of Freiburg – Online study se-

lection assistant for the computer science

degree program

https://www.osa.uni-freiburg.de/informatik/

(UN)

Uniturm.de by Pharetis GmbH – Online

study program choice test

https://www.uniturm.de/studienwahl/studienwahltest

-studiumsfinder

3.2 Classiﬁcation Categories

During the testing phase, we identiﬁed eight categories

for our classiﬁcation of the OSAs (cf. Table 2). These

are all aspects in which the selected tools differ from

each other or which are relevant to analyze the beneﬁts

and challenges for the development of OSAs.

4 CLASSIFICATION

Table 3 gives an overview of our classiﬁcation. The

eight categories are presented in the columns and the

tested OSAs are listed in the rows. If we were unable

to identify data for the particular category, we indi-

cated N/A (not available). The classiﬁcation reveals

that the tested tools have huge differences in their com-

plexity and content. The number of questions ranges

from nine to 213, while the content includes knowl-

CSEDU 2024 - 16th International Conference on Computer Supported Education

138

Table 2: Speciﬁcation of categories for the classiﬁcation of OSAs.

Category Meaning

Assessment Goal

Overall goal of the OSA. This includes the differentiation between a general

interest test and an aptitude test for a speciﬁc degree program.

Starting Point (SP)

There are three situations students can be in when doing an OSA: (1) They have

no idea what to study, (2) they have a general interest in STEM, or (3) they are

interested in a concrete study program.

Content

Information presented to the user, e. g., standardized questionnaires, contents of

a course of study or information about studying in general.

Duration (Durat.) Planned duration of the OSA (as speciﬁed by the developers).

No. & Type of Questions

Number and type of questions or tasks as an indicator for the OSA’s complexity

and design.

Device Platforms targeted by the OSA: desktop (D), mobile (M), or app (A).

Platform If detectable: technologies or platforms the OSA is based on.

Results The type of results the OSA provides after its completion.

edge questions, standardized questionnaires based on

scientiﬁc models such as the RIASEC model (Holland,

1997), as well as further questions on personality traits,

social competencies, etc. Furthermore, it shows that

each OSA is available in a desktop and a mobile ver-

sion, while its content remains static. Finally, we found

that the used technologies are hard to identify, and the

results solely include study program recommendations

according to the user’s interests or skills, but provide

no motivation for reﬂecting on the system’s output.

Out of twelve contacted developers, we received six

answers within one week (EAH, GAU, HC, LMU, ST,

and UF). Roles involved in the development ranged

from deans of studies, scientiﬁc employees, students,

as well as a Ph.D. student, who designed the tool as

part of their dissertation. The content was largely cre-

ated based on topics of the study program, with the

goal for it to be interesting for students with all kinds

of knowledge states, and without the inclusion of scien-

tiﬁc methods. The answers revealed that all six OSAs

were initially developed as an online tool with a strong

focus on the desktop version. Moreover, each of them

is intended to last between one and two hours. Con-

cerning the used technologies, most developers have

resorted to certain providers such as the open-source

LMS Moodle

or the commercial tool Cyquest

. None

of the respective OSAs have been subjected to scien-

tiﬁc evaluation.

https://moodle.org, last accessed 2024–04–11

https://www.cyquest.net, last accessed 2024–04–11

5 DISCUSSION

In the following, we discuss the results of our classiﬁ-

cation, derive advantages and disadvantages of current

OSAs, and make recommendations for developments.

5.1 Evaluation of the Tested Tools

Through our classiﬁcation, we were able to get a broad

overview of the spectrum of currently used OSAs and

to detect relevant differences between them. We will

now discuss the individual features of the tested tools,

to be able to better assess their potential to support

study program choices.

5.1.1 Content and Complexity

The beneﬁt of an OSA highly depends on the content

it covers. Only two of the tested OSAs (EI, SIT) are

based on scientiﬁc methods – more precisely Holland’s

RIASEC model (Holland, 1997). However, they only

cover the user’s interests and calculate matching ﬁelds

of study. It is notable that these tests are comparably

short: We were able to complete both of them in under

ten minutes. Related work suggests that such a short

test cannot reﬂect the complexity of career decision-

making (Gati and Kulcs

ar, 2021). Showing interest

in a certain area does not automatically imply a good

suitability for the course of studies. More information

about the user needs to be collected, e. g., on their

skills or on personality traits.

Let’s Choose STEM: An Overview on Study Program Guiding Online Self-Assessments and Future Directions

139

Table 3: Classiﬁcation of the twelve OSAs regarding the categories described in Table 2.

OSA Assessment Goal SP Content Durat. No. & Type of

Questions

Device Platform Results

BTU Suitability

(Mathematics)

3 Knowledge questions N/A 21: Multiple

choice (MC), open

text, drag & drop

D, M,

Moodle Submitted answers

CU Matching study

programs, skills

vocational interests,

1 Questions on

vocational interests,

skills, social

competency

125 min 213: Scale (0 -

100), either/or

D, M N/A Matching study

programs &

apprenticeships, areas

of interest,

submitted answers

EAH Suitability (Mech.

Engineering)

3 Knowledge questions,

studying in general

N/A 40: True/false,

MC, open text

D, M alpha-test

GmbH

Submitted answers

EI Vocational

interests

1 RIASEC model

(Holland, 1997)

15 min 60: Scale (0 - 100) D, M N/A Matching study

programs, jobs &

apprenticeships

GAU Suitability

(Chemistry)

3 Knowledge questions,

studying in general

45 min 23: True/false, MC D, M CYQUEST

GmbH

Submitted answers

HC Suitability (STEM) 2 Knowledge questions,

studying in general

60 min 150: Scale (Likert) D, M N/A Submitted answers,

skill evaluation

HTW Suitability (Power

Engineering)

2 Knowledge questions N/A 9: MC D, M alpha-test

GmbH

Submitted answers

LMU Suitability (Physics) 3 Knowledge questions 120 min 53: Scale (Likert),

yes/no

D, M Moodle Submitted answers

PUM Suitability (Biology) 3 Knowledge questions,

studying in general

N/A 117: True/false,

D, M N/A Submitted answers,

skill evaluation

SIT Vocational interests 1 RIASEC model

(Holland, 1997)

15 min 82: Scale (0 - 100) D, M TYPO3

CMS

Areas of interest,

matching study

programs

ST Vocational interests,

personality profile

1 Questions on

vocational interests &

personality

145 min 115: Scale (Likert) D, M N/A Personality & interest

profile, matching

study programs

UF Suitability

(Computer Science)

3 Knowledge questions,

studying in general

60 min 51: Yes/no, MC D, M

Wordpress,

H5P,

JavaScript

Submitted answers

Furthermore, each of the tested OSAs is static, i. e.,

its content does not adapt to the user’s input. In this

way, the instruments cannot respond more deeply to

the user’s particular interests without increasing the

number of questions and the time required for the OSA.

This means that two potentials of digital technologies –

ﬂexible use and adaptability – are overlooked in state-

of-the-art OSAs.

5.1.2 Platform

All of the tested OSAs are accessible in a desktop and

mobile version. One of them is even available in an app

(BTU). However, our survey of the OSA developers

showed, that their main focus during the development

lied on the desktop version which might also be visible

in the targeted duration for the OSAs. Here we see

the need to better adapt career guidance systems to

the requirements of their speciﬁc user group, namely

young adults. Especially in this target group, the use

of mobile devices has now overtaken stationary desk-

top PCs.

This shift creates new challenges for the

design of study orientation tools. In their recent work,

Aragon-Hahner et al. emphasize that smartphone use

is characterized by a small display size and a shorter

attention span (Aragon-Hahner et al., 2023b; Choi and

Lee, 2012). Therefore, they examine micro-content for

mobile OSAs. A ﬁrst evaluation delivered promising

results, but their system is still a research prototype in

the development phase.

5.1.3 Results

The result types of OSAs mostly depend on the starting

point of the assessment. While tests with a SP level 1

usually suggest matching study programs, higher-level

tests (SP 2 & 3) most often only mirror back the user’s

https://www.postbank.de/themenwelten/innovationen/

digitalstudie-2022-mobile-internetnutzung-entwickelt-sic

h-rasant.html, last accessed 2024–04–11

CSEDU 2024 - 16th International Conference on Computer Supported Education

140

answers. In the rare cases where an assessment is pro-

vided, it is rather general. HC, for example, yields

results that show in which areas users have the most

skills, but not for which courses they are particularly

suited. In all cases, there is no guidance on how to deal

with the test results. On the one hand, users should

critically question the given career recommendations

and verify the validity of the test in order to avoid

overtrust in the results. On the other hand, it can be

difﬁcult for users to evaluate the results if a recommen-

dation is completely omitted. The decision on a study

program can be very complex and many personal and

environmental factors have to be regarded to make a

useful recommendation (Galliott, 2017). Therefore,

developers have to be careful when formulating the

test results. Users should be encouraged to reﬂect on

the system’s output in order to be able to ﬁnally make

a “self-assessment” in the literal sense. Overall, 75 %

of the general OSAs calculated study programs ac-

cording to our expectations, although this estimate is

highly subjective and might not apply to all users. The

only exception (ST) lacked questions about computer

science and was therefore not able to suggest a ﬁtting

course of study for our test user.

5.2 Advantages of Existing OSAs

Given that all of the tested OSAs are available in a

desktop and a mobile version, a clear advantage is

that they are time- and location-independent. OSAs

scale well for a large user base, since their execution

does not require the presence of a career counselor or

a teacher. Moreover, as there is no interaction with a

real person, the user might feel safer and potentially

respond to the questions more honestly and with less

pressure (Cameron et al., 2017). Another advantage

is that OSAs can be repeated as often as desired to

reassess one’s competencies after looking up more

information on the ﬁeld of studies. Thus, they can sup-

port well-informed decisions about whether to apply

for a speciﬁc degree program.

5.3 Disadvantages of Existing OSAs

The development of an OSA is time-consuming and la-

borious. Developers must not only have deep insights

into the study program(s), but should also be familiar

with questionnaire design and have an overview of psy-

chological research on personality traits and decision

making. Our contact with universities revealed that the

authors of course-speciﬁc OSAs have different areas

of expertise, which can lead to a large variance in the

quality of the tests. This raises ethical concerns, that

will be discussed in the next section. We believe, that

it is of utmost importance for the test results to be reli-

able, correct and comprehensible to avoid confusion

and career indecision.

Most of the OSAs provided results subjectively

matching our test user’s interests, regardless of their

duration. We still suspect a bias in the results, since

the given answer options sometimes seemed to lead

the user in a certain direction by making the “correct”

answer obvious. At the same time, very general tests

bear the risk of distorting the results by oversimplify-

ing the career choice process. For example, an OSA

might ask “Could you imagine working in a biology

laboratory?” with this being the only question related

to biology. People who would rather like to work out-

doors might unintentionally score low on this question,

even if biology would in fact be a good ﬁt for them.

Overall, our analysis shows that a careful selec-

tion of questions and content is crucial for avoiding

biased results. To cover the full spectrum of study

orientation, another important aspect of OSAs is that

they are impartial and independent. While most OSAs

are offered by state institutions, there are also some

commercial providers (e. g., EI), who only include

companies and universities with a registration or co-

operation in their results. Not showing all relevant

options is a clear shortcoming of commercial tools, as

this deprives students of potentially helpful systems,

increasing the problem of uninformedness (Galliott,

2017). Finally, avoiding social interaction also comes

with the downside of not having a professional con-

tact person at hand. This means that students cannot

discuss the results directly with a consultant, which

would be particularly important in case of uncertainty.

We found that some tools do not even list a contact

person (EAH, EI).

5.4 Ethical Aspects

Making a career decision is one of the most impor-

tant choices in one’s life (Gati and Kulcs

ar, 2021). In

fact, the decision on an unsuitable study program can

have far-reaching psychological consequences (Gati

and Kulcs

ar, 2021). Discontinuing a course of study

might lead to uncertainty on future career pathways

and fear of making another wrong decision. Moreover,

starting a course of study includes investing ﬁnancial

resources and implies social consequences (Gati and

Kulcs

ar, 2021). Therefore, OSAs need to fulﬁll gen-

eral ethical principles, for example by ensuring accu-

racy and accountability. Our classiﬁcation shows that

common OSAs often provide recommendations for

suitable courses of study. However, since the majority

do not apply scientiﬁc methods, these calculations are

usually based on criteria subjectively determined by

Let’s Choose STEM: An Overview on Study Program Guiding Online Self-Assessments and Future Directions

141

the providers. Also, the calculations are not transpar-

ent. Given the mostly practice-oriented expertise of

the developers, there is a risk that the calculated rec-

ommendations are not an ideal ﬁt for the user and thus

might violate the previously described ethical prin-

ciples. Thus we recommend for designers of OSAs

to involve experts from different ﬁelds in the devel-

opment of the tools, and carefully formulate the test

results. To prevent users from blindly following a

recommendation, research by Aragon-Hahner et al.

suggests to incorporate self-reﬂection in decision sup-

port systems for career choice – not only on the user’s

personal skills and interests (Aragon-Hahner et al.,

2023a), but also on the recommendations given by a

study orientation test (Aragon-Hahner et al., 2023b).

In addition, we want to scrutinize whether it is

always an advantage to let someone stay in their com-

fort zone. As argued before, an advantage of OSAs is

that they do not require personal contact and therefore

a student might feel more comfortable during their

career assessment (Cameron et al., 2017). However,

becoming conﬁdent in personal conversations can be

an essential skill for students, especially for future job

interviews. Yet, if a student is not ready to leave their

comfort zone or has other reasons for not taking part in

personal career counseling, e. g., health or accessibility,

OSAs are a valuable alternative. In summary, we sug-

gest that OSAs should not replace humans completely,

but rather be used in combination with personal career

counseling (Gati and Asulin-Peretz, 2011).

5.5 Possible Improvements for OSAs

Existing inconsistencies in the duration and content of

tests could be resolved through general guidelines that

establish criteria that make an OSA efﬁcient and help-

ful to users. Such guidelines could include scientiﬁc

methods, e. g., the RIASEC model (Holland, 1997),

on which OSAs need to be based, in order to make

them reliable (Gati and Asulin-Peretz, 2011; Gati and

Saka, 2001). In addition, there could be further content

speciﬁcations, e. g., that an OSA must be composed of

various areas relevant to the career choice process. CU

is an illustrative example of a comprehensive study ori-

entation test. This tool includes questions about social

competencies, skills as well as vocational preferences.

By establishing best practices to help students reﬂect

on their career aspirations, the wide variation in the

complexity of OSAs could be addressed.

The users of OSAs should be able to access their re-

sults at a later date. This can be important, e. g., when

they have an appointment for further career guidance

and want to discuss the test results with their career

counselor (Gati and Asulin-Peretz, 2011). At some

universities, taking the OSA for a speciﬁc course of

study is a requirement for enrolling in this program.

In this case, the OSA needs to provide an option to

print or save the test result, for example in the form

of a certiﬁcate of attendance. Therefore, users often

need to register on the platform. One the one hand,

this is an advantage as it allows them to know exactly

where they can ﬁnd their test results and – if necessary

– ﬁnd further information such as contact details. This

is already implemented in some of the existing OSAs

(BTU, CU, LMU, SIT, ST). On the other hand, hav-

ing to create a user account constitutes an additional

barrier for career guidance and hinders serendipitous

participation in the test. Developers should therefore

consider whether the lowest possible entry threshold

or a closed system is more suitable for their purpose.

Overall, the development of OSAs should be more

systematic and scientiﬁc. A contact person should

always be listed on the website. In the optimal case, an

OSA is developed by or with the help of experienced

career counselors. The contact information of these

experts should be provided to the users to reliably

answer arising questions about the tool and its results.

In the process, a personal conversation can eliminate

a still-existing indecisiveness and help with further

career orientation (Galliott, 2017).

5.6 Further Technologies for Career

Guidance

Our classiﬁcation focuses on the current state of the art

of OSAs. Most of the systems presented use a “classic”

question-answer format, are text-heavy, and offer little

ﬂexibility due to their static structure. Our previous

considerations show that an effective OSA requires

a certain scope. As a result, providers are likely to

resort to the obvious option of a desktop-based tool.

In the process, they lose sight of the requirements

of their target group, as young people are no longer

used to sitting in front of a computer for hours ﬁlling

out questionnaires. Given the complexity of career

choice, it seems hard to develop a one-ﬁts-all solution.

Designers could overcome this problem by making

OSAs adaptive, so that they can respond in more detail

to the user’s input, ask more targeted questions and

provide more relevant content.

Modern career guidance systems explore the possi-

bilities of mobile apps (Aragon Bartsch et al., 2022)

or immersive technologies (Demareva et al., 2020)

and provide relevant information in an interactive

format. Aragon Bartsch et al. give job insights

by sending users mobile chat messages about a pro-

fessional’s daily work routine in a temporal con-

text (Aragon Bartsch et al., 2022). They found that

CSEDU 2024 - 16th International Conference on Computer Supported Education

142

this method provides more realistic and personal im-

pressions than traditional means for career orientation.

A different example is the project “Dein Erster Tag”

by Studio2B GmbH

: They give students the possibil-

ity to immersively experience 360° videos of different

working environments in virtual reality (VR). One

advantage of such interactive systems is the ability

to “visit” the workplace or campus without having to

spend time and money to physically get there. VR, in

particular, allows to visit otherwise inaccessible places

and can render a realistic environment, giving users

the feeling of being present in the virtual space (Chris-

tou, 2010; Zheng et al., 1998). At present, the main

disadvantages of VR systems for career guidance are

the comparatively high costs for hardware and mainte-

nance, which schools are often unable to afford. Nev-

ertheless, VR can present a promising addition to the

decision-maker’s toolset.

5.7 Limitations

In this paper, we categorized and evaluated OSAs de-

veloped by German universities or institutions. We

focused on tools that deal with study programs in the

STEM ﬁeld since this segment seemed sufﬁciently

broad and yet easy to conﬁne. OSAs from other coun-

tries or languages as well as other ﬁelds of study were

not within the scope of this work. We considered

twelve instruments sufﬁcient to make veriﬁable state-

ments about the characteristics of OSAs and to obtain

sound results. The selection of the tools was not fully

random in order to cover a wide variety of different ap-

proaches. Due to resource limitations, the analysis was

conducted by a single researcher, hence, there might

be biases in the evaluation. The answering of the ques-

tionnaires was mainly done to investigate the questions

and to get a tentative impression of the results as well

as the time it takes to complete the questionnaires.

CONCLUSIONS AND OUTLOOK

Career guidance is essential for students to ﬁnd out

their professional identity. Due to the young target

group that often uses digital devices, tools for online

career orientation gain importance. OSAs can both

help students ﬁgure out promising career options and

assist in selecting those that match their personality, vo-

cational preferences, and interests. To get an overview

of the current state of the art of online career guidance

systems, we classiﬁed N=12 existing OSAs from the

https://www.deinerstertag.de/berufsberatungen/, last

accessed 2024–04–11

German-speaking area. The evaluated tools range from

tests for general study program orientation to aptitude

tests for speciﬁc degree programs in the STEM area.

Our results show that OSAs have the potential to be

a resource-saving complement to face-to-face career

counseling. They are independent of time and location

and provide a safe anonymous space in which prospec-

tive students can independently assess their suitability.

However, we also found that current OSAs differ con-

siderably in their complexity, i. e., their content and

duration. Only few are based on established scientiﬁc

methods. As the developers of OSAs usually have

domain-speciﬁc backgrounds, the tests are most likely

optimized for their content but not necessarily for their

methodology. These drawbacks could be overcome

with universal guidelines. However, a one-ﬁts-all solu-

tion seems difﬁcult to achieve. It is thus important to

tailor each individual OSA to the needs of the speciﬁc

user group. In a recently published book (Stoll and

Weiss, 2022), the “Network Online Self-Assessment”

summarizes the experiences of eight working groups

in developing and evaluating OSAs over the past ten

years. Such work should receive more attention so

that new developments can draw on the experience of

earlier creators. In the best case, an OSA should be

developed by or with the help of experienced career

counselors and incorporate solid and proven scientiﬁc

methods.

While all of the tested tools are available in

a desktop and mobile version, they rely on static

question-answer formats and could, in principle, also

be conducted on paper. They are thereby wasting

the potential of modern technologies to provide ﬂex-

ible and adaptive content that is optimally tailored

to the user group of young adults. Current OSAs

usually only include requirements and contents of

study programs, rather than experience reports as well

as real insights into daily working routines of stu-

dents (Aragon Bartsch et al., 2022). Such content

could reinforce a student’s vision of a study program.

Other supporting technologies might be 360° videos

or more interactive virtual or augmented reality ap-

proaches providing deeper insights into careers and

potential future workspaces or incorporating virtual

tasks, counselors and fellow students (cf. (Demareva

et al., 2020)).

The output of the assessment ranges from mir-

roring the user’s answers to giving concrete career

recommendations. All tested tools lacked guidance

on how to interpret the test results from a decision-

maker’s perspective. Some of them did not even list

a contact person. To avoid insecurities, OSAs should

not replace humans completely, but rather be used

in combination with personal career counseling (Gati

Let’s Choose STEM: An Overview on Study Program Guiding Online Self-Assessments and Future Directions

143

and Asulin-Peretz, 2011; Galliott, 2017). A forum or

chat could relieve student counselors, make frequently

asked questions visible, and give interested and current

students the opportunity to get in contact. Only few

OSAs allow the results to be stored or shared (or even

printed). That can be important, e. g. when students

have an appointment for further career guidance and

want to discuss the test results with their adviser (Gati

and Asulin-Peretz, 2011).

If designed well, OSAs have the potential to sup-

port students in the prescreening and in-depth explo-

ration phase of their career choice process. However,

the choice itself must and should be made by the stu-

dent. The question here is what actually constitutes a

good choice. In our view, it is important that it is based

on sufﬁcient knowledge about the course of studies,

but also that it gives the decision-maker a good “gut

feeling”. Therefore, OSAs should empower users to

reﬂect on their personal traits and expectations and, if

necessary, seek additional information, e. g., from a

professional adviser.

REFERENCES

Akosah-Twumasi, P., Emeto, T. I., Lindsay, D., Tsey, K.,

and Malau-Aduli, B. S. (2018). A Systematic Review

of Factors That Inﬂuence Youths Career Choices – the

Role of Culture. Frontiers in Education, 3.

Aragon Bartsch, S., Schneegass, C., Bemmann, F., and

Buschek, D. (2022). A Day in the Life: Exploring

the Use of Scheduled Mobile Chat Messages for Ca-

reer Guidance. In Proceedings of the 20th Interna-

tional Conference on Mobile and Ubiquitous Multi-

media, MUM ’21, page 24–34, New York, NY, USA.

ACM.

Aragon-Hahner, S., K

orber, J., Schneller, M., and Strickroth,

S. (2023a). FindMyself: A Mobile Self-Reﬂection App

to Support Students in Career Decision-Making. In

Proceedings of the 22nd International Conference on

Mobile and Ubiquitous Multimedia, MUM 2023, page

463–476, New York, NY, USA. ACM.

Aragon-Hahner, S., Sakel, S., and Strickroth, S. (2023b). “Is

Computer Science the Right Study Program for Me?”:

Concept Development of a Mobile Self-Reﬂection

App for Prospective University Students. In Proceed-

ings of the 15th International Conference on Com-

puter Supported Education, pages 163–171. INSTICC,

SCITEPRESS.

Cameron, G., Cameron, D., Megaw, G., Bond, R., Mul-

venna, M., O’Neill, S., Armour, C., and McTear, M.

(2017). Towards a chatbot for digital counselling. In

Electronic Workshops in Computing. BCS Learning &

Development.

Choi, J. H. and Lee, H.-J. (2012). Facets of simplicity for

the smartphone interface: A structural model. Interna-

tional Journal of Human-Computer Studies, 70(2):129–

142.

Christou, C. (2010). Virtual Reality in Education. In Affec-

tive, Interactive and Cognitive Methods for E-Learning

Design: Creating an Optimal Education Experience,

pages 228–243. IGI Global.

Demareva, V., Petrova, I., and Zhukova, M. (2020). Digital

Career Guidance Technologies: Using Augmented and

Virtual Reality. Annual Review of Cybertherapy and

Telemedicine, 20:131–133.

Galliott, N. (2017). Online Career Guidance: Does Knowl-

edge Equate to Power for High School Students?

Journal of Psychologists and Counsellors in Schools,

27(2):190–207.

Gati, I. and Asulin-Peretz, L. (2011). Internet-Based Self-

Help Career Assessments and Interventions: Chal-

lenges and Implications for Evidence-Based Career

Counseling. Journal of Career Assessment, 19(3):259–

273.

Gati, I. and Kulcs

ar, V. (2021). Making better career deci-

sions: From challenges to opportunities. Journal of

Vocational Behavior, 126:103545.

Gati, I. and Saka, N. (2001). Internet-Based Versus Paper-

and-Pencil Assessment: Measuring Career Decision-

Making Difﬁculties. Journal of Career Assessment,

9(4):397–416.

Germeijs, V. and De Boeck, P. (2003). Career indecision:

Three factors from decision theory. Journal of Voca-

tional Behavior, 62(1):11–25.

Harris-Bowlsbey, J. (2013). Computer-Assisted Career Guid-

ance Systems: A Part of NCDA History. The Career

Development Quarterly, 61(2):181–185.

Hasenberg, S. and Schmidt-Atzert, L. (2014). Internet-

basierte Selbsttests zur Studienorientierung. Beitr

age

zur Hochschulforschung, 36(1):8–29.

Holland, J. L. (1997). Making vocational choices: A the-

ory of vocational personalities and work environments.

Psychological Assessment Resources.

Kleiman, T. and Gati, I. (2004). Challenges of Internet-Based

Assessment: Measuring Career Decision-Making Dif-

ﬁculties. Measurement and Evaluation in Counseling

and Development, 37(1):41–55.

Smitina, A. (2010). The link between vocational identity,

study choice motivation and satisfaction with studies.

Procedia - Social and Behavioral Sciences, 5:1140–

1145. WCPCG 2010.

Stoll, G. (2019). Online-Self-Assessments zur Studienfach-

wahl – wie Hochschulen die Potentiale dieses Instru-

ments effektiv nutzen k

onnen. In Driesen, C. and Ittel,

A., editors, Erfolgreich ankommen – Strategien, Struk-

turen und Best Practice deutscher Hochschulen f

ur den

Ubergang Schule-Hochschule, pages 65–75. Waxmann,

unster.

Stoll, G. and Weiss, S., editors (2022). Online-Self-

Assessments zur Studienfachwahl: Entwicklung –

Konzepte – Qualit

atsstandards. Springer Berlin Hei-

delberg.

Tracey, T. J. G. (2010). Interest Assessment Using New

Technology. Journal of Career Assessment, 18(4):336–

344.

CSEDU 2024 - 16th International Conference on Computer Supported Education

144

Vigurs, K., Everitt, J., and Staunton, T. (2017). The evidence

base for careers websites. what works? Technical

report, Careers and Enterprise Company.

Zheng, J., Chan, K., and Gibson, I. (1998). Virtual reality.

IEEE Potentials, 17(2):20–23.

Let’s Choose STEM: An Overview on Study Program Guiding Online Self-Assessments and Future Directions

145