Usability Study on Dutch e-Recruiting Services:

Limitations and Possibilities from the

Applicants' Perspective

Chris Jansen, Elfi Ettinger and Celeste Wilderom

Department of Information Systems & Change Management

University of Twente, The Netherlands

Abstract. In this research, applicants' perceived expectations, limitations and

service improvements concerning two Dutch e-Recruiting services (monster-

board.nl and vacant.nl) are investigated. Data from interviews and videotapes

have been analysed. The main perceived limitation in regard to e-Recruiting

sites is the lack of personal communication and contact. The majority of usabil-

ity problems stem from layout issues (monsterboard.nl), search functionality

(vacant.nl) and lack of information (both). Better search and matching func-

tions and the inclusion of personal elements into e-Recruiting service offerings

were the foremost desires of users.

1 Introduction

In the Netherlands, there are currently 15 large e-Recruiting services on the market

with vacancies ranging from 1,500 to 81,000 jobs [1]. While the presence of e- Re-

cruiting services is increasing, research into applicant perceptions of these services is

just emerging [2]. The widespread availability and large number of e-Recruiting ser-

vices show that finding jobs online is an important asset for applicants who seek

employment as well as for recruiters who aim to fill open positions as quickly as

possible [3], [4]. Understanding how the quality of online services is perceived by

applicants is important for an e-Recruiting site to be successful gaining and sustaining

customers, i.e. returning applicants [5]. Therefore, e-Recruiting services are chal-

lenged to analyze and incorporate the needs of their applicants into the design of their

information systems so as to offer desirable services. In this context, other research

has shown that applicants’ decision to use an e- recruiting site largely depends on the

specific resources or service attributes available [6], [7]. More than one in five job

seekers has rejected vacancies simply based on poorly designed recruiting websites

[8]. Moreover, due to the quick turnover of applicants, large numbers of e-Recruiting

services fail [9]. Consequently, there is strong need to study limitations and possibili-

ties of effective e-Recruiting sites. We reason, in line with previous research on hu-

man computer interaction (HCI), that the usability of e-Recruiting sites affects wheth-

er or not the site effectively facilitates the filling of job vacancies [10], [11]. This

research endeavors to discover applicants' perceptions, attitudes and expectations

Jansen C., Ettinger E. and Wilderom C. (2009).

Usability Study on Dutch e-Recruiting Services: Limitations and Possibilities from the Applicants’ Perspective.

In Proceedings of the 3rd International Workshop on Human Resource Information Systems, pages 120-128

DOI: 10.5220/0002202401200128

 SciTePress

while using two Dutch e-Recruiting services (i.e. monsterboards.nl and vacant.nl) and

gain insights on service innovations for encouraging potential (re)-use of these sites

[12]. The paper is organized as follows. In section two, the research scope is de-

scribed, including the research questions, test objects and participants. In section

three, the research method is described, including the experimental procedure, ques-

tionnaire and task structure, and data analysis. In section four, the results are pre-

sented. Conclusions and implications of this study are presented in section five.

2 Research Scope

This paper aims at exploring applicants' perceptions of limitations in using e-

Recruiting sites as well as exploring which features users associate with highly usea-

ble e- Recruiting sites. Accordingly, the main question is: What are the limitations of

e-Recruiting sites in general, and of monsterboard.nl and vacant.nl in specific, and

how can e- Recruiting sites enhance the usability and service offerings to encourage

(re)use of these services? The following three sub-questions are investigated:

(1) What expectations and experiences do applicants have regarding e-Recruiting

services?

(2) What are the perceived limitations of e-Recruiting services from the applicants'

perspective?

(3) How can e-Recruiting services improve their service offerings so that applicants

will re-use the services?

3 Method

3.1 Website Test Objects

Two websites offering e-Recruiting services were selected. The selection criteria

were twofold; the sample had to include both one generally focused site and one site

with a specific target group, and the second criterion was that the two sites have simi-

lar features. Including both a general and a specific site allows applicants to expe-

rience two "types" of e-Recruiting sites and provides a better basis for evaluation.

Similarity in features makes it possible to compare usability results obtained from

both sites. The decision was made to use monsterboard.nl and vacant.nl, the former

being a general e-Recruiting site and the latter specifically aimed at university gra-

duates [6]. Both sites offer functions to search for jobs, online resume forms and

newsletter registration, but differ in the amount and up-to-date status of vacancies,

and website structure.

3.2 Participants

This study was conducted with a convenience sample of eight applicants, all of whom

were students at the University of Twente, the Netherlands. The students participated

121

voluntarily and were selected mainly on basis of educational level. Half of the partic-

ipants were in a master's program, one participant had already graduated and three

were bachelor students. The participants represented six different fields of study. The

group was divided equally by gender, and the average age was 23.25 years. The ma-

jority of participants (seven out of eight) had little to no prior experience with e-

Recruiting sites.

3.3 Experimental Procedure

We conducted eight sessions held in the same video lab. The video lab contained a

laptop, three TV screens, a webcam and a video recorder. The video recorder was

used during each session to record the TV screen linked to the computer screen, and

the webcam used to record facial expressions and the applicant's voice. Each of the

eight session lasted between 2-2.5 hours, resulting in a total of over 16 hours of web-

cam and video data. The test and interview procedure remained constant during all

sessions. Two researchers took field notes and observed the users performing the

tasks. The applicants were asked to "think-aloud" during the whole session and to

provide arguments for their answers. The applicants filled out a pre-task and a post-

task questionnaire. To evaluate the usability of the e-Recruiting services all applicants

performed nine tasks on both monsterboard.nl and vacant.nl (see Table 1).

Table 1. Task List.

1. You are interested in working at Brunel Engineering; search all current vacancies at Brunel

Engineering.

2. You want to apply at Rabobank. Search for the recruiter's contact information.

3. You want to find a job in Friesland and Groningen. Search for a position related to your current

studies that may be suitable for you after graduation.

4. You want an IT job in Utrecht. Search for a full-time position in the IT sector in Utrecht.

5. You are interested in all university-level vacancies in Noord-Holland. Search for these vacan-

cies.

6. You want to contact the recruiting sites’ service personnel. Search for their contact information.

7. You are interested in the actual vacancies available; search all jobs in horeca and sort them

based on when they were posted, and select those posted in the last 30 days.

8. You are interested in receiving the sites’ latest information. Sign up for the newsletter.

9. You want to make a profile. Register and put a short resume profile online.

3.4 Data Analysis

The analysis of the collected data was divided into three parts: (1) the pre-task ques-

tionnaire, (2) the post-task questionnaire and (3) the task analysis. The answers to the

pre-task and post-task questionnaire were derived by analyzing the filled-in forms,

field notes, and video data. The results were supported by rich descriptions, i.e.,

quotes from the video analysis. To more reliably detect defects and usability prob-

lems, the applicants’ execution of tasks and website navigation was observed. For this

analysis, the video data of both the computer screen and the facial expressions were

analyzed, so that interpretation of verbalized defects (i.e. emphasis of certain salient

defects, perceived overlaps, similarities or contradictions) would not be as difficult to

122

judge. For the task analysis, one coder watched the full video and took note of all

occurring defects. By means of a second assessment of 50% of the video data (eight

hours), defects were categorized into broad defect categories. A second coder watch-

ed 10 % of the video material and independently derived defect categories. Compre-

hension issues in regard to labeling, inconsistencies, lacking categories and overlap

between defect categories were discussed and resolved. In order to enhance code

validity (internal and external homogeneity), one test applicant was asked to do

“member checking” of the codes and also watched one full video session. The coding

procedure resulted in a final set of six defect categories (see Appendix A). After hav-

ing agreed on the labeling of the final defect categories and explaining the defects in

one sentence, one coder watched the full video data and counted the total number of

defects for each task and session on both e-Recruiting sites. The combined use of

questionnaire analysis and website task analysis enhances the in-depth understanding

of the research issues [13].

4 Results

It is interesting to note that all applicants were familiar with monsterboard.nl although

the majority (7 out of 8) almost never used e-Recruiting sites. Surprisingly, none of

the applicants knew of the online career site for university graduates, vacant.nl.

4.1 Expectations General Site

All applicants stated the importance of including extensive search functionality in e-

Recruiting sites’ design. The search engine should be easy to use and offer detailed

selection options. One applicant supported this by arguing that "it should be easy to

make a selection to filter out all the jobs I am not interested in." In addition, all appli-

cants expected to see enough vacancies or fitting jobs on e-Recruiting sites. The local

availability of enough suitable jobs was also suggested. The term 'fitting' is worth

noting, because the interviewed applicants expected vacancies that matched their

education and interests and not just any random vacancy. Another expectation verba-

lized by most applicants (6 out of 8) was the ability to register and apply, i.e. fill in

their personal data into online resume forms. This was mentioned in combination

with the expectation to find matches with vacancies and can be described as the op-

tion for Resume forms/Upload. Half of the applicants desired extensive compa-

ny/recruiter information. One applicant acknowledged that he wanted to know "what

a company is about" and others liked to have as much company information as possi-

ble in order to able to see if they would fit into the firm, rather than just meet the

general criteria of the vacancy.

4.2 Expectations Academic Site

Five applicants mentioned that an academic site should have academic vacancies

only. No jobs with low qualifications should be posted and irrelevant companies

123

should not be allowed to post vacancies. One applicant articulated, "The offer they

have of jobs is supposed to be of a different level" and another stated more specifical-

ly, "You don't want to see that you can be a garbage man in that and that city at an

academic e-Recruiting site." A more personal approach is expected by half of the

applicants (4 out of 8). Some applicants mentioned that you should be able to include

your master’s thesis subject in your personal information. Others were interested in

exchanging information with students who are doing a bachelor/master thesis at the

different companies they are interested in. One applicant said that "in graduate boards

you get more sense of quality, I expect" and another suggested including an option to

enhance online communication with other registered users: "It would be useful to get

to know people who also graduated in your field with whom you can interact with." A

third expectation was verbalized by three applicants as information on career oppor-

tunities. This relates to the need for information from a company on growth and ca-

reer possibilities and possibilities of on-the-job training or traineeships. Some similar-

ities can be found between these findings and those in another study concerning e-

Recruiting site attributes for job seekers [6].

4.3 Limitations

Applicants pointed out that e-Recruiting sites have several limitations in comparison

to other ways of job searching options. Foremost mentioned was the lack of personal

communication and anonymity. The applicants verbalized this as lack of personal

contact, lack of face-to-face contact, lack of direct interaction or inability to steer the

information exchange. One applicant supported this by saying, "A resume does not

represent everything that you are.” A second applicant mentioned, "You are not face-

to-face with recruiters so people do not get a real and authentic first impression of

you." And a third said, "These sites lack personal contact with people who have expe-

riences or are part of the company you want to work for." This limitation is inter-

preted as the lack of personal communication. A second limitation was linked to the

lack of personal contact. One applicant pointed out: "Sometimes it takes a week to get

a reply, or longer" and another stated that she had doubts concerning responsiveness

in regard to information requests. These results indicate a lack of responsiveness. In

other research it has also been noted that the level of interactivity is a specific and

important element for e-Recruiting success [14], [8]. A third limitation was formu-

lated by one applicant as: "You don't know if all the offers are there; or if you are

missing something." Another applicant assumed that there might be more bad or

wrong matches between job and applicant, because the job ad description might not

be understood correctly. Finally, an applicant explained, "You want to know what the

company is about and the people who work there, that's not going to happen using an

internet site." This can be understood as a lack of information richness.

4.4 Website Task Results

In the process of completing website tasks the applicants perceived or stumbled upon

certain defects. The defects were categorized according to the table in Appendix B.

124

There is a large difference between the total amount of defects identified for vacant.nl

and the number identified on monsterboard.nl. Data entry defects occurred most fre-

quently on vacant.nl (23), followed by lack of information (17), matching (13), and

layout defects (10). The amount of unique defects per category was highest for data

entry defects (7) followed by lack of information (4), layout (4) and matching defects

(3). Vacant.nl needs to add search fields to select (1) employment type (i.e. fulltime,

part-time), (2) up-to-date status of vacancies, (3) keyword field in quick search, (4)

multiple provinces and cities and (5) the ability to use multiple keywords together. In

the lack of information category, 41% of these defects concerned insufficient contact

information under the contact link. Twenty-nine percent relate to difficulties in using

the search engine. A lack of company/recruiter information was related to 24% of

these defects. An interesting result of the website task analyses is that layout prob-

lems and lack of information are the two main defect categories for both e-Recruiting

sites. In other research it has been pointed out that website content is not the only

important component of a successful e-Recruiting site. Rather, formatting attractive-

ness and functionality have been found to be important, too [15]. Complementary

research has even showed that formatting attractiveness could even be more impor-

tant than usability issues [16]. Formatting attractiveness can be defined as the clarity

and logical structure of a website design. To cater well to applicants, e-Recruiting site

designers should consider adjusting the layout and providing more information on

how to use the sites. Placement and provision of company/recruiter information can

also be largely improved on for both websites. Other researchers also noted that job

seekers frequently lack information on organizational attributes, i.e., more informa-

tion about the job and/or organization makes these sites more attractive for applicants

[17], [14]. The study participants emphasized the need for more personal contact

which could be supported by this option. Changing the current placement of contact

and newsletter information needs to be improved on both sites. While the search

options on monsterboad.nl were judged to be extensive enough, the placement of

search functions could be improved. A more obvious home button, better labeling and

use of head menus would all be worthwhile to consider. For vacant.nl the search

functionality should be expanded and the quick search menu should have a keyword

search field. The layout of the search engine also needs to be redesigned. The appli-

cants stated that for the resume fields at monsterboard.nl, better examples and more

explanations could reduce the amount of defects. The same is applicable to vacant.nl,

although here designers should extend the resume forms with the addition of more

fields.

4.5 Suggestions for Improvement

When the applicants were inquired about features that would make them return to the

e-Recruiting sites, 62.5% (5 out of 8) stated they would just use the site for searching

for a job. "Just searching for a job, placing a resume and finding company informa-

tion" was the goal of one applicant. A second applicant stated, "I don't expect so

much from an internet recruiting site, I just want to be informed of jobs and that's it."

A third applicant was "just looking for a job." An interesting finding was that 75% (6

out of 8) of the applicants specifically argued that social (network) or career elements

125

would not make them come back to use the site. One applicant mentioned, "You don't

want to make this into a Hyves or something." Further, one applicant explained,

"There are already so many social sites and things." The majority of applicants did

not expect social elements. If personal network elements were to be added to an e-

Recruiting site, the applicants felt that it would be important to maintain a difference

between business and private life. One applicant said: "E- Recruiting sites should be

"work-related and branch oriented, this is a nice addition. "It was also suggested that

unprofessional networks should be excluded, which implies that e-Recruiting sites

have to be cautious about which network elements to include. The applicants in this

research clearly require these elements to only be focused on their professional niche

and career. Half of the applicants felt that published reports about other users’ work

experiences in different companies would make them re-use the e-Recruiting site,

provided that the sources are reliable and objective. Three out of eight applicants felt

that newsfeed elements such as new vacancy updates and resume statistics would

make them come back to use the site. One applicant reported that it would be good if

one could choose specific categories of jobs and regularly receive updates on new

vacancies.

5 Conclusions

In this paper, we investigated two Dutch e-Recruiting sites, i.e., monsterboard.nl and

vacant.nl, to determine usability and possible service innovations of e-Recruiting

sites. Perceptions, experiences and desires of eight applicants were analyzed using

questionnaires and website task analysis. The findings imply that the e-Recruiting

sites are expected to have extensive search functionality; sophisticated matching func-

tionality; resume creation options; provide detailed vacancy, recruiter and company

information; and aim for more personal and proactive contact between job seekers

and recruiters. The backbone of a highly useable e-Recruiting site lies in its ability to

support a multitude of correct matches between results and search terms that are up to

date. Further, results show that the aesthetics of vacant.nl (website design) are per-

ceived as providing a better overall impression to new users, but the functionality of

monsterboard.nl swings the balance in their favor [18]. While monsterboard.nl has

many layout issues accompanied by a perceived lack of information, vacant.nl re-

quires more extensive search engines, more appropriate vacancies and also scores low

on information quality. Applicants stated that the limitations of e- Recruiting sites

mainly revolve around the lack of personal communication, responsiveness and in-

formation [19]. Several recommendations of how providers can learn from the per-

ceived expectations and limitations were derived. This study shows that applicants

require a clear overview of the e-Recruiting sites content and improving website

design is one way to improve the usability of these sites. The applicants also expect

more personal contact and interaction with the recruiter and company of their prefe-

rence. Therefore, adding more organizational information can enhance an e-

Recruiting services’ use. Another new research avenue that might be stimulated by

this study would be to extensively observe (i.e. with eye tracking) recruiters navigat-

ing different recruiting sites and collecting data to compare the usability of these sites.

126

This could also provide further insights on the intensity and frequency with which

certain elements draw applicants' attention. Other authors have successfully used a

Job Site Evaluation Framework (JSOF) to compare multiple e- Recruiting sites [20].

References

1. An overview of all national vacancy sites, http://www.alle-vacatures.nl, retrieved Feb 21,

2009.

2. Lee, I., 2005. Evaluation of Fortune 100 companies' web sites. Human Systems Manage-

ment, 24(2): 175.

3. Gueutal, H.G., Stone, D.L., 2005. The Brave New World of eHR : Human Resource Man-

agement in the Digital Age. Jossey Bass. San Fransisco.

4. Feldman, D.C., Klaas, B.S., 2002. Internet job hunting: A field study of applicant expe-

riences with on-line recruiting. Human Resource Management, 41(2): 175-192.

5. Grönroos, C., 2007. Service Management and Marketing: Customer Management in Ser-

vice Competition ed. 3. John Wiley & Sons Ltd. West Sussex.

6. Koong, K.S., 2002. An identification of Internet Job Board attributes. Human Systems

Management, 21(2): 129-135.

7. Ettinger, E., Wilderom, C., 2008. Sustainable e-recruiting portals: How to motivate appli-

cants to stay connected throughout their Careers. in Conference of the Association of Inter-

net Researchers. IT University of Copenhagen, Denmark.

8. Maurer, S.D., 2007. Developing effective e-recruiting websites: Insights for managers from

marketers. Business Horizons, 50(4): 305.

9. Lin, B., 2002. Data warehousing management issues in online recruiting. Human Systems

Management, 21(1): p. 1-8.

10. Parry, E., 2008, Drivers of the adoption of online recruitment: Ana analysis using diffusion

of innovation theory, in E-HRM in theory and practice. Elsevier. Amsterdam.

11. Smith, A.D., Rupp, W.T., 2004. Managerial challenges of e-recruiting: extending the life

cycle of new economy employees. Online Information Review, 28(1): 61-74.

12. The e-recruitment sites used for this study: http://www.monsterboard.nl and

http://www.vacant.nl, retrieved Feb 18, 2009.

13. Miles, M. B., Huberman, A. M. 1994. Qualitative data analysis. (2nd Ed.). Sage. Thousand

Oaks, CA.

14. Breaugh, J.A., 2000. Research on Employee Recruitment: So Many Studies, So Many

Remaining Questions. Journal of Management, 26(3): 405.

15. Cober, R.T., Brown, D.J., Levy, P.E., 2004. Form, content, and function: An evaluative

methodology for corporate employment Web sites. Human Resource Management, 43(2-

3): 201-218.

16. Thompson, L.F., Braddy, P.W., and Wuensch, K.L., 2008. E-recruitment and the benefits

of organizational web appeal. Computers in Human Behavior, 24(5): 2384-2398.

17. Barber, A.E., 1993. Job postings and the decision to interview: a verbal protocol analysis.

Journal of Applied Psychology, 78(5): 845.

18. Bartram, D., 2000. Internet Recruitment and Selection: Kissing Frogs to find Princes. In-

ternational Journal of Selection and Assessment, 8(4): 261.

19. Smyth, B., Bradley, K., Rafter, R., 2002. Personalization techniques for Online recruitment

services. Communications of the ACM, 45(5): 39-40.

20. Terzis, V., 2005. Job Site Evaluation Framework (JSEF) and comparison of Greek and

foreign job sites. Human Systems Management, 24(3): 223.

127

Appendix

Appendix A. Defect Categories.

Category Name Explanation

1. Layout defects The applicant fails to spot a particular element within a page of the web-

site.

2.Terminology defects The applicant does not understand the meaning of certain words.

3. Lack of Information

defects

The applicant perceives a lack of information to perform tasks successful-

ly.

4. System defects The applicant loses data or experiences a problem due to system problems.

5. Matching defects The applicant does not receive the matches expected with the used search

terms.

6. Data Entry defects The applicant is lacking the option to select/insert data which he/she

wishes to use.

Appendix B. Defects e-Recruiting site.

Defect Categories Monster tasks Monster resume Vacant tasks Vacant resume

Total Unique Total Unique Total Unique Total Unique

Layout 42 7 1 1 10 4 3 1

Terminology 3 3 19 4 1 1 8 3

Lack of information 21 5 13 5 17 4 4 3

System 9 3 2 1 1 1 2 1

Data entry - - 6 2 23 7 1 1

Matching 7 3 - - 13 3 - -

Total 82 21 41 13 65 20 18 9

128