EyeRec: An Open-source Data Acquisition Software for Head-mounted

Eye-tracking

Thiago Santini, Wolfgang Fuhl, Thomas K

ubler and Enkelejda Kasneci

Perception Engineering Group, University of T

ubingen, T

ubingen, Germany

Keywords:

Eye Movements, Recording Software, Pupil Detection, Calibration, Open-source, Data Acquisition.

Abstract:

Head-mounted eye tracking offers remarkable opportunities for human computer interaction in dynamic sce-

narios (e.g., driving assistance). Although a plethora of proprietary software for the acquisition of such eye-

tracking data exists, all of them are plagued by a critical underlying issue: their source code is not available to

the end user. Thus, a researcher is left with few options when facing a scenario in which the proprietary soft-

ware does not perform as expected. In such a case, the researcher is either forced to change the experimental

setup (which is undesirable) or invest a considerable amount of time and money in a different eye-tracking

system (which may also underperform). In this paper, we introduce EyeRec, an open-source data acquisition

software for head-mounted eye-tracking. Out of the box, EyeRec offers real-time state-of-the-art pupil detec-

tion and gaze estimation, which can be easily replaced by user implemented algorithms if desired. Moreover,

this software supports multiple head-mounted eye-tracking hardware, records eye and scene videos, and stores

pupil and gaze information, which are also available as a real-time stream. Thus, EyeRec can be an efﬁcient

means towards facilitating gazed-based human computer interaction research and applications. Available at:

www.perception.uni-tuebingen.de

1 INTRODUCTION

In the past two decades, the number of researchers

using eye trackers has grown enormously (Holmqvist

et al., 2011). These researchers stem from several

distinct ﬁelds (Duchowski, 2002). For instance, eye

tracking has been employed from simple and ﬁxed

scenarios (e.g., language reading (Holsanova et al.,

2006; Rayner, 1998)) to complex and dynamic cases

(e.g., driving (Kasneci et al., 2014)). Naturally, these

distinct ﬁelds usually have speciﬁc needs, which have

lead to the spawning of several proprietary systems

with different capabilities. In fact, Holmqvist et

al. (Holmqvist et al., 2011) report that they were

able to ﬁnd 23 companies selling video-based eye-

tracking systems in 2009. Typically, these commer-

cial systems rely on closed-source software, offering

their eye tracker bundled with their own software so-

lutions. Although some companies offer technical

transparency, satisfying researchers that want to know

how data are gathered and analyzed by the proprietary

software, one issue persists: what to do if the soft-

ware fails to perform as desired. Facing such an is-

sue, a researcher has few options: 1) changing the ex-

perimental scenario until the software performs prop-

erly, which is clearly undesirable and counterproduc-

tive, or 2) switching to a different eye-tracking sys-

tem, which not only requires a considerable amount of

time and money but also offers no guarantees that the

alternative system will perform as desired. Few open-

source alternatives do exist, such as openEyes (Li

et al., 2006), PyGaze (Dalmaijer et al., 2014), and the

very promising Pupil (Kassner et al., 2014); however,

these focus on their own eye trackers or depend on

existing APIs from manufacturers.

In this paper, we introduce EyeRec, an open-

source data acquisition software for head-mounted

eye tracking. In fact, we started with the development

of EyeRec as we found the pupil detection algorithm

of a proprietary software of a mobile head-mounted

eye tracker to be underperforming in outdoor scenar-

ios. The key advantages of EyeRec are:

• Research Speed-up: the code is freely avail-

able. Users can easily replace built-in algorithms

to prototype their own algorithms or use the soft-

ware as is for data acquisition.

• Multiple Head-mounted Eye Tracking Hard-

ware: tested with several commercial eye trackers

and USB cameras. Any eye tracker that exposes

386

Santini, T., Fuhl, W., Kübler, T. and Kasneci, E.

EyeRec: An Open-source Data Acquisition Software for Head-mounted Eye-tracking.

DOI: 10.5220/0005758903840389

In Proceedings of the 11th Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications (VISIGRAPP 2016) - Volume 3: VISAPP, pages 386-391

ISBN: 978-989-758-175-5

its cameras via USB should work out of the box.

• Real-time: a low latency software pipeline allows

its usage in time-critical applications.

• State-of-the-art Pupil Detection: ElSe (Fuhl

et al., 2016), the top performer pupil detection al-

gorithm, is fully integrated. Moreover, to demon-

strate the ease of introducing new algorithms to

EyeRec, we have integrated the popular Star-

burst (Li et al., 2005),

Swirski et al. (

Swirski et al.,

2012), and ExCuSe (Fuhl et al., 2015) algorithms.

• Data Streaming: non-video data is streamed

in real time through UDP/TCP connections, al-

lowing easy integration with external buses (e.g.,

CAN).

• Full Open-source Software Stack: EyeRec ﬁlls

the data acquisition gap. Combined with open-

source eye-tracking data analysis software, such

as EyeTrace (K

ubler et al., 2015), a full open-

source software stack is available.

The remainder of the paper is divided as follows.

Section 2 describes the overall design and perfor-

mance of EyeRec. Section 3 and Section 4 give details

on the built-in pupil detection and gaze estimation al-

gorithms, respectively. Section 5 concludes this work

and presents future work.

2 EyeRec

EyeRec is written in C++ and makes extensive use of

the OpenCV (Bradski et al., 2000) library for image

processing, VideoMan (Martirena, Javier B., 2015) li-

brary for video input, and the Qt (Qt Project, 2015)

framework for its Graphical User Interface (GUI).

At the moment, its development is focused on the

Windows version; nonetheless, porting to other plat-

forms should be possible as all components are cross-

platform.

EyeRec is designed to work with monocular head-

mounted eye trackers. Its architecture is based on a

master/slave paradigm, based on the idea that infor-

mation is most relevant when a new eye image be-

comes available. Thus, the eye camera is deﬁned

as the master and the scene camera as a slave. It is

worth noticing that this master/slave scheme can be

extended to a binocular system by treating the sec-

ond eye camera as a slave. Whenever the master

frame grabber captures a new eye image and its as-

sociated timestamp, EyeRec fetches the latest scene

image from the slave frame grabber and bundles the

images and timestamp in a single entity hereby re-

ferred to as entry. Additionally, EyeRec can be con-

ﬁgured to undistort the ﬁeld image if necessary, and

camera calibration is available. This completes the

image acquisition stage.

Afterwards, the newly generated entry is sent to

the image processing stage. At this stage, the pupil

detection algorithm is applied, and the resulting esti-

mated pupil position is used to estimate the gaze posi-

tion. Both pupil and gaze position estimates as well as

other useful information (e.g., pupil area) are added to

the entry, which is then forwarded to the data stream-

ing, data journaling, and GUI update stages.

At the data streaming stage, all non-frame data

present in the entry are transmitted through a TCP

or UDP server (based on user conﬁguration). At the

data journaling stage, non-frame data are appended

to a journal ﬁle, and frame data are appended to the

pertinent videos. At the GUI update stage, pupil and

gaze position are overlaid on frame data, which are

then displayed to the user (see Figure 1).

Each of the aforementioned stages runs in its own

execution thread. Image acquisition, image process-

ing, and data streaming stages are all considered time-

critical and run with highest priority. Remaining

stages run with regular priority.

Figure 1: A subject with high degree of myopia (9.00D

SPH) and astigmatism (5.00D CYL) stares at the Record-

ing Start button in the GUI after a ﬁve point calibration. A

challenging situation due to the glasses black frame, thick

lenses, reﬂections, and eye camera position (off axis and not

centered relative to the subject’s eye).

An additional beneﬁt from EyeRec’s master/slave

architecture is that the data rate is synchronized with

the master image sensor. Thus, provided that the mas-

ter sensor produces new images at a constant rate,

EyeRec will generate data entries at a constant rate,

including the most relevant (i.e., the latest available)

slave image. In comparison, the data recorded with

proprietary software during our tests exhibited a large

variance in the intersample period length, which sug-

gests that software buffers are used to synchronize the

data. This can be seem in Figure 2 and has a large im-

pact in derived data that depends on the timestamps

EyeRec: An Open-source Data Acquisition Software for Head-mounted Eye-tracking

387

(e.g., eye velocity) – for instance, eye velocity esti-

mates with the similar physical displacement displays

velocity discrepancies up to a factor of four times dur-

ing our tests when using the software vendor. As a re-

sult, algorithms that depend on this information, such

as the Velocity Threshold Identiﬁcation (Salvucci and

Goldberg, 2000) for saccade/ﬁxation classiﬁcation,

can have their performance impaired.

-65%

-20%

Expected

+20%

+65%

1 101 201

Intersample Period (ms)

Samples

EyeRec

Vendor SW

Figure 2: Period between adjacent samples produced by Ey-

eRec and by proprietary software for the same eye tracker

device. While the proprietary software exhibits large vari-

ance, EyeRec’s variance is only due to the rounding of the

timestamp representation.

2.1 Data Format

Instead of reinventing the wheel, EyeRec follows the

same data format as described in the Dikablis Soft-

ware Version 2.0 User Manual. This makes possible

to use data recorded with EyeRec in data analysis soft-

ware that support Dikablis recordings. Upon record-

ing, eye and scene videos are recorded, and data is

stored in a journal ﬁle. Additionally, a ﬁle containing

points used for calibration is also generated.

2.2 Supported Eye Trackers

Currently, the software supports Ergoneers’ Dikablis

Essential and Dikablis Professional eye trackers (Er-

goneers, 2015). However, any eye tracker that ex-

poses its cameras as USB cameras should work effort-

lessly. For instance, EyeRec is able to use regular web

cameras instead of an eye tracker, although the built-

in pupil-detection methods are heavily dependent on

the quality of the eye image.

2.3 Pipeline Latency

We evaluated the latency of the software pipeline im-

plemented in EyeRec using a Dikablis Professional

eye tracker and a machine running Windows 8.1 with

an Intel

 Core

i5-4590 @ 3.30GHz CPU and

8GB of RAM. Table 1 shows the latency of each stage

relative to the moment a new eye image is available,

so each consecutive step includes the processing time

of prior steps. Data was collected during 10 minutes,

yielding 18,000 samples. The eye tracker used for this

evaluation provides 30 frames per second (for a single

eye). Therefore, the deadline for the real-time stages

is 33 ms, which is met with a slack larger than 20 ms.

Table 1: Latency relative to the moment a new eye image is

available for each stage of the software pipeline (measured

from 18,000 samples).

Stage Mean Latency (ms) Std. Dev.

Image Acquisition 1.36 0.32

Image Processing 9.16 0.83

Data Streaming 9.54 0.84

GUI Update 10.95 0.99

Data Journaling 17.25 2.07

2.4 Hardware Requirements

Given the slack for the reference machine, we esti-

mate that any Intel

 Core

machine with at least

2GB of RAM should be able to handle the software

requirements.

3 PUPIL DETECTION

Overall, we evaluated ﬁve candidates for the pupil de-

tection algorithm.

Starburst (Li et al., 2005) ﬁrst removes the

corneal reﬂection and then selects pupil edge candi-

dates along rays extending from a starting point. Re-

turning rays are sent from the candidates found in the

previous step, collecting additional candidates. This

process is repeated iteratively using the average point

from the candidates as starting point until conver-

gence. Afterwards, inliers and outliers are identiﬁed

using the RANSAC algorithm, a best ﬁtting ellipse is

determined, and the ﬁnal ellipse parameters are deter-

mined by applying a model-based optimization.

Swirski et al. (

Swirski et al., 2012) starts with

Haar features of different sizes for coarse positioning.

For a window surrounding the resulting position, an

intensity histogram is calculated and clustered using

the k-means algorithm. The separating intensity value

between both clusters is used as threshold to extract

the pupil. A modiﬁed RANSAC method is applied to

the threshold-border.

SET (Javadi et al., 2015) ﬁrst extracts pupil pixels

based on a luminance threshold. The resulting image

is then segmented, and the segment borders are ex-

tracted using the Convex Hull method. Ellipses are

VISAPP 2016 - International Conference on Computer Vision Theory and Applications

388

ﬁt to the segments based on their sinusoidal compo-

nents, and the ellipse closest to a circle is selected as

pupil.

ExCuSe (Fuhl et al., 2015) selects an initial

method (best edge selection or coarse positioning)

based on the intensity histogram of the input image.

The best edge selection ﬁlters a Canny edge image

based on morphologic operations and selects the edge

with the darkest enclosed value. For the coarse posi-

tioning, the algorithm calculates the intersections of

four orientations from the angular integral projection

function (Mohammed et al., 2012). This coarse posi-

tion is reﬁned by analyzing the neighborhood of pix-

els in a window surrounding this position. The image

is thresholded, and the border of threshold-regions is

used additionally to ﬁlter a Canny edge image. The

edge with the darkest enclosed intensity value is se-

lected. For the pupil center estimation, a least squares

ellipse ﬁt is applied to the selected edge.

ElSe (Fuhl et al., 2016) applies a Canny edge de-

tection method, removes edge connections that could

impair the surrounding edge of the pupil, and evalu-

ates remaining edges according to multiple heuristics

to ﬁnd a suitable pupil ellipse candidate. If the initial

approach fails, the image is ﬁrst downscaled. Then,

its response to a surface difference ﬁlter is multiplied

by its response to a mean ﬁlter, and the maximum re-

sponse position is taken. This position is then reﬁned

on the upscaled image based on its pixel neighbor-

hood. As ElSe is the default algorithm in EyeRec (see

Section 3.1), its workﬂow is presented in more details

in Figure 3.

3.1 Detection Rates

Table 2 shows the detection rates of all evaluated al-

gorithms up to an Euclidean pixel distance of ﬁve to

the hand-labeled pupil center. The data set is taken

from (Fuhl et al., 2015;

Swirski et al., 2012). All al-

gorithms are used with the parameters provided by the

authors. For Starburst, the starting location was set to

the center of the image.

Table 2: Detection rate up to an error of ﬁve pixels of all

integrated algorithms on all data sets proposed in (

Swirski

et al., 2012) and (Fuhl et al., 2015), totaling 39,001 images.

SET Starburst

Swirski ExCuSe ElSe

14.0% 9.8% 24.8% 58.4% 74.05%

Based on these results, we chose ElSe as the default

pupil detection algorithm. Nonetheless, Starburst,

Swirski, and ExCuSe are also available to demon-

strate EyeRec’s modularity and how easily one can

replace such essential parts.

4 GAZE ESTIMATION

The gaze estimation consists of a mapping of the pupil

position in the eye image to the scene plane. This is

typically done in two steps: calibration and projec-

tion. During calibration, a pupil position estimate is

collected while the subject focuses on a known posi-

tion in the scene image, forming a calibration pair.

4.1 Integrated Calibration

Typically, vendors use a ﬁxed amount of calibration

pairs (e.g., D-Lab always uses 4 points (Ergoneers,

2014)). EyeRec is unrestricted in this regard as it can

collect as many calibration points as the user desires

and, therefore, can reﬁne calibration as well as report

an estimate of calibration accuracy. Moreover, all col-

lected pairs of pupil estimates and scene image coor-

dinates are saved, making it possible to perform gaze

estimation ofﬂine, e.g., removing individual calibra-

tion pairs, in case the online gaze estimation did not

perform as desired.

4.2 Integrated Gaze Estimation

A gaze estimation based on the OpenCV

cv::ﬁndHomography function is integrated in

EyeRec. After a set of calibration pairs are col-

lected, they are fed into the cv::ﬁndHomography

function, yielding a calibration matrix. After suc-

cessful calibration, each pupil position estimate is

combined with the calibration matrix through the

cv::perspectiveTransform function to generate a gaze

position estimate. This procedure requires at least

four calibration points.

5 FINAL REMARKS

In this paper, we introduced a new data acquisi-

tion software for head-mounted eye trackers. Ey-

eRec has several key advantages over proprietary

software (e.g., openness) and other open-source al-

ternatives (e.g., multiple eye trackers support, im-

proved pupil detection algorithm). In this way, Ey-

eRec ﬁlls the data acquisition gap, enabling a full

open-source software stack for for acquisition and

analysis of eye-tracking data. The main shortcom-

ing at the moment is the fact that the performance of

the integrated gaze estimation method is not on par

with state-of-the-art implementations, which usually

consider device-speciﬁc effects. Besides solving the

aforementioned shortcoming, future work includes

automatic 3D eye model construction (Tsukada and

EyeRec: An Open-source Data Acquisition Software for Head-mounted Eye-tracking

389

Figure 3: ElSe (Fuhl et al., 2016) workﬂow. (1) input image. (2) Canny edge image. (3) The remaining edges after curvature

analysis, analysis of the enclosed intensity value, and shape. (4) The curvature with the most enclosing low intensity values

and the most circular ellipse is chosen as the pupil boundary. (5) The resulting pupil center estimate. In case the initial

estimate fails to produce a valid ellipse: the input image (6) is downscaled (7). (8) The downscaled image after convolution

with a mean and a surface difference ﬁlters. (9) The threshold for pupil region extraction is calculated and used for pupil area

estimation. (10) The resulting pupil center estimate.

Kanade, 2012;

Swirski and Dodgson, 2013), support

for remote gaze estimation (Model and Eizenman,

2010; Yamazoe et al., 2008), additional calibration

methods (Guestrin and Eizenman, 2006; Pirri et al.,

2011), real-time eye movement classiﬁcation based

on Bayesian mixture models (Kasneci et al., 2014;

Kasneci et al., 2015; Santini et al., 2016), automatic

blink detection, adding support for other eye track-

ers, and adding support for binocular systems as well

as external, user deﬁnable, triggers events. Feedback

from the community on future features is also wel-

come.

Source code, binaries for Windows, and extensive

documentation are available at:

www.perception.uni-tuebingen.de

REFERENCES

Bradski, G. et al. (2000). The OpenCV Library. Doctor

Dobbs Journal, 25(11):120–126.

Dalmaijer, E. S., Math

ot, S., and Van der Stigchel, S.

(2014). Pygaze: An open-source, cross-platform tool-

box for minimal-effort programming of eyetracking

experiments. Behavior research methods, 46(4).

Duchowski, A. T. (2002). A breadth-ﬁrst survey of eye-

tracking applications. Behavior Research Methods,

Instruments, & Computers, 34(4):455–470.

Ergoneers (2014). D-Lab Manual.

Ergoneers (2015). Dikablis Glasses.

http://www.ergoneers.com/en/hardware/eye-

tracking/.

Fuhl, W., K

ubler, T., Sippel, K., Rosenstiel, W., and Kas-

neci, E. (2015). Excuse: Robust pupil detection in

real-world scenarios. In Computer Analysis of Im-

ages and Patterns 2015. CAIP 2015. 16th Interna-

tional Conference. IEEE.

Fuhl, W., Santini, T., K

ubler, T., and Kasneci, E. (2016).

Else: Ellipse selection for robust pupil detection in

real-world environments. In Proceedings of the Sym-

posium on Eye Tracking Research and Applications.

ACM. Forthcoming.

Guestrin, E. D. and Eizenman, M. (2006). General theory

of remote gaze estimation using the pupil center and

corneal reﬂections. Biomedical Engineering, IEEE

Transactions on, 53(6):1124–1133.

Holmqvist, K., Nystr

om, M., Andersson, R., Dewhurst, R.,

Jarodzka, H., and Van de Weijer, J. (2011). Eye track-

ing: A comprehensive guide to methods and measures.

Oxford University Press.

Holsanova, J., Rahm, H., and Holmqvist, K. (2006). Entry

points and reading paths on newspaper spreads: com-

paring a semiotic analysis with eye-tracking measure-

ments. Visual communication, 5(1):65–93.

Javadi, A.-H., Hakimi, Z., Barati, M., Walsh, V., and

Tcheang, L. (2015). Set: a pupil detection method

using sinusoidal approximation. Frontiers in neuro-

engineering, 8.

Kasneci, E., Kasneci, G., K

ubler, T. C., and Rosenstiel, W.

(2014). The applicability of probabilistic methods to

the online recognition of ﬁxations and saccades in dy-

namic scenes. In Proceedings of the Symposium on

Eye Tracking Research and Applications, ETRA ’14,

pages 323–326, New York, NY, USA. ACM.

Kasneci, E., Kasneci, G., K

ubler, T. C., and Rosenstiel, W.

(2015). Online recognition of ﬁxations, saccades, and

smooth pursuits for automated analysis of trafﬁc haz-

ard perception. In Koprinkova-Hristova, P., Mlade-

nov, V., and Kasabov, N. K., editors, Artiﬁcial Neu-

ral Networks, volume 4 of Springer Series in Bio-

/Neuroinformatics, pages 411–434. Springer Interna-

tional Publishing.

Kassner, M., Patera, W., and Bulling, A. (2014). Pupil: An

VISAPP 2016 - International Conference on Computer Vision Theory and Applications

390

open source platform for pervasive eye tracking and

mobile gaze-based interaction. In Proceedings of the

2014 ACM International Joint Conference on Perva-

sive and Ubiquitous Computing: Adjunct Publication,

UbiComp ’14 Adjunct, pages 1151–1160, New York,

NY, USA. ACM.

ubler, T., Sippel, K., Fuhl, W., Schievelbein, G., Aufre-

iter, J., Rosenberg, R., Rosenstiel, W., and Kasneci,

E. (2015). Analysis of eye movements with Eyetrace,

volume In press. Communications in Computer and

Information Science (CCIS), Biomedical Engineer-

ing Systems and Technologies. Springer International

Publishing.

Li, D., Babcock, J., and Parkhurst, D. J. (2006). openeyes:

A low-cost head-mounted eye-tracking solution. In

Proceedings of the 2006 Symposium on Eye Tracking

Research &Amp; Applications, ETRA ’06, pages 95–

100, New York, NY, USA. ACM.

Li, D., Winﬁeld, D., and Parkhurst, D. J. (2005). Star-

burst: A hybrid algorithm for video-based eye track-

ing combining feature-based and model-based ap-

proaches. In Computer Vision and Pattern Recogni-

tion Workshops, 2005. CVPR Workshops. IEEE Com-

puter Society Conference on, pages 79–79. IEEE.

Martirena, Javier B. (2015). The VideoMan Library.

http://videomanlib.sourceforge.net/.

Model, D. and Eizenman, M. (2010). User-calibration-free

remote gaze estimation system. In Proceedings of the

2010 Symposium on Eye-Tracking Research & Appli-

cations, pages 29–36. ACM.

Mohammed, G. J., Hong, B. R., and Jarjes, A. A.

(2012). Accurate pupil features extraction based on

new projection function. Computing and Informatics,

29(4):663–680.

Pirri, F., Pizzoli, M., and Rudi, A. (2011). A general method

for the point of regard estimation in 3d space. In Com-

puter Vision and Pattern Recognition (CVPR), 2011

IEEE Conference on, pages 921–928. IEEE.

Qt Project (2015). The Qt Framework. http://www.qt.io/.

Rayner, K. (1998). Eye movements in reading and informa-

tion processing: 20 years of research. Psychological

bulletin, 124(3):372.

Salvucci, D. D. and Goldberg, J. H. (2000). Identifying

ﬁxations and saccades in eye-tracking protocols. In

Proceedings of the 2000 symposium on Eye tracking

research & applications, pages 71–78. ACM.

Santini, T., Fuhl, W., K

ubler, T., and Kasneci, E. (2016).

Bayesian identiﬁcation of ﬁxations, saccades, and

smooth pursuits. In Proceedings of the Symposium

on Eye Tracking Research and Applications. ACM.

Forthcoming.

Swirski, L., Bulling, A., and Dodgson, N. (2012). Robust

real-time pupil tracking in highly off-axis images. In

Proceedings of the Symposium on Eye Tracking Re-

search and Applications, pages 173–176. ACM.

Swirski, L. and Dodgson, N. A. (2013). A fully-automatic,

temporal approach to single camera, glint-free 3d eye

model ﬁtting. In Proceedings of ECEM 2013.

Tsukada, A. and Kanade, T. (2012). Automatic acquisi-

tion of a 3d eye model for a wearable ﬁrst-person vi-

sion device. In Proceedings of the Symposium on Eye

Tracking Research and Applications, pages 213–216.

ACM.

Yamazoe, H., Utsumi, A., Yonezawa, T., and Abe, S.

(2008). Remote gaze estimation with a single camera

based on facial-feature tracking without special cali-

bration actions. In Proceedings of the 2008 Sympo-

sium on Eye Tracking Research & Applications, pages

245–250. ACM.

EyeRec: An Open-source Data Acquisition Software for Head-mounted Eye-tracking

391