High-Speed and Robust Monocular Tracking

Henning Tjaden

, Ulrich Schwanecke

, Fr

eric A. Stein

and Elmar Sch

omer

Computer Science Department, RheinMain University of Applied Science, Unter den Eichen 5, Wiesbaden, Germany

Institute of Computer Science, Johannes-Gutenberg University, Staudingerweg 9, Mainz, Germany

Keywords:

Optical Tracking, Monocular Pose Estimation System, Infrared LED, Camera, High-Speed, Calibration.

Abstract:

In this paper, we present a system for high-speed robust monocular tracking (HSRM-Tracking) of active

markers. The proposed algorithm robustly and accurately tracks multiple markers at full framerate of current

high-speed cameras. For this, we have developed a novel, nearly co-planar marker pattern that can be identiﬁed

without initialization or incremental tracking. The pattern also encodes a unique ID to identify different

markers. The individual markers are calibrated semi-automatically, thus no time-consuming and error-prone

manual measurement is needed. Finally we show that the minimal spatial structure of the marker can be used

to robustly avoid pose ambiguities even at large distances to the camera. This allows us to measure the pose

of each individual marker with high accuracy in a vast area.

1 INTRODUCTION AND

BACKGROUND

Tracking of moving objects is a very important aspect

in many ﬁelds of application such as robotics, auto-

motive, sports, health, or virtual reality. Often it is

the basis to automatically supervise, classify and op-

timize motion sequences such as in athletic training

or medical therapy (Fitzgerald et al., 2007; Vito et al.,

2014). Other applications can be found in industrial

assembly, where optical tracking can be used to assist

humans or robots in order to increase their productiv-

ity and reliability or enables them to work more au-

tonomously (Ong and Nee, 2004; Zetu et al., 2000).

Over the past decades, a large number of different

solutions have been developed to track the position

and orientation of objects based on various technolo-

gies such as ultrasound, magnetism, inertial, or opti-

cal sensors (Welch and Foxlin, 2002). Most widely

used are probably the optical tracking systems which

can be divided into active and passive systems. Pas-

sive systems use just the visible light to detect high

contrast artiﬁcial ﬁducials or natural structures. Ac-

tive systems use additional light sources to facilitate

the detection of specially designed ﬁducials. Thereby,

they usually work with infrared light to minimize the

inﬂuence of ambient lighting.

Active optical tracking systems can be divided fur-

ther into systems where the additional light source

is attached to the camera – also called active cam-

era systems – and systems with light emitting devices

such as light emitting diodes (LEDs) attached to the

ﬁducials – also called active tracker systems. While

active camera systems utilize very lightweight and

keen targets, active tracker systems usually provide

a larger total working volume and higher precision.

Optical tracking systems often determine the

three-dimensional position of a ball-like ﬁducial

based on two or more camera images and triangula-

tion. While this can be done very efﬁciently utilizing

epipolar constraints, it requires the ﬁducials always

to be seen by (at least) two cameras. In some applica-

tions, this may not always be guaranteed and a system

based on the image of just a single monocular camera

is needed. In this work we therefore focus on tracking

with a single monocular optical camera.

In its simplest form, optical tracking of the pose of

a rigid object with a single camera is based on passive

planar markers (Olson, 2011; Herout et al., 2013).

The advantage of these passive systems is their low

complexity. Users can simply print or even draw their

own markers. But, even though the latest methods

achieve higher robustness against fast movement by

tracking arbitrary patterns based on feature descrip-

tors (Wagner et al., 2008; Ozuysal et al., 2010), or

using a parametrized model of the object under inves-

tigation (Schmaltz et al., 2012; Prisacariu and Reid,

2012), passive systems are still limited in distance

range and stability under low lighting conditions or

lack in terms of precision or high-speed performance.

462

Tjaden H., Schwanecke U., Stein F. and Schömer E..

High-Speed and Robust Monocular Tracking.

DOI: 10.5220/0005267104620471

In Proceedings of the 10th International Conference on Computer Vision Theory and Applications (VISAPP-2015), pages 462-471

ISBN: 978-989-758-091-8

 2015 SCITEPRESS (Science and Technology Publications, Lda.)

Figure 1: The proposed markers consist of seven LEDs ar-

ranged in a cross-shaped pattern, where M

deﬁne

the right handed marker coordinate frame.

Active tracking systems overcome most of the

previously mentioned limitations as they are less de-

pendent on the given lighting conditions. One of

the main tasks of active systems is the identiﬁca-

tion of the individual lights or retro-reﬂective tags

each marker is composed of. (Naimark and Foxlin,

2005) e.g. presented a technique for encoding indi-

vidual LED markers by amplitude modulation, used

in an initialization step. After initialization the LEDs

are tracked incrementally from frame to frame. Such

techniques come with the downside of potential track-

ing losses, after which the system would have to get

re-initialized.

Monocular tracking of purely co-planar markers

suffers from pose ambiguities, while markers with

spatial structure are likely to occlude themselves.

(Faessler et al., 2014) recently presented a system,

that can track a single marker composed of four or

ﬁve LEDs at 90 Hz. Although these LEDs can be ar-

ranged arbitrarily, they should span a volume as large

as possible to avoid pose ambiguities. The resulting

self-occlusions are dealt with a time consuming com-

binatorial brute force approach, that is used to ini-

tialize and re-initialize the LED identiﬁcation. Once

initialized the LEDs are tracked incrementally in con-

junction with motion prediction. The spatial positions

of the marker LEDs are calibrated with a commercial

multi-camera motion capturing system.

In this paper we present a high-speed and robust

monocular (HSRM) tracking system that is superior

to all other present systems regarding accuracy, track-

ing range and robustness, while still being low-cost on

the hardware side. The system can estimate the pose

of multiple markers, each of them composed of seven

infrared LEDs, at frequencies far over 500 Hz, allow-

ing robust tracking even of fast-moving objects. Our

three main contributions are: First, a method to iden-

tify each individual LED of a marker solely based on

2D geometrical constraints not requiring any initial-

ization or frame-to-frame tracking. Second, an empir-

ical proof that the minimal 3D structure of our nearly

co-planar marker is an optimal tradeoff between the

occurrence of self-occlusions and the avoidance of

pose ambiguities. Third, a semi-automatic marker

calibration algorithm avoiding time-consuming and

error-prone manual measurements and ensuring accu-

rate tracking results.

2 MONOCULAR MARKER

TRACKING

In the following we describe our marker pattern and

its identiﬁcation as well as the tracking of multiple

markers in detail. We also show that our approach

is robust to measurement noise and pose ambiguities

and present an algorithm to semi-automatically cali-

brate our markers with high accuracy.

2.1 Preliminaries

The proposed markers consist of seven infrared LEDs

connected to a small battery pack (Figure 1). For our

prototype we chose SMD (surface-mounted device)

LEDs which are perceptible from large viewing an-

gles up to nearly π/2. We tested our system with a

variety of monochrome USB 3.0 high-speed cameras,

operating from 90 Hz to 500 Hz with resolutions up to

2048×2048 pixels. The cameras have pre-calibrated

intrinsics and are equipped with an infrared ﬁlter to

reduce the inﬂuence of ambient light.

The marker LEDs are arranged in a constrained

geometrical cross-shaped pattern. The spatial LED

positions are denoted relative to the marker coordi-

nate frame and are referred to as M

= (x

)

,i =

0,.. .,6. Their corresponding 2D projections into

the camera image are denoted by m

= (x

)

,i =

0,.. .,6. LED M

= (0,0, 0)

deﬁnes the origin of

the right handed marker coordinate frame, while the

vectors

−−−−→

and

−−−−→

deﬁne the corresponding

x- and y-axis, respectively. M

,.. .,M

lie in the xy-

plane and M

is slightly elevated for stabilization pur-

poses as will be explained in section 2.2.3. In sec-

tion 3 we show, that the exact spatial positions of all

LEDs can be measured automatically (up to a scaling

factor) in a pre-calibration step.

We refer to the pose of a marker with respect to

the camera coordinate frame as P

= [R

], with

∈SO(3) describing the rotation of the marker and

∈R

being the location of the origin of the marker

coordinate frame in the camera coordinate frame. Per-

spective projection of a 3D point M including deho-

mogenisation is denoted by π(M) = (x/z,y/z)

High-SpeedandRobustMonocularTracking

463

(a) Case L5.

(b) Case L41.

(d) Case L42b.

Figure 2: Raw input video frames of a marker under various perspective transformations showing the different conﬁgurations

distinguished during correspondence assignment. The dominant lines l

and l

are drawn and their participants are framed by

a dashed line. (a) An example where ﬁve LED projections lie on l

. (b) An example where l

has four participants. (c) An

example where two dominant lines appear in the projection pattern. In this case m

lies between m

and m

on l

. (d) The

second case where two dominant lines appear. In this case m

lies outside of m

and m

on l

2.2 Tracking Method

The pose of each marker is determined based on the

known 3D positions M

of the seven LEDs and their

corresponding 2D projections m

. Therefore, in a ﬁrst

step we segment the projected LEDs in each video

frame and determine their exact 2D positions as ex-

plained in detail in section 2.2.1. After the 2D pro-

jections are segmented they have to be assigned to

the corresponding LEDs. Since all LED projections

look alike they cannot be distinguished by simple lo-

cal analysis. In section 2.2.2 we show how to deter-

mine the correct correspondences, while section 2.2.3

explains how to estimation the pose of the markers.

Finally, in section 2.2.4 the extension to multi-marker

tracking is presented.

2.2.1 LED Segmentation and 2D Localization

The LEDs will appear as bright blobs in the video

frames. These regions are saturated to a great extent

and therefore are signiﬁcantly brighter than the rest

of the scene. To reduce the inﬂuence of ambient light,

we utilize an infrared ﬁlter, so that almost only the

LEDs remain visible in the camera image. To segment

the blobs, we use a simple binary thresholding, keep-

ing only pixels with an intensity above a prescribed

threshold s. The remaining pixels are grouped into

connected regions Ω

using a union-ﬁnd algorithm.

In a next step we determine the 2D positions



10i

00i

01i

00i



(1)

of the LED projections by calculating the intensity

centroid of each region Ω

based on the moments

pqi

∑

(x,y)∈Ω

I(x,y)

. (2)

Thereby, we used a quadratic weighting of the pixel

intensities I(x, y) to reduce the inﬂuence of border

pixels, that tend to ﬂicker due to their lower intensities

compared to pixels near the centroid. Experiments

showed that (2) reduces jitter of the measurements

similarly to applying a low-pass ﬁlter (e.g. Gaussian),

while being computationally less expensive.

In a last step all determined LED projections m

are normalized and undistorted to ideal image coordi-

nates m

using the pre-calibrated camera intrinsics.

2.2.2 Determining the 2D/3D Correspondences

Our constrained cross-shaped marker pattern allows

to determine the correspondences between 3D LED

position M

and ideal image coordinate m

for each

individual frame. Thereby, the only restriction is that

all seven LEDs of a marker have to be visible in the

respective frame.

Because m

,.. .,m

always must lie on a straight

line, we start by investigating the dominant lines in

the set of projections M = {m

,.. .,m

}. The ﬁrst

dominant line l

can either have four participants

(case L4, see (Fig. 2(b)), ﬁve participants (case L5,

see Fig. 2(a)) or more than ﬁve participants. The lat-

ter case only occurs under very shallow viewing an-

gles with low measuring accuracy and will therefore

not be further considered.

VISAPP2015-InternationalConferenceonComputerVisionTheoryandApplications

464

Due to the structure of the marker there exist per-

spective transformations where m

together with m

and m

forms a second dominant line l

(see

Figs. 2(c) and 2(d)). Thus, case L4 is further sepa-

rated into L41 (one line) and case L42 (two lines).

These two cases can be differentiated based on re-

gression lines to all thirty-ﬁve subsets of four points

from M . Therefore, the lines and their correspond-

ing points are grouped by quality and angle, where

quality is determined by the distance of the furthest

outlier to the line. The cases L5, L41 and L42 now

can be distinguished as follows, based on simple 2D

geometric criteria:

L5: The two points that do not belong to l

deﬁne

a second line l

. The closest point to the inter-

section of l

and l

must be m

. The point that

lies solely on one side of l

is m

. The points

are assigned descendingly according

to their absolute distance to l

. Finally m

and m

are assigned based on their signed distance to the

line l

given by m

L41: The point belonging to l

furthest from the cen-

troid of the three points not lying on l

must be

. Next, m

and m

can be assigned ascend-

ingly according to their distance to m

. One of

the three remaining LEDs (either m

or m

) will

lie solely on one side of l

. It can be detected and

assigned by calculating the three signed distances

to l

. Assuming it is m

, we construct line l

given by m

and assign m

and m

ascend-

ingly to their distance to it.

L42: We ﬁrst identify l

by the fact that all points in-

cident to l

lie on the same side of line l

, except

which is the intersection of l

and l

. Next,

we can enumerate m

, m

and m

in descending

order according to their distance to m

. Now the

LED that lies solely on one side of l

is either m

or m

. Thereby the correct assignment is deter-

mined by the signed distance to l

as in L41. As-

suming that this assignment was m

, there are two

possibilities (L42a and L42b) to assign m

and

, which can not be robustly distinguished by

2D criteria (see Figs. 2(c) and 2(d)). We therefore

solve this ambiguity with regard to the 3D struc-

ture of the marker. This is done by calculating the

two homographies H

and H

for both possible

assignments from all six coplanar LEDs. These

homographies allow us to calculate two pose esti-

mations (Xu et al., 2009) called P

H46

and P

H64

To determine the correct assignment, we calcu-

late the average reprojection errors (see eqn. (3)

in section 2.2.3) for both pose estimations using

all seven LEDs and choose the assignment com-

bination that yields the smaller error.

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

0.09

0.1

-150 -100 -50 0 50 100 150

Reprojection error

Rotation around the x-axis [deg]

proj

√

pointer

Figure 3: A synthetic experiment, demonstrating the effec-

tiveness of our pointer LED. We rotated the model points

around the x-axis and measure the reprojection errors e

proj

and e

pointer

to a chosen ground truth angle α = 60

◦

with

= (0,0,10)

in the image plane. Our model consists

of ﬁve points: M

= (1,1,0)

, M

= (1,−1,0)

, M

(−1,−1,0)

, M

= (−1, 1, 0)

, M

pointer

= (0, 0, −0.1)

For further explanation please refer to (Schweighofer and

Pinz, 2006). We plot

√

pointer

for visualizing purposes.

2.2.3 Pose Estimation

Once all seven correspondences are known, the pose

= [R

] of the marker in the associated frame

can be determined. Thereby a ﬁrst estimate P

is cal-

culated based on the homography H given by the six

coplanar correspondences (M

),i = 0,.. .,5 (Xu

et al., 2009). Next, P

is reﬁned by iteratively mini-

mizing the reprojection error in the image space

proj

(P,n) =

∑

i=0

−π(RM

+ t)k

. (3)

Using a Levenberg-Marquardt solver (Levenberg,

1944; Marquardt, 1963) this results in the pose P

argmine

proj

(P,6). There exist ambiguities when esti-

mating the pose from coplanar markers, which cause

the pose to ﬂip at large distances (Schweighofer and

Pinz, 2006). This is because in general the reprojec-

tion error e

proj

(P,n) has two distinct local minima (see

Fig. 3). Following (Schweighofer and Pinz, 2006) we

take P

and calculate the corresponding second pose

belonging to the other minimum of e

proj

(P,5) using

only the six coplanar correspondences. Schweighofer

and Pinz then choose the correct pose by comparing

the reprojection errors for both solutions, assuming

that the desired pose will yield a smaller error. Unfor-

tunately, in practice this approach only yields the cor-

rect solution with a probability of about 0.75 to 0.95,

depending on the angle between the optical axis and

the z-axis of the marker and its distance to the camera.

Inspired by (Yang et al., 2012) we choose the correct

solution by utilizing the non coplanar LED M

as a

pointer to the correct pose. Thereby, the reprojection

error of the non coplanar point M

pointer

projected on

High-SpeedandRobustMonocularTracking

465

the image point m

pointer

yields a second error function

pointer

(P) = km

pointer

−π(RM

pointer

+ t)k

(4)

that is robust to measurement noise and suitable to

distinguish the two poses (see Fig. 3). Synthetic ex-

periments showed that e

pointer

(P) and e

proj

(P) share

the sought minimum while being contrary at the delu-

sive minimum of e

proj

(P). Thus, a robust pose esti-

mation is given by

= argmin

P∈{P

}

pointer

(P). (5)

Reﬁning the pose P

using all seven correspon-

dences results in the ﬁnal pose estimation P

argmin e

proj

(P,6).

2.2.4 Multi-marker Tracking

Our tracking algorithm can easily be extended to

multi-marker tracking. The image processing steps

are exactly the same as described in section 2.2.1 until

the centeroids of all LED projections are determined.

We then use the k-means algorithm (MacKay, 2002)

to cluster the projections with an adapted initializa-

tion scheme. Thereby we set k = dn/7e, where n is

the number of visible LEDs in the respective frame.

So, false positive detections e.g. caused by reﬂec-

tions of sunlight or markers that are not fully visible,

in many cases will end up in a separate smaller clus-

ter and can easily be ﬁltered. In order to speed up and

assure correct convergence of k-means, we initialize

the cluster centers b

by exploiting the constraint that

each marker cluster must have exactly seven mem-

bers. This is done by repeating the following proce-

dure k times, i.e. for i = 0, ..., k −1 do

1. Randomly choose a point m

from all unassigned

LED projections.

2. Assign the six (or less, if not more are unassigned)

closest to m

LED projections to it.

3. Set the location of the current center b

to the

barycenter of m

and its assigned LED projec-

tions.

After this initialization k-means usually only needs

one or two iterations for convergence.

For each cluster (marker) the 3D-2D correspon-

dences can be determined independently as described

in section 2.2.2. We found that, although k-means

is a rather simple algorithmic choice for separating

the markers and likely to fail if the projections of the

markers are too close together, it still works ﬁne for

many application scenarios. For example when track-

ing a human arm and there are two markers attached

to the upper and the lower arm, critical conﬁgurations

are unlikely to occur.

The identiﬁcation of each individual marker (de-

termined by the clusters) is based on the cross-ratio

CR(M

−M

k·kM

−M

k·kM

−M

(6)

of the collinear LEDs M

,.. .,M

. By varying the

positions of M

, M

along the line given by M

, M

new unique marker IDs can be constructed. All mark-

ers to track have to be calibrated beforehand, so that

the correspondences between LED projection clusters

and markers can be found by simply comparing the

cross-ratio CR(m

) of each cluster with

the cross-ratio of each marker.

3 MARKER CALIBRATION

Measuring the positions of the LEDs of a marker

manually is error-prone, inconvenient and should

therefore be reduced to a minimum. Automatic mea-

suring of the spatial LED positions is convenient,

achieves maximum tracking accuracy and can be done

by only measuring the distance d

between M

and

manually as shown next. Thereby, d

determines

the overall scale of the corresponding marker.

Our calibration algorithm is based on a sequence

of n + 1 image frames I

, j = 0, ..., n recorded

while translating and rotating the marker arbitrarily

in front of the camera at a rather short distance. For

our experiments we record 500 frames within a pe-

riod of 5 seconds, i.e. one frame each 10ms. For each

individual frame I

we then extract the set of normal-

ized and assigned LED projections M

. Since the 3D

structure of the marker is not yet known at this point,

perspectives yielding case L42 (section 2.2.2) cannot

be assigned correctly. These usually rarely occurring

frames are omitted automatically. The complete cali-

bration scheme is split into an initialization and a re-

ﬁnement step, as explained in the following.

3.1 Initialization

As illustrated in algorithm 3.1 we ﬁrst select a coarse

subset M containing only every o-th projection set

. In our experiments we use a subsample offset

o = 50, i.e. the respective frames were recorded with

an offset of 500 ms which promotes wide baselines.

In the next step we select all pairs (M

)

∈ M ×

M,k < k

and calculate the relative camera pose P

C,l

from M

to M

using the ﬁve-point algorithm (Nis-

ter, 2004). For each pair we then determine a set of

spatial positions M

i,l

via linear triangulation relative

to P

C,l

. To merge these sets of position estimates, they

VISAPP2015-InternationalConferenceonComputerVisionTheoryandApplications

466

need to be transformed into a common marker coor-

dinate system. This is done by applying a similarity

transformation T

that enforces

Algorithm 3.1: Marker Calibration.

Data: d

, M

, j = 0,. ..,n, subsample offset o

Result: M

, i = 0,. ..,6

1 /* Initialization */

2 M ←{M

j·o

| j = 0,. ..,

/o};

3 foreach (M

)

∈ M ×M,k < k

4 P

C,l

← rel. camera pose from M

to M

;

5 Triangulate M

i,l

using P

C,l

, i = 0,.. .,6;

6 M

i,l

← T

i,l

with sim. T

, i = 0,.. .,6;

7 end

8 M

←



med



, i = 0,.. .,6;

9 /* Refinement */

10 R ← {M

};

11 repeat

12 foreach projection set M

13 Calculate P

M,j

; // section 2.2.3

14 P

C, j

← [R

M,j

|−R

M,j

];

15 foreach m

i, j

∈ M

\{m

0, j

3, j

} do

16 Fill system A

= b

using P

C, j

;

17 end

18 end

19 e ← 0;

20 foreach M

∈ R do

21 Solve A

= b

;

22 e ← e + kM

−x

23 M

← x

;

24 end

25 until ∆e < ε;

3,l









, T

0,l









5,l









. (7)

Once transformed into the marker coordinate system,

we calculate the median values x

med

, y

med

and z

med

all M

i,l

for each dimension independently. The ﬁnal

position estimates of the initialization are then given

by the points M



med



3.2 Reﬁnement

In order the increase the accuracy and reliability of

our tracking system the spatial position estimates

coming from the initialization step can be further im-

proved by a subsequent reﬁnement step. Experiments

showed that the standard deviation of repeated ini-

tial calibration increases with the distance between

marker and camera. In order to overcome this prob-

lem and ensure exact calibration results, we apply an

iterative reﬁnement step that robustly converges to the

desired result.

The basic idea of the reﬁnement step (see Algo-

rithm 3.1) is to interleave and decouple the reﬁnement

of the camera poses and the reﬁnement of the spatial

LED positions, similar to (Lakemond et al., 2013).

During reﬁnement, each camera and each LED posi-

tion is treated independently. We use all of the previ-

ously recorded projection sets M

, j = 0, ...,n for the

reﬁnement step. The overall scaling is preserved by

using M

= (0,0,0)

and M

= (d

,0,0)

as ﬁxed

points that remain unaffected.

In each iteration we ﬁrst calculate all marker poses

M,j

with the method described in section 2.2.3 us-

ing every M

and the current M

. Thereby the cor-

responding camera poses can be derived as P

C, j

M,j

|−R

M,j

] = [R

C, j

]. Afterwards, all these

updated camera poses are used to re-triangulate the

LEDs. We therefore determine the intersection points

of the projection rays from each camera center c

through each m

i, j

such that

−x

) ×v

i, j

= 0 (8)

with v

i, j

= R

C, j

i, j

,1]

. For each LED this can

be formulated as a least squares problem. Thereby,

based on eqn. (8) a linear system A

= b

has to

be solved. The overall error per iteration is given by

e =

∑

−x

k. The algorithm stops if the change

∆e of the error is smaller than a prescribed bound ε.

In our experiments we set ε = 0.0001.

4 EVALUATION

In this section we demonstrate the capabilities and

limitations of HSRM-Tracking. We start by present-

ing a brief runtime performance analysis followed by

a detailed discussion of the reliability and repeata-

bility of the automatic calibration method as well

as the measurement accuracy of our pose estimation

method. We conclude our evaluation by demonstrat-

ing that the robustness to pose ambiguities and rapid

motion of our method outperforms any other present

monocular tracking system.

4.1 Performance Analysis

For our experiments we used a two megapixel

USB 3.0 camera

with a ﬁxed-focus 9 mm lens and

the prototype marker seen in Figure 1. The quan-

tity d

= 114.2 mm, i.e the scale of the marker was

measured manually using a caliper. The automatically

XIMEA xiQ MQ022MG-CM. See: www.ximea.com

High-SpeedandRobustMonocularTracking

467

calibrated spatial positions of its LEDs are shown in

Table 1. The calibration thus yields a cross-ratio of

CR(M

) ≈ 3.99 for the marker.

Table 1: Spatial LED positions and standard deviations of

the calibration process for our prototype marker.

[mm] M

x 75.91 ±.02 37.91 ±.01 −0.14 ±.05 0.28 ±.05 −38.29 ±.03

y −0.12±.01 −0.04 ±.01 −37.97 ±.07 38.15 ±.08 0.4 ±.01

z 0.14 ±.01 0.24 ±.01 0.33 ±.03 −0.03 ±.02 −11.21 ±.03

We tested a C++ implementation of our system on

a commodity quad core CPU @ 2.6 GHz. Thereby

the whole tracking process is performed in a sin-

gle thread. Figure 4 shows a plot of the computa-

tion times subdivided into the combined 2D image

processing steps and the marker pose estimation in-

cluding correspondence assignment. With an aver-

0.2

0.4

0.6

0.8

0 100 200 300 400 500

Time [ms]

Frame

Overall

Correspondences and 6DOF pose estimation

2D image processing

Figure 4: Computation times of the optical tracking method.

All measurements were averaged over 100 runs using a pre-

recorded sequence of 500 frames at full 2048×1088 resolu-

tion. In this sequence a single marker was held in hand and

translated and rotated arbitrarily in front of the camera.

age computation time of about 1 ms per frame we

can easily track the pose at full framerate (170 fps)

of our camera even with multiple markers. This en-

ables tracking a single marker at frequencies of more

than 1000 Hz given a suitable camera. Each addi-

tional marker would only increase the runtime by ap-

proximately 0.6 ms if not processed in parallel.

Table 2: Timings of the image processing step.

[px] 640 ×480 1280 ×1024 2048 ×1088 2048 ×2048

[ms] ≈ 0.1 ≈ 0.3 ≈ 0.4 ≈ 0.9

The image processing mostly depends on camera

resolution (see Table 2) and is nearly independent of

the number of visible markers. Our implementation is

optimized using SSE2 instructions, especially speed-

ing up the LED region grouping step.

4.2 Calibration Reliability

The reliability of our marker calibration method can

be determined by its repeatability when calibrating a

marker multiple times. Since the spatial positions of

the LEDs remain the same for each calibration, the

standard deviation of the calibrated spatial positions

should ideally be zero.

In practice the spatial positions after the initial-

ization are already accurate to several tenths of a

millimeter. For experimental evaluation of our re-

ﬁnement method we have compared it to an estab-

lished implementation of full sparse bundle adjust-

ment (SBA) (Lourakis and Argyros, 2009). We can

show that our method reduces the standard deviation

of the calibration initialization about an order of mag-

nitude while SBA only reduces it by a factor of 2 to

3. To further analyze this difference, we have con-

ducted experiments where we randomly distorted the

initial estimates of M

by ±5 mm for each calibration.

Our reﬁnement method was still able to robustly re-

duce the standard deviation to several hundredths of a

millimeter while SBA varies about an order of mag-

nitude more. This shows the reliability of our reﬁne-

ment method even for poor initial estimations.

4.3 Measurement Accuracy

We ﬁrst analyze the translational accuracy of our pose

estimation. Thereby, we are particularly interested in

the depth accuracy since this is the most critical part

in monocular tracking. We ﬁxed the camera and the

marker on a straight rail of 2 meters length and set up

a lasermeter

next to the camera casting its beam onto

the marker. The camera was ﬁxed at one end of the

rail and the marker was oriented parallel to the im-

age plane. Starting at 400 mm distance we moved the

marker in 100 mm steps according to the lasermeter

towards the other end of the rail. At each position p

we recorded 500 optical measurements and calculated

the average marker translation vector

M,p

as the mean

of all measurements. Afterwards, we calculated the

measured relative translation as d

= k

M,p

−

M,0

for every position and compared it to the lasermeter

as shown in Figure 5(a). We also analyzed the stan-

dard deviations of the translation parameters t

, t

and

and the orientation parameters R

, R

and R

(the

Euler angles around the x-, y- and z-axis) across the

500 samples. Their growth in relation to the distance

to the camera is shown in Figure 5(b).

The results show that our system is able to mea-

sure the position of the marker with an accuracy of

about ±1 mm even at a distance of almost 2 meters.

Unfortunately the lasermeter we used only can mea-

sure with an accuracy of ±1.5 mm. Note that the

Precaster Enterprises HANS CA770 laser meter. See:

http://www.precaster.com.tw

VISAPP2015-InternationalConferenceonComputerVisionTheoryandApplications

468

-2.5

-2

-1.5

-1

-0.5

0.5

1.5

0 200 400 600 800 1000 1200 1400

Difference [mm]

Relative translation according to the lasermeter [mm]

Optical measurement

Lasermeter

(a) Comparison between the lasermeter and the optical measurement.

0.001

0.01

0.1

400 600 800 1000 1200 1400 1600 1800

Standard deviation [mm]/[deg]

Distance to the camera [mm]

(b) Standard deviations of the measured 6DOF of the marker.

Figure 5: Results of our relative position accuracy measurements from about 400 mm to about 1900 mm distance to the

camera/lasermeter. (a) The plot shows the difference from the optical measurements to the lasermeter. The error bars are due

to the lasermeter. (b) Standard deviations of marker pose parameters in relation to the distance to the camera.

depth accuracy is strongly dependent on the quality

of the intrinsic camera calibration.

-1

-0.5

0.5

-60 -40 -20 0 20 40 60 80

Difference [deg]

Relative rotation angle according to the goniometer [deg]

Rotation around the x-axis

Rotation around the y-axis

Goniometer

Figure 6: Results of our relative rotation accuracy measure-

ments. The plot shows the difference from the optical mea-

surements to a goniometer.

Next we analyze the rotational accuracy of the

pose estimation. For this, the marker was ﬁxed at a

distance of 1 meter to the camera and attached to a

goniometer. We rotated the marker around its x-axis

from −70

◦

up to 70

◦

and around the y-axis from −60

◦

up to 80

◦

in 10

◦

steps. These angle boundaries are due

to self occlusions of the LED pattern that occur in our

particular experimental setup.

We again recorded 500 measurements at each an-

gle α and compared the relative angle differences. For

this, we converted the marker rotation matrix R

M,α

into the corresponding unit quaternion q

M,α

and cal-

culated the average

M,α

. We then determined the

measured relative rotation as the angle of

M,α

−1

M,0

for every following orientation and compared it to the

goniometer as shown in Figure 6. Our results show

that the rotational measurement error is below ±1

◦

4.4 Robustness to Pose Ambiguities

In this experiment we demonstrate the robustness of

the proposed method to pose ambiguities at large dis-

tances to the camera that monocular tracking systems

usually suffer from. For this we ﬁxed the camera on

a static tripod and attached the marker to a wheeled

tripod. Starting at about 50 cm distance to the camera

we recorded a sequence of 2290 frames while manu-

ally moving the marker away up to 7.5 m. Based on

this pre-recorded sequence we monitored the number

of pose ﬂips (Figure 7) using our proposed method

and compared it to the following three other methods:

Method 1: is the simplest approach estimating the

pose by minimizing the reprojection error of only

the six coplanar correspondences i.e.

= P

= argmine

proj

(P,5) (9)

without considering other solutions.

Method 2: also only considers the six coplanar cor-

respondences but makes use of the improvement

of (Schweighofer and Pinz, 2006) by choosing

= argmin

P∈{P

}

proj

(P,5), (10)

based on the cumulated projection error of all six

correspondences for both solutions.

Method 3: is similar to method 1 but uses all 7 (non-

coplanar) correspondences to determine

= P

= argmine

proj

(P,6), (11)

considering the 3D structure of the marker.

All methods are based on P

, the pose roughly esti-

mated from homography.

The results show that method 1 performs worst

with the most pose ﬂip occurrences in total, starting

at about 2 meters distance to the camera. Method 2

demonstrates how the strategy of Schweighofer and

Prinz reduces the number of pose ﬂips when using

a purely coplanar marker. Results show that in this

case the ﬂips ﬁrst occur at a distance of about 3 me-

ters to the camera and that their frequency rises with

increasing distance. Although method 3 performs bet-

ter than method 2 regarding the total number of ﬂips,

High-SpeedandRobustMonocularTracking

469

-10

[deg]

-10

0 500 1000 1500 2000 2500 3000 3500 4000 4500 5000 5500 6000 6500 7000 7500

Distance to the camera [mm]

Method 1: 532 ﬂips

Method 2: 459 ﬂips

Method 3: 426 ﬂips

Proposed: 0 ﬂips

Figure 7: Results of our pose ﬂip comparison experiment. We plot the R

in relation to t

because it was inﬂuenced the most

of all six pose parameters by the ﬂips in our setup. The values are plotted in chronological order from frame 0 to frame 2289.

Negative values of R

indicate the occurrence of a pose ﬂip.

(a) Frisbee with the

marker.

M,0

M,153

(b) A 3D visualization of the tracked trajectory rendered from the cameras perspective.

Figure 8: Results of our rapid motion tracking experiment. (a) The frisbee that was used with the attached prototype marker.

(b) A visualization of the recorded trajectory where we draw the coordinate axes of the marker at each tracked location. The

x-, y- and z-axis of the marker are visualized red, green and blue and their length is equal to the radius of the frisbee (140 mm).

the ﬁrst ﬂips already occur about 0.5 meters closer to

the camera. It can also be seen that the marker pose

is estimated further away from the camera due to the

elevation of M

when the minimization ends up in the

false minimum.

Our proposed method did not ﬂip once in the

whole sequence, which demonstrates the effective-

ness of the pointer LED even at large distances de-

spite its relatively small elevation. In all our exper-

iments we did not manage to cause pose ﬂips in the

range where the LEDs were sufﬁciently visible to the

camera to determine a pose.

4.5 Robustness to Rapid Motion

In this last experiment we attached the prototype

marker to a frisbee (see Figure 8(a)) in order to

demonstrate the capabilities of our system to robustly

capture rapid motion. The exposure time of the cam-

era was set to 2 ms for this experiment. Although

this hardware setup would not be feasible for a re-

alistic sports analysis of frisbee throws (at least not

in this form), it is still a challenging example of com-

bined rapid translational and rotational movement of a

rigid object. We captured and tracked a 0.955 seconds

long disc throw ﬁlmed from diagonal above the scene.

The sequence consists of 154 frames that were all

successfully tracked (see Figure 8(b)). The captured

trajectory

started at t

M,0

= (2577.4, 60.3,4564.8)

and ended at t

M,153

= (−2896.8,−660.8,7093.4)

including nine full turns. Accordingly the frisbee

traveled a total distance of 6.073 meters at an aver-

age speed of approximately 23 km/h. Note that also

in this experiment no pose ﬂips occured.

4.6 Failure Modes

Each marker can only be tracked if all seven LEDs are

visible and separable in the respective frame. Hence

self-occlusions caused by M

lead to tracking fail-

ures. Filtering multiple false positive LED detections

fails, if e.g. they appear at different sides around a

marker and therefore do not end up in a single sepa-

rate cluster. When tracking multiple markers, track-

ing will fail if their projected LED patterns are too

close or even overlap due to the nature of the em-

ployed k-means algorithm.

All measurements are in mm.

VISAPP2015-InternationalConferenceonComputerVisionTheoryandApplications

470

5 CONCLUSION AND FUTURE

WORK

In this paper, we presented HSRM-Tracking, a

method for robustly estimating and tracking the pose

of multiple infrared markers with a single monocu-

lar camera. Thereby, individual markers can be cor-

rectly recognized in each single camera frame and

distinguished based on the cross ratio of four collinear

LEDs. Our evaluation results show that HSRM-

Tracking is able to precisely capture ﬁne and rapid

movement up to 1000 Hz in a large area neglecting

bandwidth limitations of current cameras.

The proposed method could easily be adapted for

use in a multi-camera system where each camera runs

in parallel in a separate tracking thread. Thus, cam-

eras with different frame rates could be combined to

track the markers asynchronously and contribute to

a synchronized result whenever a new measurement

is available, making camera synchronization unnec-

essary. Being able to estimate the marker pose from a

single camera would also vastly increase the track-

ing volume of a multi camera setup and could be

used in conjunction with stereo methods, whenever

the marker is visible in more than one camera. Such a

setup would also beneﬁt from the LED identiﬁcation

scheme, since the markers could be used in order to

dynamically calibrate the multi-camera system with-

out having to solve stereo correspondence problems.

REFERENCES

Faessler, M., Mueggler, E., Schwabe, K., and Scaramuzza,

D. (2014). A monocular pose estimation system based

on infrared LEDs. In IEEE International Conference

on Robotics and Automation (ICRA).

Fitzgerald, D., Foody, J., Kelly, D., Ward, T., Markham, C.,

McDonald, J., and Caulﬁeld, B. (2007). Development

of a wearable motion capture suit and virtual reality

biofeedback system for the instruction and analysis of

sports rehabilitation exercises. In EMBS 2007, pages

4870–4874.

Herout, A., Szentandrasi, I., Zacharia, M., Dubska, M., and

Kajan, R. (2013). Five shades of grey for fast and

reliable camera pose estimation. In IEEE Conference

on Computer Vision and Pat. Rec., pages 1384–1390.

Lakemond, R., Fookes, C., and Sridharan, S. (2013).

Resection-intersection bundle adjustment revisited.

ISRN Machine Vision, 2013:8.

Levenberg, K. (1944). A method for the solution of cer-

tain non-linear problems in least squares. Quarterly

Journal of Applied Mathmatics, II(2):164–168.

Lourakis, M. A. and Argyros, A. (2009). SBA: A Software

Package for Generic Sparse Bundle Adjustment. ACM

Trans. Math. Software, 36(1):1–30.

MacKay, D. J. C. (2002). Information Theory, Inference

& Learning Algorithms. Cambridge University Press,

New York, NY, USA.

Marquardt, D. W. (1963). An algorithm for least-squares

estimation of nonlinear parameters. SIAM Journal on

Applied Mathematics, 11(2):431–441.

Naimark, L. and Foxlin, E. (2005). Encoded led system for

optical trackers. In Proceedings of the 4th IEEE/ACM

International Symposium on Mixed and Augmented

Reality, pages 150–153.

Nister, D. (2004). An efﬁcient solution to the ﬁve-point

relative pose problem. IEEE Transactions on Pattern

Analysis and Machine Intelligence, 26(6):756–770.

Olson, E. (2011). AprilTag: A robust and ﬂexible vi-

sual ﬁducial system. In Proceedings of the IEEE In-

ternational Conference on Robotics and Automation

(ICRA), pages 3400–3407. IEEE.

Ong, S. K. and Nee, A. (2004). Virtual Reality and

Augmented Reality Applications in Manufacturing.

Springer Verlag.

Ozuysal, M., Calonder, M., Lepetit, V., and Fua, P. (2010).

Fast keypoint recognition using random ferns. Pat-

tern Analysis and Machine Intelligence, IEEE Trans-

actions on, 32(3):448–461.

Prisacariu, V. and Reid, I. (2012). Pwp3d: Real-time seg-

mentation and tracking of 3d objects. International

Journal of Computer Vision, 98(3):335–354.

Schmaltz, C., Rosenhahn, B., Brox, T., and Weickert, J.

(2012). Region-based pose tracking with occlusions

using 3d models. Machine Vision and Applications,

23(3):557–577.

Schweighofer, G. and Pinz, A. (2006). Robust pose

estimation from a planar target. Pattern Analy-

sis and Machine Intelligence, IEEE Transactions on,

28(12):2024–2030.

Vito, L., Postolache, O., and Rapuano, S. (2014). Mea-

surements and sensors for motion tracking in motor

rehabilitation. Instrumentation Measurement Maga-

zine, IEEE, 17(3):30–38.

Wagner, D., Reitmayr, G., Mulloni, A., Drummond, T., and

Schmalstieg, D. (2008). Pose tracking from natural

features on mobile phones. In Proceedings of the 7th

IEEE/ACM International Symposium on Mixed and

Augmented Reality, pages 125–134.

Welch, G. and Foxlin, E. (2002). Motion tracking: No sil-

ver bullet, but a respectable arsenal. IEEE Comput.

Graph. Appl., 22(6):24–38.

Xu, C., Kuipers, B., and Murarka, A. (2009). 3d pose esti-

mation for planes. In ICCV Workshop on 3D Repre-

sentation for Recognition (3dRR-09).

Yang, H., Wang, F., Xin, J., Zhang, X., and Nishio, Y.

(2012). A robust pose estimation method for nearly

coplanar points. In Proceedings NCSP ’12., pages

345–348.

Zetu, D., Banerjee, P., and Thompson, D. (2000). Extended-

range hybrid tracker and applications to motion and

camera tracking in manufacturing systems. IEEE

Transactions on Robotics and Aut., 16(3):281–293.

High-SpeedandRobustMonocularTracking

471