CELL TRACKING AND DATA ANALYSIS OF IN VITRO
TUMOUR CELLS FROM TIME-LAPSE IMAGE SEQUENCES
Kuan Yan, Fons J. Verbeek
Section Imaging and Bioinformatics, Leiden Institute of Advanced Computer Science, Leiden University
Niels Bohrweg 1, 2333 CA Leiden, The Netherlands
Sylvia Le Dévédec, Bob van de Water
Section Toxicology, Leiden/Amsterdam Center for Drug Research, Leiden University
Niels Bohrweg 1, 2333 CA Leiden, The Netherlands
Keywords: Object tracking, Cell tracking, Cellular phenotype, Tumour cell, Time-lapse video, Cell migration analysis,
KDE mean shift, Steepest descent, High-throughput, High-content, Video analysis, Image sequence.
Abstract: In this paper, we address the problem of the analysis of cellular phenotype from time-lapse image sequences
using object tracking algorithms and feature extraction and classification. We discusses the application of
an object tracking algorithm for in the analysis of high content cell-migration time-lapse image sequence of
extremely motile cells; these cells are captured at low time-resolution.. The small size of the objects and
significant deformation of the object during the process renders the tracking as a non-trivial problem. To
that end, the ‘KDE Mean Shift’, a real-time tracking solution, is adapted for our research. We illustrate that
in a simulation experiment with artificial objects, with our algorithm an accuracy of over 90% can be
established. Based on the tracking result, we propose several morphology and motility based measurements
for the analysis of cell behaviour. Our analysis requires only initial manual interference; the majority of the
processing is automated.
1 INTRODUCTION
Cytomics is the study of cell systems, referred to as
the cytome, at the level of the single cell; it attempts
to understand the molecular architecture and the
functionality of the cell system. Much of this is
achieved by using a combination of molecular and
imaging techniques allowing the various components
of a cell to be visualised by time-lapse image
sequence either in vivo or in vitro. When applied to
larger volumes of cells studied under different
experimental conditions, such a study is usually
referred to as a screen.
An automated cell screening for a 200-target
RNA study will result in 400 image sequences; each
of the sequences contains over 100 cells. Manually
analyzing these videos is virtually impossible.
Therefore, video tracking and data analysis are used;
smart application of these techniques is crucial.
Existing studies of cell tracking (Rogers, et al,
2006) illustrate physical solutions to cell tracking;
this research focuses on in vivo cell tracking using
Magnetic Resonance Imaging (MRI). This approach
is complex and requires advanced MRI equipment
suitable for high-resolution imaging. Computer
vision studies
(Li and Kanade, 2007) also show that
level-set based tracking algorithms can produce good
tracking results directly from sequences as captured
from a fluorescence or bright field microscope. In
these studies, the tracking accuracy reaches an
average of 85%. The method can deal with cell
proliferation by splitting trajectories.
To date, measurements on cell-phenotype have
not been extracted from time-lapse image sequences.
Moreover, the phenomena that are captured in time-
lapse in image sequences, e.g. metastatic behaviour
of a target cell-line, are often subtle and therefore it
requires a range of features be evaluated. The image
sequence is captured such that the relevant
observations on behaviour can be deduced from the
measurements. Sampling below the standard video-
rate is also necessary as otherwise; the amount of
data will not be manageable. Challenges regarding
281
Yan K., Verbeek F., Le Dévédec S. and van de Water B. (2009).
CELL TRACKING AND DATA ANALYSIS OF IN VITRO TUMOUR CELLS FROM TIME-LAPSE IMAGE SEQUENCES .
In Proceedings of the Fourth International Conference on Computer Vision Theory and Applications, pages 281-286
DOI: 10.5220/0001804302810286
Copyright
c
 SciTePress

the analysis of such image sequences are addressed
in this paper.
The structure of this paper is as follows: In
section 2, we describe the pre-processing of raw
image data; this includes segmentation of the images
and subsequent object extraction. The objects that we
extract are cells and these behave as non-rigid bodies
while moving. In section 3, we present our approach
to non-rigid objects tracking and explain our
algorithm for a metastatic/motile cell model based on
time-lapse image sequences of sparse time sampling
(6 min/frame). We estimate the robustness and
accuracy by introducing a test with artificial
images/objects. In section 4, we illustrate our method
with a data set typical for cytomics in which cell
migration is analysed using cell tracking and
phenotypic characterisation.
In this paper, sequence images are obtained at a
fixed rate of 6 min/frame from a CCD camera
mounted on a fluorescence microscope. In case of
artificial images, the situation with the fluorescence
microscope is simulated, i.e. bright objects on a dark
background. The algorithm is implemented as an
ImageJ plug-in in JAVA. Data analysis is performed
by PRTOOLS, a pattern recognition library for
MATLAB.
2 IMAGE PREPROCESSING
In order to be able to trace the objects in our image
sequence, i.e. the cells, all objects must be labelled.
Therefore a segmentation algorithm including the
object labelling was developed. The segmentation
and labelling need to operate in concert as sometimes
the objects appear in clusters. In our algorithm, we
identify these clusters are and then properly separate
them.
The original images are slightly smoothed using a
standard Gaussian filter to remove speckle noise.
Imperfections in the illumination of the field of view
are corrected through a rolling ball
algorithm.(Sternber, 1983)
For the visualization of the cells under the
microscope, a staining procedure is applied.
Variations in the staining of individual cells result in
different intensity profiles of the cells. Consequently,
a global segmentation is less effective for the object
extraction. Therefore, instead, we have implemented
a local adaptive version of the isodata method. This
method employs a spherical kernel with a radius in
the range of [15-20] pixels; this radius depends on
settings derived from the imaging protocol. A
convolution with this kernel results in a in a local
threshold value for each pixel. In order to prevent
problems in large areas of background, the global
average intensity is used a lower boundary of the
threshold; pixels evaluated under that boundary are
set to background. Next, we evaluate the set of
labelled objects resulting from the segmentation as
being part of a cluster of cells or not. To that end, the
initial binary objects are used as a mask in the
original image and a gray-value watershed is
employed to provide further separation in case cells
are clustered (cf. Fig.1). The gray-value watershed
results in a more accurate separation as compared to
the binary watershed (cf. Fig.2).
Now, each separated foreground object is labelled as
a cell and the binary mask of these cells with the
corresponding intensity information will be used as
the initial model for KDE Mean Shift tracking
algorithm.
Figure 1: Left to Right (1) Original Image (2) Binary mask
(3) Watershed without Gaussian filter (4) Watershed with
Gaussian filter.
Figure 2: Left to Right (1) Original Image (2) Binary
watershed (3) Gray-value watershed.
3 CELL TRACKING
We chose the Kernel Density Estimation (KDE)
Mean Shift with Steepest Descent Tracking (KDE
Mean Shift) as a tracking solution due to its
robustness and recognized performance in real-time
tracking (Yang et al., 2003). Although initially
designed for real-time tracking, the algorithm
demonstrates outstanding performance in time-lapse
imaging applications as typical for cytomics.
3.1 KDE Mean Shift with Steepest
Descent Tracking
In computer vision, KDE mean shift (Yang, et al,
2003) is a popular kernel based tracking algorithm
and considered as real world application of an
approach to localization. KDE Mean Shift Tracking
consists of two steps:
VISAPP 2009 - International Conference on Computer Vision Theory and Applications
282

 non-parametric density estimation using an
initial module.
 steepest descent to locate the local maximum in
a gradient space of density estimation.
Each trajectory begins with cells in the first
frame. These cells are converted into initial model
defined in a 3D feature space: (1) the x coordinate of
a binary mask of a cell, (2) the y coordinate of a
binary mask of a cell, (3) the intensity value at each
pixel (x, y).
Given n data points x
n
in the d-dimensional space
R
d
, the kernel density estimator with kernel function
K(x) and window bandwidth h, can be expressed as:
()
∑
=
⎟
⎟
⎠
⎞
⎜
⎜
⎝
⎛
−
=
n
i
i
d
dk
kh
h
xx
K
nh
c
xf
1
2
,
,
ˆ
(1)
A generally used class of kernels is the radial
symmetric kernels, expressed as:
() ( )
2
1
2
1
2
1
2
2
x
d
N
exK
−
∑−
−
−
∑= π
(2)
The mean shift estimation is completed by
steepest descent through iterative computation of:
• the mean shift vector
(
)
k
xm
• the shift module by
(
)
kkk
xmxx +=
+1
The steepest descent requires estimation of a
gradient space
()
(
)
xkxg '−=
, where term
(
)
k
xm is
the new mean shift (cf. Eq. 3). Due to the shape
change (deformation) of the cells, the steepest
descent does not necessarily converge at the centre
of mass of the true candidate. We chose the cell
closest to the stationary point as a true candidate.
()
x
h
xx
g
h
xx
gx
xm
n
i
i
n
i
i
i
−
⎟
⎟
⎠
⎞
⎜
⎜
⎝
⎛
−
⎟
⎟
⎠
⎞
⎜
⎜
⎝
⎛
−
=
∑
∑
=
=
1
2
1
2
(3)
We perform cell tracking on time-lapse image
sequences with different levels of magnification (cf.
Fig.3 and Fig.4). If proper segmentation is applied to
the cell regions then KDE Mean Shift is resistant to
morphology and motility behaviour of cancer cells.
In Figure 4, an estimation of cell motion is
illustrated. Still, estimation of the performance of
KDE using time-lapse image sequences from real
cells is subjective. Therefore, we have developed an
objective and reproducible estimation method. We
initiated this development by testing tracking
accuracy using a time-lapse series containing
artificially generated objects.
Figure 3: Tracking of tumour cells at 40X fluorescence.
Figure 4: Tracking of tumour cells at 20X fluorescence.
3.2 Error Estimation of the Tracking
Our artificial object test, referred to as ‘virtual cell
test’, is utilized for error estimation of the tracking
algorithm. The test of artificial objects rules out
complex situations that occur in “live cells”.
Moreover, the imaging is ideal and illumination
deficiencies do not occur whilst the behaviour of the
virtual “cell” can be controlled completely. The
virtual cell test simulates two basic cell migration
behaviours: (1) shifting position and (2) extending
protrusion.
Table 1.
Parameter name Description
minT Minimum travel distance of a cell
maxT Maximum travel distance of a cell
minH
Minimum extension in y-axis
direction, can be shrinkage
maxH
Maximum extension in y-axis
direction
minW
Minimum extension in x-axis
direction, can be shrinkage
maxW
Maximum extension in x-axis
direction
R
Initial radius of virtual cell
(equivalent to size)
Cell Number of virtual cells in video
Seq
Length of the time-lapse
sequence
Each “virtual cell test” object has several
predefined parameters to control their behaviour (cf.
Table 1). Once created, a virtual cell object will
randomly shift or extend and is restricted by these
CELL TRACKING AND DATA ANALYSIS OF IN VITRO TUMOUR CELLS FROM TIME-LAPSE IMAGE
SEQUENCES
283

parameters. The motion and shape change of the
objects is self-organizing without any predefined
motion model (cf. Fig. 5). In pseudo code, this is
expressed as follows:
Pseudo Code: #virtual cell extending
With a existing virtual cell;
Generate a width in (minW, maxW);
Generate a height in (minH, maxH);
Pseudo Code: #virtual cell position
shift
Do{
Generate a step size in (mint, maxT)
If(causes collision){
Discard the step size;
}
Else{
Keep the step size;
}
}while(step size is not feasible);
Pseudo Code: #generate cell shape
For i in x-width to x+width step 1{
For j in y-height to y+height step 1{
If (i,j) in the range of ellipse
defined by width, height, x, and
y position{
pixel(i,j) is given a
intensity value weighted
by the distance between
(i,j) and mass centre
}
}
}
Image sequences are acquired by recording the
motion of these virtual objects; in this manner, the
true trajectory of cell migration can be captured and
used in error estimation. The score system in our
error estimation consists of two factors, i.e.:
(1) The accomplish ratio ‘T’ between the total
number ‘n’ of trajectory determined by tracking
algorithm and the total number ‘N’ of trajectory
captured during the video generation:
(
)
NTrajectory
NnTrajectory
T
∈
∩∈
=
(4)
(2) The percentage ‘O’ of identical ‘Node’
between each trajectory s determined by the tracking
algorithm and trajectory S as captured from the
image frames:
()
S
Node
SsNode
O
∈
∩
∈
=
(5)
From these factors, the final score is computed:
()
SNode
SsNode
N
n
OTScore
∈
∩∈
⋅=⋅=
(6)
By adding the accomplished ratio ‘T’, require the
tracking algorithm to correctly tracking each
trajectory and capable of identify all possible
trajectories (cf. Fig. 5). Under different behaviour
parameters, KDE mean shift shows an overall
performance above 90% (cf. Table 2).
Figure 5: A sample of cell tracking using virtual cell video.
Table 2: Virtual cell test using different parameters. Each
set generate 10 videos with the same length (30 frames).
The mean error is the average of 10 videos, similarly, the
std error.
Set minT maxT minH maxH minW
1 5 10 20 40 30
2 10 20 20 40 30
3 5 10 20 40 30
4 10 20 20 40 30
5 5 10 20 40 30
6 10 20 20 40 30
7 5 10 20 40 30
8 10 20 20 40 30
Set maxW R cell
Avg
Corr
%
Std
Corr
%
1 60 30 5 99.4 1.87
2 60 30 5 96 6.1
3 60 30 20 86.51 3.66
4 60 30 20 74.4 6.89
5 60 30 5 100 0
6 60 30 5 99.26 1.61
7 60 30 20 97.05 2.91
8 60 30 20 89.47 3.91
Mean
92.76 3.37
3.3 Feature Measurements
We have divided the feature measurements into
motility and morphology measurements based on
biological meaning. Area, perimeter, centre of mass
are used as basic features to describe cell size and
position. In addition, shape is described by
Extension, Dispersion and Elongation which are
derived from moment invariants (Huang, et al,
2003)(Gonzalez, et al, 2004)(Verbeek, 1995)( van
der Putten, et al, 2007)
[4][5][7][8]
; These features are
calculated from normalized moments:
()()()
∑∑
−
−
−−=
1
0
1
0
,
sy
sx
qp
pq
yxFyyxx μ
(7)
VISAPP 2009 - International Conference on Computer Vision Theory and Applications
284

()
∑∑
−
−
=
1
0
1
0
00
,
sy
sx
yxF μ
(8)
1
2
,
00
+
+
==
qp
r
r
pq
pq
μ
μ
η
(9)
and the first and second moment invariant
20021
η
η
φ
+=
(10)
()
2
11
2
20022
4 ηηηφ +−=
(11)
henceforth condensed in:
(
)
211
2 φφπλ +=
(12)
(
)
212
2 φφπλ −=
(13)
1
ln
λ
=Extension
(14)
21
ln
2
1
λλ =Dispersion
(15)
2
1
ln
2
1
λ
λ
=Elongation
(16)
These three measurements (cf. Eqs. 14-16) are
shape descriptors, that are, due to internal
normalization, scale-free and resistant to noise.
Extension measures roughness of a long protrusion
and dispersion measures roughness of the small
protrusions. Elongation measures how cell is
elongated in its major axis. An active cell line
intends to have a high value in all three invariants.
Further, Absolute Position Shift is calculated
as the position shift of one cell between first frame
(cell position
()
00
, yx ) and last frame (cell
position
()
mm
yx , ):
()()
2
0
2
0
yyxxs
mma
−+−=
(17)
Trajectory Length is the total length of
trajectory:
Current position
()
ii
yx ,
Next position
()
11
,
++ ii
yx
()()
∑
=
++
−+−=
m
i
iiiit
yyxxs
0
2
1
2
1
(18)
Velocity is calculated by the shift of centre of
mass divided by time-interval:
()( )
t
yyxx
v
iiii
i
2
1
2
1
−+−
=
++
(19)
Motion Linearity is the ratio between absolute
position shift and trajectory length; normalized by
trajectory length, motion linearity is scale-free:
Since we aim to characterize the cellular
phenotype, the study of feature measurements in one
cell in one particular frame has little biological
meaning. Therefore, we generalized the frame-based
measurement into a cell-based measurement.
Considering measurements as a discrete signal (cf.
Fig. 4), it can be generalized by:
(
)
(
)
TtACostf +
+
=
ω
α
(20)
The amplification A, the frequency
α
, and shift
T may well preserve the majority structure of such
discrete signal (measurement vs. time). With these
cell-based measurements (cf. Table 3), we are able to
compare cell behaviour and establish significant
changes in behaviour. In the next section, a test for
reliability of our principles is discussed.
Table 3: The cell-based measurements.
Cell-based
measurement
Description
Average size
Average size for one cell through
all frames
Average extension
Average extension for one cell
through all frames
Std extension
Standard deviation of extension for
one cell through all frames
Average dispersion Same as average extension
Std dispersion Same as std extension
Average elongation Same as average extension
Std elongation Same as std extension
Average velocity
Average velocity for one cell
through all frames
Average motion
linearity
Average motion linearity for one
cell through all kernel window
4 RELIABILITY TEST AND
BIOLOGICAL EXPERIMENTS
In Figure 6, we show the result of an experiment
with motility stimulation. In two groups, cells are
treated with a growth factor (experiment) and
expected to move faster with respect to the control
group (control). The effect of this stimulation is
observed in our measurement results. (Alberts, et al,
1994)
Average Velocity of All Cell in Ctrl Group vs. Experiment Group
0
0.5
1
1.5
2
2.5
3
3.5
4
4.5
5
control experiment control experiment
group 1 group 2
Average Velocity
average velocity
Figure 6: Control vs. EGF treated.
In addition, multiple subgroups of cells from the
same sample culture are determined using K-Mean
CELL TRACKING AND DATA ANALYSIS OF IN VITRO TUMOUR CELLS FROM TIME-LAPSE IMAGE
SEQUENCES
285

Clustering (cf. Fig. 7). The number of clusters is
validated by the lowest Davies-Bouldin Index (DBI)
score (cf. Fig. 8). The result of DBI score confirms
three natural subgroups: (1) Normal cells (2)
Correctly treated cells (3) Incorrectly treated cells.
These measurements are consistent with results from
similar experiments on different dates.
-2
0
2
-2
0
2
-2
0
2
Cell Size
Unsupervised Clustering on Cell Beavhior
Cell Elongation
C
e
ll
V
e
l
oc
it
y
Figure 7: Cell behaviour clustering.
2 4 6 8 10
1.4
1.6
1.8
2
2.2
2.4
2.6
2.8
Number of Cluster
Davies-Bouldin Validity Index
Clustering Validation using DBI Score
Figure 8: Clustering validation using Davies-Bouldin
validity index.
5 CONCLUSIONS
Object tracking has been studied comprehensively in
computer vision. We investigated object tracking
algorithms to support cytomics research and we
demonstrated how these can be successfully applied.
Our results, i.e. object tracking and data analysis, on
real data illustrate application on image sequences
depicting a metastatic/motile cell model.
We developed an artificial object test and this test
shows that our approach of the KDE Mean Shift can
provide an accuracy over 90% (85% in level set
tracking); for cell-tracking analysis this is acceptable.
The measurements on the cells resulting from the
tracking present correct conclusions in relation to the
biological experiment. The feedback from the “wet-
lab” indicates that labour time of post-experiment
data analysis is reduced enormously (≥ 300%) while
accuracy of cell-migration analysis has significantly
increased. Moreover, automation allows processing
of large volumes of data.
Finally, the tracking analysis of migrating
(tumour) cells provides sufficient confidence to
continue further research on structural level tracking.
ACKNOWLEDGEMENTS
This research has been partially supported by the
BioRange Programme of the Dutch ministry of
Economic affairs (BSIK grant). We would like to
thank Hans de Bont for his assistance in the
microscopy.
REFERENCES
Yang, CJ., Duraiswami,, R. DeMenthon,, D. and Davis,
L.S. (2003). Mean-shift analysis using quasi Newton
methods, In Image Processing, International
conference on Volume 2, Issue , 14-17 Sept. 2003
Perceptual, Interfaces & Reality Laboratory,
University of Maryland, USA
Li, K. and Kanade, T. (2007). Cell Population Tracking
and Lineage Construction Using Multiple-Model
Dynamics Filters and Spatiotemporal Optimization, In
Proceedings of the 2nd International Workshop on
Microscopic Image Analysis with Applications in
Biology (MIAAB). Carnegie Mellon University, USA
Alberts, B., Johnson, A., Lewis, J., Raff, M., Roberts, K.
and Walter, P., (1994). Molecular Biology of the Cell,
Garland Science Publishing, New York, ISBN 0-8153-
4072-9, USA
Huang, C., Rajfur, Z., Borchers, C., Schaller, M.D. and
Jacobson, K. (2003). phosphorylates paxillin and
regulates cell migration, In Journal Nature,
Department of Cell and Developmental Biology,
Comprehensive Center for Inflammatory Disorders,
University of North Carolina, USA.
Gonzalez, R.C., Woods, R.E. and Eddins, S.L., (2004)
Digital Image Processing using MATLAB, Person
Prentice Hall Publishing, ISBN 0-13-008519-7, UK
Verbeek, F.J. (1995). Three dimensional reconstruction
from serial sections including deformation correction
(Delft, Delft University of Technology), the
Netherlands.
van der Putten, P., Bertens, L.F.M., Liu, J., Hagen, F.,
Boekhout T., Verbeek F.J. (2007). Classification of
Yeast Cells from Image Features to Evaluate Pathogen
Conditions, In Proc. SPIE Vol. 6506, MultiMedia
ontent Access: Algorithms & Systems, Vol. 6506 (Eds
Hanjalic, A., Schettini, R., Sebe, N.), 65060I-1,65060I-
14
Rogers, W.J., Meyer, C.H. and Kramer, C.M. (2006),
Technology Insight: In Vivo Cell Tracking by Use of
MRI, In Nat Clin Pract Cardiovasc Med, Nature
Publishing Group, USA
Sternber, S., (1983), Rolling ball algorithm, In Biomedical
Image Processing, IEEE Computer, January 1983
VISAPP 2009 - International Conference on Computer Vision Theory and Applications
286