An Image-based Ensemble Kalman Filter for Motion Estimation
Yann Lepoittevin
1
, Isabelle Herlin
1
and Dominique B
´
er
´
eziat
2,3
1
Inria Paris-Rocquencourt, Domaine de voluceau, 78150, Le Chesnay, France
2
LIP6, Sorbonne Universities, UPMC Univ Paris 06, UMR 7606, 75005, Paris, France
3
LIP6, CNRS, UMR 7606, Paris, 75005, France
Keywords:
Data Assimilation, Ensemble Kalman Filter, Localization, Motion Estimation, Optical Flow.
Abstract:
This paper designs an Image-based Ensemble Kalman Filter (IEnKF), whose components are defined only
from image properties, to estimate motion on image sequences. The key elements of this filter are, first,
the construction of the initial ensemble, and second, the propagation in time of this ensemble on the studied
temporal interval. Both are analyzed in the paper and their impact on results is discussed with synthetic and real
data experiments. The initial ensemble is obtained by adding a Gaussian vector field to an estimate of motion
on the first two frames. The standard deviation of this normal law is computed from motion results given by
a set of optical flow methods of the literature. It describes the uncertainty on the motion value at initial date.
The propagation in time of the ensemble members relies on the following evolution laws: transport by velocity
of the image brightness function and Euler equations for the motion function. Shrinking of the ensemble is
avoided thanks to a localization method and the use of observation ensembles, both techniques being defined
from image characteristics. This Image-based Ensemble Kalman Filter is quantified on synthetic experiments
and applied on traffic and meteorological images.
1 INTRODUCTION
This paper revisits the use of optical flow methods to
estimate motion on discrete image sequences. Based
on the concept of data assimilation the filter relies
on evolution laws of the dynamics underlain in the
image sequence. The aim is to retrieve a dense-in-
time motion estimation from a sparse-in-time discrete
image sequence. 4D-Var data assimilation methods
have been successfully used for motion estimation, as
for instance in (Papadakis et al., 2007; Titaud et al.,
2010; B
´
er
´
eziat and Herlin, 2011). However, these ap-
proaches present some major restrictions:
• an adjoint model is required, which needs addi-
tional theoretical work and software development,
• processing an image sequence requires a num-
ber of forward integration of the model and back-
ward integration of its adjoint during the opti-
mization process. Even if the code is parallelized,
the whole process necessitates more computation
time than a single integration of the model, as
done with a filtering approach.
• they provide an estimate of motion fields on the
whole time interval, from the image sequence, but
no uncertainty measure is at hand. That uncer-
tainty value is however mandatory for further in-
terpretation of motion results.
The design of filtering methods, and in particu-
lar the ones based on the Ensemble Kalman Filter
(EnKF) (Evensen, 2003), is an alternative that is not
affected by the previous limitations. EnKF is an adap-
tation of the Kalman filter (Kalman, 1960) that does
not use an analytical description of the Probability
Density Function (PDF) but samples it. To implement
the filter, an ensemble of motion fields is constructed
at initial date and propagated in time. At each date,
an estimate of motion is given by the mean of the
members and its uncertainty PDF is described by the
spread of the ensemble.
As experiments are done with a small-size ensem-
ble (an ensemble of 40 members is constructed for the
experiments described in the paper), in order to keep
the computational cost as low as possible, the values
of the different members usually come closer one to
each other during temporal integration: this is named
“shrinking of the ensemble”. Several approaches are
used to avoid this deficiency. Inflation and localiza-
tion methods (Anderson and Anderson, 1999; Hamill
et al., 2001; Oke et al., 2007) either increase the vari-
ance values or decrease the covariance values of the
matrices computed on the ensemble members before
the analysis step. Other methods, such as the deter-
ministic EnKF (Sakov and Oke, 2008), correct the
members obtained after the analysis step so that the
covariance matrices computed from these members
437
Lepoittevin Y., Herlin I. and Béréziat D..
An Image-based Ensemble Kalman Filter for Motion Estimation.
DOI: 10.5220/0005259804370445
In Proceedings of the 10th International Conference on Computer Vision Theory and Applications (VISAPP-2015), pages 437-445
ISBN: 978-989-758-091-8
Copyright
c
2015 SCITEPRESS (Science and Technology Publications, Lda.)
become closer to the ones modeled with the Kalman
filter. Beyou et al. (Beyou et al., 2012) apply EnKF
for motion estimation. Each motion member is writ-
ten as the sum of a mean and a perturbation. When
computing the analysis, this formalism involves an
additional inversion of the observation error covari-
ance matrix. Shrinking is avoided by resampling the
ensemble before further integration. In our paper, the
methods chosen to solve the shrinking issue are only
based on image properties and should not lead to pro-
hibitive computational time. A precise tuning of the
inflation parameter is required to simultaneously get
accurate results and keep the spread of the ensemble.
Several methods of the literature automatically define
the optimal value (Anderson, 2007). However, this
inflation parameter may not be directly defined from
image properties. The deterministic EnKF computes
the covariance matrix associated to the perturbations
after the analysis step, conducting to an overload of
computations. Due to the resampling process applied
in (Beyou et al., 2012), the error covariance matrices,
computed from the ensemble members, do not char-
acterize the uncertainty on motion estimation. Con-
sequently, these three approaches are not used in the
following. In order to avoid shrinking, observation
ensembles (Houtekamer and Mitchell, 1998) are first
constructed, based on characteristics of image acqui-
sitions. This is described in Subsection 3.4. Second,
an innovative localization process is defined to limit
spurious covariances between pixels. It is based on:
1- distance between pixels (points separated by a high
distance are usually independent), 2- similarities be-
tween pixels (points belonging to the same objects
or structures are usually highly correlated). It is de-
scribed in Subsection 3.3.
Section 2 defines the mathematical notations and
summarizes the EnKF algorithm, while Section 3 de-
scribes the characteristics of the method discussed in
the paper: an original method for constructing the en-
semble, evolution laws and propagation in time of the
ensemble. Section 4 discusses the results obtained on
a synthetic experiment and on real data.
2 ENSEMBLE KALMAN FILTER
As said before, the aim of this research work is to
produce a dense-in-time motion estimation from a
sparse-in-time discrete image sequence.
2.1 Mathematical Setting
Let define the following notations that are used in the
remaining of the paper. x =
x y
T
is a pixel belong-
ing to the image domain Ω, with .
T
the transpose op-
erator. The sequence of N
O
images
I
O
l
, l ∈ 1..N
O
is defined on Ω. It has been acquired on a given tem-
poral interval 0..T .
The data assimilation method requires the definition
of a state vector X whose value at time index k is
X
k
=
w
T
k
I
k
T
. w =
u v
T
denotes the motion
field and I is a synthetic image field with the same
physical properties than the studied image sequence.
If an image acquisition is available at index k, it is
used to compute the observation vector Y
k
. The aim
of data assimilation is to get an estimate, or analysis,
X
(a)
k
of the true state vector X
(r)
k
from the so-called
background state vector X
(b)
k
and, if available, the ob-
servation vector Y
k
.
Ensemble methods rely on a number of members that
evolve simultaneously in time. X
j
k
denotes the state
vector at index k of the j
th
member of the ensemble. If
an observation ensemble is constructed from images,
Y
j
k
is the j
th
member of this observation ensemble at
index k.
. denotes the mean over the ensemble members.
2.2 Algorithm
The Kalman Filter (Kalman, 1960) performs a time
integration of the model: it provides an approxima-
tion of the true state and an uncertainty value of this
estimation, given the following elements:
• A background value X
(b)
0
. Its uncertainty is de-
scribed by a PDF, which is supposed to be a zero-
mean normal law with covariance matrix B
(b)
0
.
• Observations Y
k
, at some time indexes k for
which images are available. Their uncertainty is
described by zero-mean normal laws of covari-
ance matrices R
k
.
• At index k, the state vector X
k
and the observation
vector Y
k
are compared thanks to an observation
operator IH, such that IH(X) belongs to the obser-
vation space.
• Time integration of the state vector is obtained
from a linear model IM:
X
k
= IM
X
k−1
. (1)
Equations of the Kalman filter, in case of a linear
observation operator IH, are the following:
1. At time index k, the background value X
(b)
k
is ob-
tained from the estimation (also named analysis)
X
(a)
k−1
at index k − 1 by applying:
X
(b)
k
= IM
X
(a)
k−1
. (2)
VISAPP2015-InternationalConferenceonComputerVisionTheoryandApplications
438
The propagation in time of the uncertainty covari-
ance matrix B
(b)
k
satisfies:
B
(b)
k
= IMB
(a)
k−1
IM
T
. (3)
For sake of simplicity, IM denotes both the linear
model and its associated matrix.
2. If no observation is available at k, the estimation
X
(a)
k
and its uncertainty B
(a)
k
are taken equal to that
of the background.
3. If an observation vector Y
k
is available at k, then
the analysis is given by the analysis equation:
X
(a)
k
= X
(b)
k
+ K
Y
k
− IHX
(b)
k
, (4)
where :
K = B
(b)
k
IH
T
IHB
(b)
k
IH
T
+ R
k
−1
, (5)
with IH denoting both the linear operator and its
associated matrix. The uncertainty covariance
matrix verifies:
B
(a)
k
= B
(b)
k
− KIHB
(b)
k
. (6)
Having summarized the Kalman filter equations,
two major issues have to be discussed. First, time
propagation of the background covariance matrix B
(b)
k
with Eq. (3), by the linear model IM, leads to pro-
hibitive computational requirements for large-sized
state vectors. Second, if IM is non linear, its ap-
proximation by its tangent linear model M is used for
the propagation of the uncertainty in Eq. (3). Con-
sequently, the resulting value B
(b)
k
is only an approx-
imation of the true value. In this paper, the model
IM, expressing the Lagrangian constancy of velocity,
includes non linear equations and its propagation is
affected by the approximation. These two issues of
computational burden and approximation are avoided
by the Ensemble Kalman filter (Evensen, 2003). An
ensemble of background state vectors X
(b), j
k
is defined
at each time index k. It samples the uncertainty co-
variance matrix B
(b)
k
. Let denote X
(b)
k
= X
(b), j
k
the
mean over the ensemble members. An approximation
of the covariance matrix is obtained from the ensem-
ble using the equation:
B
(b)
k
≈ (X
(b), j
k
− X
(b)
k
)(X
(b), j
k
− X
(b)
k
)
T
. (7)
Initialized at initial date 0, the ensemble is prop-
agated in time by integrating each member using the
model IM as in Eq. (1). If no observation is available
at time index k, the estimation X
(a)
k
is set equal to X
(b)
k
and the uncertainty is approximated using Eq. (7). If
an observation is available at k, an analysis is com-
puted for each member j by:
X
(a), j
k
= X
(b), j
k
+ K(Y
k
− IHX
(b), j
k
) , (8)
where :
K = B
(b)
k
IH
T
IHB
(b)
k
IH
T
+ R
k
−1
. (9)
The estimation is defined as X
(a)
k
= X
(a), j
and its
uncertainty is approximated by replacing
(b)
by
(a)
in
Eq. (7). It should be noted that all members are in-
volved in the computation of B
(b)
k
and consequently
in the estimation of each element X
(a), j
k
.
The ensemble Kalman filter is naturally paralleliz-
able, as time integration of each member is indepen-
dent from the others. Moreover, matrix products and
inversions are done by the parallel linear algebra li-
brary PLASMA (Agullo et al., 2009) that enables to
significantly reduce computational costs.
3 IMPLEMENTATION OF AN
ENSEMBLE KALMAN FILTER
FOR MOTION ESTIMATION
This section describes the core of this paper: the de-
sign of an ensemble Kalman filter for motion esti-
mation from an image sequence. It is named Image-
based Ensemble Kalman Filter or IEnKF.
3.1 Evolution Model
As previously explained, the ensemble members are
integrated in time by the numerical model IM ob-
tained by time discretization of a continuous evolu-
tion model IM
c
. The heuristics for the evolution of
motion and images are the following:
• Lagrangian constancy of velocity w:
dw
dt
=
∂w
∂t
+ (w · ∇)w = 0 . (10)
• Transport of image brightness. This is used both
for the image observations and for the synthetic
image function I included in the state vector:
∂I
∂t
+ w · ∇I = 0 . (11)
The state vector is defined by X =
w
T
I
T
.
Eqs. (10) and (11) are then summarized by:
∂X
∂t
+ IM
c
(X(t)) = 0 . (12)
Temporal discretization of Eq. (12) with an explicit
Euler scheme leads to:
X
k
= IM
X
k−1
(13)
that is applied for propagating each member of the
ensemble.
AnImage-basedEnsembleKalmanFilterforMotionEstimation
439
3.2 Ensemble Construction
The motion estimation method highly depends on the
design of the initial ensemble at the beginning of the
image sequence. This ensemble should span a vec-
torial space that contains the truth and properly sam-
ples the uncertainties. It is usually obtained by adding
some perturbations to an initial guess with a Monte
Carlo method. This subsection describes an innova-
tive alternative where the ensemble is designed from
the image observations.
The state of each ensemble member is composed
of a motion field and an image. However, there is no
need to construct an ensemble of images, as the image
component is only used as a display of the underlain
dynamics. Consequently, at initial date, the members
include the same image, the first acquisition, but var-
ious motion fields.
There are lots of methods available in order to ap-
proximate motion between two observations, see for
instance (Sun et al., 2010; Baker et al., 2011) for a
survey on the subject. The approaches differ in their
formulation of the optical flow, the spatial smoothness
assumptions, the implementation tools (coarse-to-fine
estimation, texture decomposition, median filtering,
etc.). For each approach, various results are obtained
according to parameters’ values. One method may be
the best for one part of the temporal sequence while
another one succeeds in the remaining. The same ob-
servation is valid in space for the different regions of
the image domain. This conducted to the design of
new optical flow methods in which the data term is
varying in the space-time domain according to image
properties as in (Mac Aodha et al., 2010). Being in
the same spirit, we came to the conclusion that gener-
ating motion fields from a large set of codes, varying
formulation and parametrization, is the best way to
ensure spanning a vector space of motion fields in-
cluding the true motion field and to assess the uncer-
tainty on estimation.
The optical flow algorithms, applied in the paper,
are variational methods that rely on the brightness
constancy hypothesis (Horn and Schunk, 1981). They
estimate motion by minimizing a cost function, which
includes, at least, a data term and a weighted regular-
ization term. Modifying this weight value allows the
user to choose smoothness properties of the solution.
A small value conducts to a result strongly relying on
the data and containing outliers linked to acquisition
noise, whereas a higher one produces smooth solu-
tions. Three types of norm are implemented to com-
pute the cost function: the quadratic norm (Horn and
Schunk, 1981), the Charbonnier norm (Brox et al.,
2004) and the Lorentzian norm (Black and Anandan,
1996). For each method, a coarse-to-fine approach is
used in order to converge while minimizing the cost
function. Before each resolution change, the motion
field at the current scale is smoothed by a median fil-
ter, whose size is also a parameter. Its value impacts
the quality of the solution as discussed in (Sun et al.,
2010). Another possibility to control smoothness of
the motion field is to integrate this median filtering
directly in the cost function. A weight is then given to
the corresponding additional term as in (Li and Osher,
2009).
Motion is estimated between the first two images
with this set of methods and various parametrizations
concerning the regularization weight, the norm type,
the filter type (median or weighted median). Such
motion result is denoted m
j
with j spanning the set
of methods. Let denote m = m
j
the mean of results.
An uncertainty measure on motion is defined by the
covariance matrix:
B = (m
j
− m)(m
j
− m)
T
. (14)
The motion field corresponding to the mean of the
ensemble is then chosen as initial value. The motion
ensemble is constructed from this value and the nor-
mal law associated to the covariance matrix B. One
member is denoted w
j
. The size of that ensemble is
40 in the experiments.
3.3 Ensemble Propagation
As stated in Section 1, a localization method is used in
order to limit the shrinking that occurs after comput-
ing analysis with Eqs. (8) and (9). This localization
corrects the spurious covariances that appear when
computing the covariance matrix from a small-sized
ensemble with Eq. (7). These covariances link pixels
that should be independent because too far one from
each other or belonging to different objects of the im-
age. Let ρ denote the localization matrix and ◦ the
point-wise matrix product (also known as Hadamard
product). The localization process is applied before
computing the analysis from background and obser-
vations. Let define
L
(b)
k
= ρ ◦ B
(b)
k
(15)
and replace B
(b)
k
by L
(b)
k
in the analysis equations
Eqs. (8) and (9). We obtain the following equations:
X
(a), j
k
= X
(b), j
k
+ K
L
(Y(k) − IHX
(b) j
k
) (16)
where :
K
L
= L
(b)
k
IH
T
IHL
(b)
k
IH
T
+ R
k
−1
, (17)
that compute the analysis X
(a), j
k
of member j at time
index k.
VISAPP2015-InternationalConferenceonComputerVisionTheoryandApplications
440
The main group of methods for defining the ρ ma-
trix, denoted ρ
d
(where the subscript d stands for dis-
tance), is defined according to distances between pix-
els, as in (Hamill et al., 2001; Haugen and Evensen,
2002; Brankart et al., 2003). ρ
d
is thus designed so
that pixels separated by a distance higher than a given
threshold get an almost null correlation. Let x
1
and x
2
denote two pixels of the image domain, ρ
d
depends on
their distance d
12
= ||x
1
− x
2
||
2
and is defined by:
ρ
d
(x
1
, x
2
) =
1 +
d
12
a
× exp
−
d
12
a
. (18)
The parameter a, involved in Eq. (18), is called decor-
relation distance. The values of ρ
d
, depending on d
12
,
are displayed on Fig. 1 for three values of a.
Figure 1: Values of ρ
d
, for a = 1, 2, 10, as a function of the
distance between pixels.
We propose a method that also nullifies correlation
between pixels that belong to different objects. The
localization matrix ρ
s
(where the subscript s stands
for similarity) depends on pixels similarities. Let de-
note I
1
and I
2
the gray values of pixels x
1
and x
2
and
s
12
= ||I
1
− I
2
||
2
. ρ
s
is defined by:
ρ
s
(l
1
, l
2
) =
1 +
s
12
a
l
× exp
−
s
12
a
l
. (19)
Our innovative localization matrix ρ, used in
Eq. (15), is then defined as a function of ρ
d
and ρ
s
.
Consequently, the correlation values between pixels
that are far apart or belong to different objects are al-
most null.
3.4 Observations
As defined in Subsection 2.1, the state vector is writ-
ten X =
w
T
I
T
. The synthetic image I satisfies the
assumption that brightness values are transported by
velocity, as explained in Subsection 3.1. The observa-
tion operator IH is defined as the linear projection on
the second component of the state vector:
IH(X) = I . (20)
This observation operator is used during the assimi-
lation process to compare the function I with the im-
age acquisitions: the motion field is estimated in order
minimize the discrepancy between them.
As explained in Section 1, observation ensembles
are used in order to avoid shrinking when propagating
the ensemble members in time. In the case of motion
estimation from images, the formulation of the obser-
vation ensemble associated to one image acquisition
is straightforward. Information on the sensor charac-
teristics allows the user to model the acquisition noise
as an additive zero-mean Gaussian noise, whose stan-
dard deviation is usually included in the metadata as-
sociated to the sensor. This is written as:
I
O
k
= I
(r)
k
+ N (0, σ), (21)
where I
O
k
is the acquired image of the unknown truth
I
(r)
k
. The image ensemble associated to I
O
k
is then ob-
tained with the same heuristics. Each member I
j
k
is
obtained by adding a gaussian noise to the acquired
image:
I
j
k
= I
O
k
+ N (0, σ). (22)
Experiments showed that an ensemble of 20 elements
is sufficient to correctly sample the error law and
avoid shrinking. Considering additional members in-
creases the memory requirements without decisive
improvement on results. Having less members de-
creases the accuracy of motion estimation.
4 RESULTS ON MOTION
ESTIMATION
4.1 Synthetic Experiment
Starting from the initial image and motion field, dis-
played on Fig. 2, the model IM is integrated in time,
as explained in Subsection 3.1. Motion fields ob-
Figure 2: Initial conditions of ground-truth.
tained from that integration are used as ground-truth
for evaluating results. Six image snapshots are taken
every ten time steps and displayed on Fig. 3. They
AnImage-basedEnsembleKalmanFilterforMotionEstimation
441
are the image observations of the assimilation experi-
ment. The observation ensemble is constructed from
these data as explained in Subsection 3.4. A motion
Figure 3: Snapshots of the synthetic experiment.
ensemble of 40 members is created as explained in
Subsection 3.2.
IEnKF is then applied to estimate motion either
without localization or with localization during com-
putation of the analysis using respectively Eq. (8) or
Eq. (16). The decorrelation distance that is used for
the localization process has a value of 1.
Errors statistics on the discrepancy between esti-
mation and ground-truth are computed and analyzed.
The one concerning the angular error is obtained by
taking the spatial Average of the Absolute values of
Angular Errors (AAAE). First, AAAE are computed
for 20 members (randomly chosen among the set of
40) and visualized as blue points, at each date of the
studied temporal interval, on Fig. 4. Second, AAAE is
computed for the ensemble mean and represented as
a red curve on Fig. 4. Figures on norm errors are sim-
ilar. As it can be seen, the initial ensemble is widely
spread and the first analysis step, at date 10, signifi-
cantly reduces the errors. One can see the effects of
localization on the bottom figure, compared to the top
one. First, the ensemble is spread enough for a longer
time. When the shrinking is important, the error co-
variance matrix becomes small. At the limit, when the
matrix is almost null, the analysis step has no more ef-
fect on the ensemble (as it can be seen in the analysis
formula, Eq. (8)). Keeping an ensemble spread large
enough, as long as possible, is then of major inter-
est for online image processing. Second, as visible
at the first analysis step, the localization increases the
accuracy of the estimation. At the end of the studied
temporal interval, the angular error, in degrees, is 0.7
with localization and 1.7 without. Third, the accu-
racy of the estimation still improves at the 5th or 6th
analysis step with localization, whereas no effect oc-
curs anymore after the 4th computation of an analysis
without localization.
Figure 4: spatial Average of the Absolute values of Angu-
lar Errors (AAAE) over time without localization (top) and
with localization (bottom). The blue points correspond to
the AAAE of members of the ensemble and the red curve to
the AAAE of the ensemble mean.
Fig. 5 displays, on top, the norm of the error be-
tween the estimation and the truth and, on the bot-
tom, the variance of the ensemble results. It can be
observed that the spread of the ensemble has a good
geographical correspondance with the errors.
If computing motion between the 5th and 6th ob-
servations with the whole set of motion estimation al-
gorithms (see Subsection 3.2) and analyze the results,
the best method shows an average angular error of 5
degrees, to be compared with the 1-degree error ob-
tained by our method.
VISAPP2015-InternationalConferenceonComputerVisionTheoryandApplications
442
Figure 5: Top: Norm of the error between the estimation
and the ground truth. Bottom: Variance of the ensemble
results.
4.2 Experiment on Real Data
IEnKF was applied on traffic images from the
database KOGS/IAKS of the Karlsruhe Univer-
sity (http://i21www.ira.uka.de/image sequences/,
1995). The previously described implementation of
IEnKF is used in order to estimate motion of cars. As
quality assessment is not at hand with arrow or color
displays, results are visualized on Fig. 6, thanks to
trajectories of characteristic points. Each trajectory
is obtained by advecting the chosen point with the
estimated motion field. On each image of the figure,
a colored circle displays the current position of one
characteristic point and the corresponding curve its
whole trajectory. As it can be seen, motion is cor-
rectly estimated, in norm and orientation, for the taxi
and the van coming from the right. This conclusion
is still valid when defining new characteristic points.
The black car on the left gets weaker quality motion
results due to its low contrast.
A second sequence of the same data base has
been processed by IEnKF and results are given on
Fig. 7 with trajectories of characteristic points dis-
Figure 6: Motion results given as trajectories, with the posi-
tion (circle) of a characteristic point on the first frame (top)
or the last one (bottom).
Figure 7: Motion results given as trajectories, with the posi-
tion (circle) of a characteristic point on the first frame (top)
or the last one (bottom).
played on the first and last images. As cars are ac-
celerating when the traffic light turns green, the al-
gorithm slightly underestimates their speed. A future
implementation of the method will involve an accel-
eration term in the state vector in order to improve the
estimation.
Results obtained on a third sequence of this
database are displayed on Fig. 8. The computed tra-
jectories of three characteristic points are shown and
one can visualize that one car undertakes a U-turn.
IEnKF was also applied on meteorological satel-
lite data where images are acquired every 15 minutes
in the infrared domain with a 5 km resolution. The
AnImage-basedEnsembleKalmanFilterforMotionEstimation
443
Figure 8: Motion results given as trajectories, with the posi-
tion (circle) of a characteristic point on the first frame (left)
or the last one (right).
Figure 9: Advection of the boundary by the estimated mo-
tion fields.
foreseen operational application is to improve short
term forecast of cloud cover. A first experiment con-
cerns a sequence displaying convective cells. Results
are given on Fig. 9 by advecting the boundary of one
cloud with the estimated motion field. One can see
that these points remain on the boundary of the con-
vective cell during time integration. This accurate
tracking confirms that the cell velocity is well esti-
mated.
IEnKF was applied on a second meteorological
sequence. Results are given on Fig. 10 by advecting
the boundary of one cloud with the estimated motion
field. This again demonstrates correctness of the esti-
mation.
Figure 10: Advection of the boundary by the estimated mo-
tion fields.
5 CONCLUSION
The paper describes a motion estimation method,
based on the ensemble Kalman filter, which is named
IEnKF. The initial motion ensemble is defined from
image properties and a number of optical flow meth-
ods from the literature. The usual shrinking of the en-
semble is delayed thanks to the construction of obser-
vation ensembles and to a localization method, both
defined from image properties. The approach was
tested on synthetic experiments whose ground-truth
allows quantifying results. It was also applied on
two major applications. On traffic images, the ap-
proach allows a correct estimation of motion, which is
demonstrated by computing trajectories of character-
istic points on moving cars. On meteorological satel-
lite acquisitions, the method demonstrates its perfor-
mance with the correct tracking of clouds boundary.
Future work concerns the extension of the state
vector to take into account acceleration and a math-
ematical description of objects and structures dis-
played on the sequence. Both components will impact
the estimation of motion. For being able to process
images online, two main issues will be investigated
as, for now, computation on a 100 × 100 pixels image
takes approximately 10 minutes and several gigabytes
of active memory. First, on the methodological part,
model reduction is foreseen with small-size motion
and image subspaces. The reduced state vector will
then contains around 10-20 components and applica-
tion of IEnKF will be at low memory cost. Second, all
components of code will be fully parallelized: IEnKF,
computation of motion and image bases, and model
reduction.
ACKNOWLEDGEMENTS
This research has been partially funded by the DGA.
REFERENCES
Agullo, E., Demmel, J., Dongarra, J., Hadri, B., Kurzak,
J., Langou, J., Ltaief, H., Luszczek, P., and Tomov, S.
(2009). Numerical linear algebra on emerging archi-
tectures: The PLASMA and MAGMA projects. Jour-
nal of Physics: Conference Series, 180(1):012037.
Anderson, J. L. (2007). An adaptive covariance inflation
error correction algorithm for ensemble filters. Tellus
A, 59:210–224.
Anderson, J. L. and Anderson, S. L. (1999). A Monte
Carlo implementation of the nonlinear filtering prob-
lem to produce ensemble assimilations and forecasts.
Monthly Weather Review, 127:2741–2758.
VISAPP2015-InternationalConferenceonComputerVisionTheoryandApplications
444
Baker, S., Scharstein, D., Lewis, J., Roth, S., Black, M.,
and Szeliski, R. (2011). A Database and Evaluation
Methodology for Optical Flow. International Journal
on Computer Vision, 92(1):131.
B
´
er
´
eziat, D. and Herlin, I. (2011). Solving ill-posed image
processing problems using data assimilation. Numer-
ical Algorithms, 56(2):219–252.
Beyou, S., Cuzol, A., Gorthi, S. S., and M
´
emin, E. (2012).
Weighted ensemble transform Kalman filter for image
assimilation. Tellus A, 65.
Black, M. J. and Anandan, P. (1996). The robust estima-
tion of multiple motions: Parametric and piecewise-
smooth flow fields. Computer Vision and Image Un-
derstanding, 63(1):75 – 104.
Brankart, J.-M., Testut, C.-E., Brasseur, P., and Verron, J.
(2003). Implementation of a multivariate data assimi-
lation scheme for isopycnic coordinate ocean models:
Application to a 1993–1996 hindcast of the North At-
lantic Ocean circulation. Journal of Geophysical Re-
search: Oceans, 108.
Brox, T., Bruhn, A., Papenberg, N., and Weickert, J. (2004).
High accuracy optical flow estimation based on a the-
ory for warping. In Springer-Verlag, editor, Proceed-
ings of European Conference on Computer Vision,
volume 4, pages 25–36, Prague, Czech Republic.
Evensen, G. (2003). The ensemble Kalman filter: Theoret-
ical formulation and practical implementation. Ocean
Dynamics, 53:343–367.
Hamill, T. M., Whitaker, J. S., and Snyder, C. (2001).
Distance-dependent filtering of background error co-
variance estimates in an ensemble Kalman filter.
Monthly Weather Review, 129:2776–2790.
Haugen, V. E. J. and Evensen, G. (2002). Assimilation
of SLA and SST data into an OGCM for the Indian
Ocean. Ocean Dynamics, 52(3):133–151.
Horn, B. and Schunk, B. (1981). Determining optical flow.
Artificial Intelligence, 17:185–203.
Houtekamer, P. L. and Mitchell, H. L. (1998). Data as-
similation using an ensemble Kalman filter technique.
Monthly Weather Review, 128:796–811.
http://i21www.ira.uka.de/image sequences/ (1995).
Nibelungen-platz.
Kalman, R. E. (1960). A new approach to linear filtering
and prediction problems. Transactions of the ASME–
Journal of Basic Engineering, 82(Series D):35–45.
Li, Y. and Osher, S. (2009). A new median formula with
applications to PDE based denoising. Communication
in Mathematical Sciences, 7(3):741–753.
Mac Aodha, O., Brostow, G., and Pollefeys, M.
(2010). Segmenting video into classes of algorithm-
suitability. In Proceedings of Conference on Com-
puter Vision and Pattern Recognition, pages 1054–
1061.
Oke, P. R., Sakov, P., and Corney, S. P. (2007). Impacts of
localisation in the EnKF and EnOI: experiments with
a small model. Ocean Dynamics, 57(1):32–45.
Papadakis, N., Corpetti, T., and M
´
emin, E. (2007). Dynam-
ically consistent optical flow estimation. In Proceed-
ings of International Conference on Computer Vision,
Rio de Janeiro, Brazil.
Sakov, P. and Oke, P. R. (2008). A deterministic formula-
tion of the ensemble Kalman filter: An alternative to
ensemble square root filters. Tellus.
Sun, D., Roth, S., and Black, M. (2010). Secrets of optical
flow estimation and their principles. In Proceedings
of European Conference on Computer Vision, pages
2432–2439.
Titaud, O., Vidard, A., Souopgui, I., and Le Dimet, F.-X.
(2010). Assimilation of image sequences in numerical
models. Tellus A, 62:30–47.
AnImage-basedEnsembleKalmanFilterforMotionEstimation
445
