VIBRO-VISUAL IMAGE FEATURE EXTRACTION WITH
CORRELATION IMAGE SENSOR
Circular and Doubly Circular Vibration for Arbitrary Complex Differentials
Shigeru Ando and Toru Kurihara
Department of Information Physics and Computing, University of Tokyo, Tokyo, Japan
Keywords:
Optical Flow Equation, Weighted Integral Method, Modulated Imaging, Correlation Image Sensor, Computer
Vision, Velocity Field Measurement, Particle Image Velocimetry.
Abstract:
In VISAPP 2009, we showed that exact optical flow can be determined using only a single pixel and frame
of the correlation image sensor(Ando et al., 2009). In this paper, using the newest device of it, we present
a theory and experimental evaluation of a bio-inspired vibro-visual correlation imager with various feature
extraction capability. Mimicking the involuntary movement (microsaccade) of human eyes, it vibrates rapidly
and finely a mirror in its visual axis so as to generate an equivalent vibration of every pixel in a doubly circular
locus. The time-varying intensity is captured by a correlation image sensor (CIS) with synchronous reference
signals to the vibration, and complex first/second order differentials and Laplacian are obtained as the image
features. General theoretical foundations and an implementation result of this system using a novel 640×512
pixel device are presented. Several experimental results using it including a realtime control of resolution and
edge detection from a combined use of the first and second order differentials are shown.
1 INTRODUCTION
For image recognition systems, tasks for extracting
brightness gradient, edges, corners, ridges, etc. and
localizing them accurately are extremely important.
Almost all methods proposed so far are for an im-
age array that has captured already by a sensing de-
vice. In those methods, reduction of various noises
and artifacts caused by the sensing device and spatio-
temporal integration/sampling through it has been
treated as one of major subjects for achieving a sat-
isfactory performance (Ando, 2000b; Ando, 2000a).
The spatio-temporal sampling/quantizationbefore ex-
tracting those structures by conventional image sen-
sors is often a significant obstacle to perform desired,
detailed, and accurate analysis of them.
The goal of our study is the realization of vibro-
visual imaging system mimicking involuntary eye
movements of human vision that can extract impor-
tant image features during the capturing process of
continuous intensity distribution (Ando et al., 2002;
Hontani et al., 1999; Hontani et al., 2002). The
involuntary eye movements are the small and per-
petual vibration of eyeball when the human vision
gazes at an object. When the image sensor is vi-
brated in a period sufficiently shorter than the frame
interval, the continuous intensity distribution on its
surface is counter-vibratorily shifted. This causes
a time-varying incident light, i.e., temporal modu-
lation, on the pixel according to the local structure
of the intensity distribution around the pixel. The
temporal modulation based sensing scheme of spa-
tial structures, being free from spatio-temporal sam-
pling/quantization error, have been studied in various
areas (Tang, 1978; Ikuta, 1985; Storrs and Mehrl,
1994; Hlyo and Samms, 1986; Wang et al., 1997;
Hongler et al., 2003), but most of them are for point-
by-point sensing. Our study is different in the use of
parallel imaging/demodulation device, i.e., the corre-
lation image sensor (Ando and Kimachi, 2003; Ando
et al., 2007; Han et al., 2010) for this purpose. In this
paper, we present theoretical foundation and experi-
mental evaluation of a bio-inspired vibro-visual cor-
relation imager with various feature extraction capa-
bility. Novel doubly circular vibration and simultane-
ous three frequency demodulation schemes are intro-
duced. Theoretical foundation is constructed by using
complex differential (d-bar) operator theory.
186
Ando S. and Kurihara T..
VIBRO-VISUAL IMAGE FEATURE EXTRACTION WITH CORRELATION IMAGE SENSOR - Circular and Doubly Circular Vibration for Arbitrary
Complex Differentials.
DOI: 10.5220/0003838701860191
In Proceedings of the International Conference on Computer Vision Theory and Applications (VISAPP-2012), pages 186-191
ISBN: 978-989-8565-03-7
Copyright
c
 2012 SCITEPRESS (Science and Technology Publications, Lda.)

2 THEORY
2.1 Complex Differential Operator
Let us introduce a coordinate transform
(x,y) → (z,z
∗
) = (x+ jy,x− jy) (1)
in the image plane (x,y) (j: imaginary unit). Any
function f of (x,y) can be expressed equivalently us-
ing the coordinate (z,z
∗
) as f(z,z
∗
). The differen-
tials in (z, z
∗
) are also expressed as(Brandwood, 1983;
van den Bos, 1994)
∂
∂z
∗
≡
1
2
(
∂
∂x
+ j
∂
∂y
),
∂
∂z
≡
1
2
(
∂
∂x
− j
∂
∂y
). (2)
For simplicity of notation, we often abbreviate the co-
efficient 1/2. The first order complex differentials are
the complex notation of gradient vector and its conju-
gate as
∂f
∂z
∗
=
∂f
∂x
+ j
∂f
∂y
,
∂f
∂z
=
∂f
∂x
− j
∂f
∂y
,
and the second order ones are the complex second or-
der differential and Laplacian as
∂
2
f
∂(z
∗
)
2
=
∂
2
f
∂x
2
−
∂
2
f
∂y
2
+ 2j
∂
2
f
∂x∂y
∂
2
f
∂z∂z
∗
=
∂
2
f
∂x
2
+
∂
2
f
∂y
2
.
All of them are greatly important as primitive im-
age features for image pattern recognition and under-
standing.
By using the complex differentials, the Taylor ex-
pansion of f(ζ,ζ
∗
) around (0,0) is expressed as
f(ζ,ζ
∗
) =
∞
∑
r=0
1
r!

ζ
∂
∂z
+ ζ
∗
∂
∂z
∗

r
f(0,0)
=
∞
∑
r=0
1
r!
r
∑
k=0

r
k

ζ
k
(ζ
∗
)
r−k
∂
r
f(0,0)
∂z
k
∂(z
∗
)
r−k
.
(3)
2.2 Circular Vibratory Scan
Let us consider first a circular vibratory scan with ra-
dius ε (∼ a few pixels long) and angular velocity ω.
Let us place the origin at a pixel of interest. Then,
using the Taylor expansion formula for complex dif-
ferentials, the time-varying intensity received by the
pixel is expressed as
f(t) = f(εe
jωt
,εe
− jωt
)
=
∞
∑
r=0
ε
r
r!
r
∑
k=0

r
k

e
j(2k−r)ωt
∂
r
f(0,0)
∂z
k
∂(z
∗
)
r−k
,
(4)
in which e
j(2k−r)ωt
is the time-varying term cor-
responding to the (2k − r)th harmonic component
of the circular rotational frequency ω. The other
terms involving the complex rth order differential
∂
r
f(0,0)/∂z
k
∂(z
∗
)
r−k
is the amplitude of this har-
monic component.
e
e
-e
-e
x
y
wt
(a) circular vibration
w0-w 2w 3w-2w
frequency
amplitude
f
dz
dz
2
dz
3
dz*
dz*
2
dz*
3
-3w
(b) frequency components
Figure 1: Circular vibratory scan and its spectral encoding
of complex differentials. The dz and dz* in (b) indicate
∂/∂z and ∂/∂z
∗
, respectively.
To understand the meaning of summation on r and
k, let us illustrate visually the terms taking 2k− r (or-
der of harmonics) as horizontal axis and r (order of
amplitude ε) as vertical axis. They are then expressed
as
−4 −3 −2 −1 0 1 2 3 4
1 1
1 ε ε
1/2 ε
2
2ε
2
ε
2
1/6 ε
3
3ε
3
3ε
3
ε
3
1/24 ε
4
4ε
4
6ε
4
4ε
4
ε
4
.
Namely, for example, the 2k − r = 0 frequency com-
ponent (DC) locating at the center is constructed by a
sum of 1, ∂
2
f(0,0)/∂z∂(z
∗
), ∂
4
f(0,0)/∂z
2
∂(z
∗
)
2
,···
complex differentials with each amplitude 1, (1/2) ×
2ε
2
,(1/24) × 6ε
4
,···, respectively. For the 2k −
r = 1 frequency component (ω), it is constructed by
∂f(0,0)/∂z, ∂
3
f(0,0)/∂z
2
∂z
∗
,···. Here, let us as-
sume the radius ε of circular vibration is sufficiently
small and the intensity distribution f (z,z
∗
) is suffi-
ciently smooth. Then, the lowest order term of ε
in each frequency component can be expected to be
sufficiently larger than the higher order terms. This
means the lowest order term dominates in each fre-
quency component individually, and hence, the time-
varying intensity is expressed as
f(t) ≃
∞
∑
r=0
ε
r
r!
(
∂
r
f(0,0)
∂z
r
e
jrωt
+
∂
r
f(0,0)
∂(z
∗
)
r
e
− jrωt
)
(5)
(notice this expression is valid only under usage after
frequencydecomposition). This shows that the ampli-
tude and phase of frequency component of rω is just
VIBRO-VISUAL IMAGE FEATURE EXTRACTION WITH CORRELATION IMAGE SENSOR - Circular and Doubly
Circular Vibration for Arbitrary Complex Differentials
187

equal to rth order complex differential. For positive
frequency components, we can obtain the complex
differential of the same order as the complex ampli-
tude of the harmonics. For negative frequency com-
ponents, we can obtain the conjugate complex differ-
ential.
A significant drawback of this vibrating scheme is
that complex differentials composed of both ∂/∂z and
∂/∂z
∗
such as Laplacian can not be obtained as the
dominant term of frequency component. A solution
to this problem will be given in the next section.
2.3 Doubly Circular Vibratory Scan
As an extension, the doubly circular vibration is the
scan shown in Fig. 2(a) along the locus
(z,z
∗
) = (E e
jΩt
+ εe
jωt
,E e
− jΩt
+ εe
− jωt
)
where Ω ≪ ω and the radii ε and E are both small
enough. Therefore the Taylor expansion is expressed
as
f(t) =
∞
∑
r=0
1
r!
r
∑
k=0

r
k

×(E e
jΩt
+ εe
jωt
)
k
(E e
− jΩt
+ εe
− jωt
)
r−k
×
∂
r
∂z
k
∂(z
∗
)
r−k
f(0,0).
Using a similar argument as in the previous section,
we obtain an expression of the time-varying intensity
as
f(t) ≃
∞
∑
r=0
∞
∑
s=0
ε
r
r!
E
s
s!
{e
j(rω+sΩ)t
∂
r+s
∂z
r+s
f(0,0)
+e
j(rω−sΩ)t
∂
r+s
∂z
r
∂(z
∗
)
s
f(0,0)
+e
j(−rω+sΩ)t
∂
r+s
∂(z
∗
)
r
∂z
s
f(0,0)
+e
j(−rω−sΩ)t
∂
r+s
∂(z
∗
)
r+s
f(0,0)}, (6)
which shows that the amplitude and phase of fre-
quency component of rω + sΩ is just equal to the
complex differential ∂
r+s
f/∂u
r
∂v
s
where u,v ≡ z
when r,s > 0 and u,v ≡ z
∗
when r,s < 0. The spectral
distribution is shown in Fig. 2(b).
3 SYSTEM AND EXPERIMENTS
3.1 Optical Setup
The photographs of the setup are shown in Figs. 3
(a) and (b). The doubly circular vibration of optical
x axis
-1.50 -1.00 -0.50 0.00 0.50 1.00 1.50
y axis
-1.50 -1.00 -0.50 0.00 0.50 1.00 1.50
(a) locus of a doubly circular scan
wW
0
w+Ww-W2W 3W w+2Ww-2W
frequency
amplitude
f
dz
dz
2
dz
3
dz
dz
2
dz
3
dzdz*=D
dzdz*
2
(b) frequency components
Figure 2: Doubly circular vibratory scan and its spectral
encoding of complex differentials. (a) ε = 0.3, E = 0.8,
and ω/Ω = 18, (b) spectral components, dz and dz* indicate
∂/∂z and ∂/∂z
∗
, respectively.
axis was given by a 2-D angularly oscillating mirror
driven directly synchronously by the camera. It is in-
serted in front of the imaging lens in 45 deg angle
to capture the object in 90 deg angle from the cam-
era. The frame frequency was chosen as 12Hz to
avoid an interference with ambient light frequencies
(50Hz and 100Hz). The driving signals are a mixture
of sinusoidal waves with frequencies Ω/2π = 12Hz
and ω/2π = 48Hz. The amplitudes of them can
be changed freely during the capturing operation by
changing the volume control of audio-amplifiers used
to drive the oscillating mirror.
3.2 Experimental Results
a) Extraction of Complex 1st and 2nd Differentials
Examples of the results are shown in Figs.4 (a), (b),
and (c). Three images display the intensity, complex
gradient, and complex 2nd-order differential fields of
a moving doll. The phase of the complex image is
shown by the hue of color image. In (a), the dou-
bly circular vibration locus is captured in a specularly
highlighted part. The complex 1st order differential is
equivalent to the gradient vector. So, it can be used
to extract edges or to analyze orientational features
of texture. The complex 2nd order differential shows
zero-cross lines at edges and the phase (double of the
edge angle) is inverted across the line. Therefore, it
VISAPP 2012 - International Conference on Computer Vision Theory and Applications
188

(a) intensity (b) complex gradient (c) complex 2nd order differential
(d) intensity (e) complex gradient (f) Laplacian
Figure 4: Example of experimental results. (a) and (d) show the intensity images, (b) and (c), respectively, show the complex
first and second differential image for (a), and (f) shows the Laplacian image for (a). The phase in [0,2π] is represented by
colors from red to violet.
(a) setup (b) CIS and vibrating mirror
Figure 3: Photographs of the experimental setup and vi-
brating mirror. The vibration is given by the 2-axis angu-
larly vibrating mirror between the object and the camera.
The frame frequency is 12Hz, the vibration frequencies are
Ω/2π = 12Hz and ω/2π = 48Hz.
can be used to extract and localize edges accurately.
b) Extraction of Complex 1st Differential and
Laplacian
Examples of the results are shown in Figs.4 (d), (e),
and (f). In (e), strength and orientation of edges and
ridges of a resolution chart are displayed. (f) shows
the Laplacian. It is displayed in only two hues corre-
sponding to opposite signs because the Laplacian for
an real-valued image is always real. The amplitude of
Laplacian becomes higher where the brightness cur-
vature is large such as rising and falling lines of edges
or top of ridges. Those features are captured signifi-
cantly clearly.
c) Vibration Amplitude vs. Feature Response
Fig. 5 (a) to (d) show the result. When the vibration
amplitude is very small as shown in (a), the image
captures very detailed features smaller than the pixel
interval. According to the increase of the vibration
amplitude from (a) to (c), the captured images tend to
respond to smoothed low frequencyfeatures more and
more. This property is desirable for the realtime adap-
tive processing of image scenes. In (d), the vibration
amplitude is maximized. Although detailed features
are eliminated, image captures with very high signal-
to-noise ratio the overall orientational features, e.g.,
edges and/or textures, of object and background.
d) Application to Edge Detection
Figs. 6 (a) and (b) show an example of edge detection
from complex differentials. Two set of reference sig-
nals with frequencies Ω/2π = 12Hz and Ω/π = 24Hz
were supplied. (a) shows the intensity image. Its blur
is caused by the doubly circular vibration. (b) shows
the edge image extracted from the zero-cross of the
complex 2nd order differential. The color indicates
the edge orientation determined by the phase of com-
plex gradient image captured simultaneously.
VIBRO-VISUAL IMAGE FEATURE EXTRACTION WITH CORRELATION IMAGE SENSOR - Circular and Doubly
Circular Vibration for Arbitrary Complex Differentials
189

(a) low vibration amplitude (b) middle vibration amplitude
(c) high vibration amplitude (d) maximum vibration amplitude
Figure 5: Complex gradient images under gradually increasing level of vibration amplitude. In (a), features smaller than the
pixel interval are extracted. The phase in [0,2π] is represented by a color from red to violet.
(a)
(b)
Figure 6: Example of edge detection from complex differ-
entials. (a) and (b), respectively, show the intensity image
and the edge image extracted from the zero-cross of the
complex 2nd order differential. The color indicates the edge
orientation.
4 CONCLUSIONS
Theoretical foundation and experimental evaluation
of a bio-inspired vibro-visual correlation imager were
presented. Implementation and several experimental
results of this system using a novel 640×512 pixel
device are presented including an application to edge
detection. Although the performance of correlation
image sensor are in a depeloping level, the sensi-
tivity and accuracy of image features captured by it
were exceptionally good. Therefore, we consider the
proposed vibro-visual imaging system is sufficiently
promising for future applications.
REFERENCES
Ando, S. (2000a). Consistent gradient operators. In IEEE
Trans. PAMI, vol.22, no.3, pp.252-265.
Ando, S. (2000b). Image field categorization and
edge/corner detection from gradient covariance. In
IEEE Trans. PAMI, vol.22, no.2, pp.179-190.
Ando, S. and Kimachi, A. (2003). Correlation image sen-
sor: Two-dimensional matched detection of amplitude
modulated light. In IEEE Trans. Electron Devices,
vol.50, no.10, pp.2059-2066.
Ando, S., Kurihara, T., and Wei, T. (2009). Exact algebraic
VISAPP 2012 - International Conference on Computer Vision Theory and Applications
190

method of optical flow detection via modulated inte-
gral imaging - theoretical formulation and real-time
implementation using correlation image sensor –. In
Proc. VISAPP 2009, pp.480-487, Lisboa.
Ando, S., Nara, T., Ono, N., and Kurihara, T. (2007). Real-
time orientation-sensitive magneto-optic imager for
leakage flux inspection. In IEEE Trans. Magnetics,
vol.43, no.3, pp.1044-1051.
Ando, S., Ono, N., and Kimachi, A. (2002). Involuntary
eye-movement vision based on three-phase correla-
tion image sensor. In Proc. 19th Sensor Symp., pp.83-
86, Kyoto.
Brandwood, D. H. (1983). A complex gradient operator and
its application in adaptive array theory. In IEE Proc.
vol.130, no.1, pp.11-16.
Han, S., Sawada, T., Iwahori, T., Kawahito, S., and Ando,
S. (2010). Three-phase time-correlation image sensor
using pinned photodiode active pixels. In IS&T/SPIE
Electronic Imaging, 7536-28.
Hlyo, N. and Samms, R. W. (1986). Combined optimiza-
tion of image-gathering optics and image-processing
algorithm for edge detection. In J. Opt. Soc. Am., vol.
3, no. 9, pp.1522-1536.
Hongler, M., de Meneses, Y. L., Beyeler, A., and Jacot, J.
(2003). The resonant retina: exploiting vibration noise
to optimally detect edges in an image. In IEEE Trans.
Pattern Anal. Machine Intell., vol.25, no.9, pp.1051-
1062.
Hontani, H., Kimachi, A., and Ando, S. (1999). Involuntary
eye movement vision and its applications. In Trans.
IPSJ, CVIM, 118-2.
Hontani, H., Shibata, J., Kimachi, A., and Ando, S. (2002).
Vibratory image feature extraction based on local log-
polar symmetry. In Trans. IPJ, vol.43, no.7, pp.2309-
2318.
Ikuta, T. (1985). Active image processing. In Applied Op-
tics, vol. 24, no.18, pp.2907-2913.
Storrs, M. and Mehrl, D. J. (1994). Detection of spa-
tial derivatives of images using spatiotemporal tech-
niques. In Optical Engineering, vol. 33, no. 9,
pp.3072-3081.
Tang, C. L. (1978). Dithered-beam metrology. In Applied
Optics, vol.17, pp.3865-3868.
van den Bos, A. (1994). Complex gradient and hessian. In
IEE Proc. Vis. Image Signal Process., vol.141, no.6,
pp.380-382.
Wang, C., Ni, Y., and Devos, F. (1997). A spatio-temporal
differentiation light sensor. In Sensor and Actuators
A, vol. 62, pp.492-495.
VIBRO-VISUAL IMAGE FEATURE EXTRACTION WITH CORRELATION IMAGE SENSOR - Circular and Doubly
Circular Vibration for Arbitrary Complex Differentials
191