CONTENT-BASED IMAGE RESIZING ON MOBILE DEVICES
Sebastiano Battiato, Giovanni Maria Farinella, Nicol
`
o Grippaldi and Giovanni Puglisi
Image Processing Laboratory, Department of Mathematics and Computer Science, University of Catania, Catania, Italy
Keywords:
Content-aware Image Resizing, Computer Vision on Mobile Devices.
Abstract:
Content-aware image resizing are effective algorithms that allow to take into account the visual content of
images during the resizing process. Despite the technological advances in the context of mobile devices,
content-aware image resizing algorithms are still far to be used on a hand held device due to the computational
resources needed during the resizing. In this paper we afford this problem employing a method which has
linear complexity with respect to the number of lines (rows/columns) to be reduced/augmented. The method
has been tested, both qualitatively and quantitatively, on a mobile platform.
1 INTRODUCTION
The extensive use of different display devices with
different resolution increases the demand of image re-
sizing techniques which consider the visual content
during the resizing process. Standard resizing tech-
niques, such as scaling, can be used only to change
the size of an image of a fixed proportion with respect
to the two dimensions. Scaling does not take into ac-
count the visual importance of pixels during image re-
sizing (i.e., a resizing with respect to only one of the
two dimensions introduces artifacts and distortions).
Other standard operations, such as cropping, in which
outer parts of an image are removed, could produce
images with loss of semantic information. In the last
five years, different techniques for content-aware im-
age resizing have been proposed (Avidan and Shamir,
2007; Cho et al., 2008; Rubinstein et al., 2009; Gal-
lea et al., 2010). The main aim of a content-aware
image resizing is the preservation of relevant visual
information into the resized image. Seam Carving
(Avidan and Shamir, 2007) reduces or expands the
image by removing or duplicating connected path of
pixels (called seams) having low-energy in the energy
map corresponding to the image to be resized. Ru-
binstein et al. (Rubinstein et al., 2009) presented an
image resizing algorithm to perform combination of
Bi-cubic scaling, cropping and Seam Carving. With
this approach better results are obtained in terms of
visual quality, but the computational complexity in-
creases due to the use of different resizing opera-
tors. Among others, patch-based methods have been
also proposed for image retargeting or summariza-
tion. Cho et al. (Cho et al., 2008) suggested an al-
gorithm to find an arrangement of patches of the orig-
inal image that well fit in the resized image. Although
the techniques above produce impressive results, the
corresponding algorithms have a high computational
complexity to be employed in consumer mobile de-
vices. Gallea et al. (Gallea et al., 2010) proposed a
fast method for image retargeting based on the solu-
tion of a linear system. This model aims to find shift
values for each line (row/column) preserving the dis-
tance among the relevant ones. The linearity of the
considered model allows them to elaborate even large
images in reasonable computational time. Building
on the technique proposed by Gallea et al. (Gallea
et al., 2010), in this paper we propose a method to be
employed for content-aware image resizing on mo-
bile devices. The method has linear complexity with
respect to the number of lines (rows or columns) to be
reduced/augmented. The linear complexity makes the
approach attractive for mobile devices environment.
In this paper we have performed a set of experiments
on Nokia N900 mobile platform (Adams et al., 2010)
to evaluate both qualitative and quantitative results of
this last method. The paper is organized as follows:
Section 2 presents the image resizing method. In Sec-
tion 3 the experimental phase and the results are de-
tailed. Finally, conclusions are given in Section 4.
2 PROPOSED APPROACH
As already stated in the previous section, although the
recent improvement both in terms of memory stor-
age and computational capability of mobile devices,
87
Battiato S., Farinella G., Grippaldi N. and Puglisi G..
CONTENT-BASED IMAGE RESIZING ON MOBILE DEVICES.
DOI: 10.5220/0003843500870090
In Proceedings of the International Conference on Computer Vision Theory and Applications (VISAPP-2012), pages 87-90
ISBN: 978-989-8565-04-4
Copyright
c
2012 SCITEPRESS (Science and Technology Publications, Lda.)
currently, only fast and simplified strategies can be
actually employed on mobile devices. The proposed
approach has been hence derived from Gallea et al.
(Gallea et al., 2010) through some simplifications and
a properly tuning of the involved steps.
The proposed model considers the image as a set
of lines L = [l
1
, l
2
, . . . , l
n
] where l
i
represents a single
line and n is the dimension of the rows or columns of
the original image to be resized. Our approach aims to
find a novel set of lines L
0
= [l
0
1
, l
0
2
, . . . , l
0
n
0
], where n
0
is
the desiderate final image dimension, obtained from
the original set L by removing (or adding in case of
image enlargement) some lines without introducing,
if possible, image distortions.
The selection of the lines to be removed is hence
a fundamental step of our approach. A careful se-
lection, considering non-informative regions, should
preserve the overall quality of the final image. A
significance map is then built using a measure based
on visual salience (Itti et al., 1998) and gradient in-
formation. A weight w
i
is associated to each line
l
i
through the projection along the considered line
by using simple operators (i.e., mean, max, min).
Note that in our tests (see Section 3) several strategies
have been considered and compared to properly find
a trade-off between final image quality and complex-
ity of the approach. Starting from the set of weights
W = [w
1
, w
2
, . . . w
n
] two different strategies of line re-
moval (insertion) have been designed. The simplest
one removes (or replicates) the |n
0
n| lines corre-
sponding to the lowest weights (hence less “signifi-
cant” in terms of salience and gradient information).
The second strategy considers the removal (replica-
tion) step as an iterative process. For each iteration
it removes (or replicates) the less “significant” line
based on its weight value and then updates the whole
map of importance of pixels and the set of weights W.
Experiments results (see Section 3) show the differ-
ence between the above mentioned strategies both in
terms of visual quality (artifacts generation) and com-
putational time on a mobile device.
3 EXPERIMENTAL RESULTS
The experiments performed to assess both qualita-
tive and quantitative results are reported in the fol-
lowing subsections. Specifically, the qualitative per-
formances have been tested by employing the pro-
posed algorithms and several energy maps on a PC
with a processor Intel Core 2 Duo T5750 2 Ghz and
3 GB RAM. Quantitative performances in terms of
computational time have been obtained testing the
approaches on a mobile phone Nokia N900. The
(a) Original Image
(b) Non-Iterative approach (c) Iterative approach
Figure 1: Results obtained considering both non-iterative
(Figure 1(b)) and iterative (Figure 1(c)) approaches. The
original image shown in Figure1(a) is reduced of 20%.
considered mobile platform has an OMAP Proces-
sor with 600 Mhz of clock frequency and 1 GB
RAM. The FCam API (Adams et al., 2010) have
been used to implement the proposed resizing tech-
niques on the Nokia N900 mobile platform. These
API are open source and can be used to develop al-
gorithms and computational photography apps (Bat-
tiato et al., 2012) taking into account the new compu-
tational cameras paradigm (Adams et al., 2010).
3.1 Qualitative Evaluation
The most efficient energy map (in terms of computa-
tional costs) we have considered makes use of Sobel
filter (only). This energy map is able to take into ac-
count discontinuities (e.g., edges). With regards to the
algorithm to be used to remove a fixed number of lines
(row or columns) once the energy map is computed,
we compared two different variants of the original al-
gorithm proposed in (Gallea et al., 2010):
a non-iterative approach in which the energy map
is computed only once and then the k less signif-
icant lines, with respect to the computed energy
map, are removed (or inserted).
an iterative approach in which at each step just
one line is removed (or inserted) and the energy
map in computed on the image obtained from the
previous iteration.
VISAPP 2012 - International Conference on Computer Vision Theory and Applications
88
Although the non-iterative approach provides a
relatively low computational time, it does not obtain
satisfactory results. Indeed, the resized images con-
tain artifacts as shown by the red arrows in Figure
1(b). The quality of the resized images increases con-
sidering the iterative algorithm, since at each step the
saliency map takes into account the changes made by
previous iterations (see Figure 1(c)). Of course, a
drawback of the last approach is that the computation
time grows as the number of iterations. To deal with
this problem, rather than computing the energy map
at each iteration we compute it locally only on the
area which have been changed in the previous iter-
ation (neighborhood of the line removed/added) and
retain the other part from the previous energy map.
This modification improves the computational perfor-
mance (time and space) obtaining exactly the same
results in terms of visual quality.
Even using algorithm simplifications described
above, in the first step of the iterative process the en-
ergy map must be calculated on the entire original im-
age and this involves a considerable slowdown in the
execution of the algorithm when the image size is very
large. To overcome this problem, rather than choos-
ing the line to be removed (added) taking into account
the energy map computed on the original image, we
proceed with a multiscale approach in which a low
resolution version of the image is created with a three
levels pyramid of Gaussian and then the choice of the
line is propagated locally from the lowest resolution
to the highest resolution energy map. In this way, in
the highest levels of the pyramid we search only in a
neighborhood.
Further tests have been done to improve the qual-
ity of the resized image in the neighborhood of the
removed/added lines. In particular, we compared two
version of removal: i) a brutal removal of the less im-
portant line chosen by the algorithm and ii) the one
on which the two lines adjacent to the one to be re-
moved are averaged with the removed line in the re-
sized image. A visual comparison between these two
approaches is reported in Figure 2. As one can see
from the figure, the second approach reduces the num-
ber of artifacts in the final resized image.
3.2 Quantitative Evaluation
In order to properly evaluate the portability of the
aforementioned alternatives on mobile devices, sev-
eral quantitative tests to study their impact in terms
of computational time have been performed. Figure 3
shows the computational time on a standard PC of the
different versions of the resizing algorithm: iterative,
iterative considering neighborhood, iterative consid-
(a) Original Image
(b) Brutal removal.
(c) Removal with averaging.
(d) Gallea et al. (Gallea et al., 2010)
Figure 2: Results obtained by reducing the image in (a) at
80% of the original horizontal size. In (b) and (c) the output
obtained considering the two versions of proposed method
for line removal. In (d) the final image obtained employing
the method proposed by Gallea et al. (Gallea et al., 2010).
CONTENT-BASED IMAGE RESIZING ON MOBILE DEVICES
89
145200 187500 480000 1228800 2666000 5668704
0
5
10
15
20
25
30
35
40
45
50
55
60
65
70
75
80
85
90
Sobel Energy Map
Number of pixels
Computational time (min)
NonIterative
Iterative considering neighborhood and multiscale
Iterative considering neighborhood
Iterative
Figure 3: Execution time on a standard PC of the proposed
approaches vs. image size. Sobel has been chosen as energy
map.
145200 187500 480000 1228800 2666000 5668704
0
5
10
15
20
25
30
35
40
45
50
55
60
65
70
75
80
85
90
NonIterative Approach
Number of Pixels
Computational time (min)
Itti et al. Energy Map
Simplified Itti et al. Energy Map
Gaussian Energy Map
Sobel Energy Map
Figure 4: Execution time on a Nokia N900 of the non-
iterative approach vs. image size. Each line represents the
results obtained with different energy map: Itti et al. (blue),
simplified Itti et al. (red), Gaussian (green), Sobel (black).
ering both neighborhood and multiscale, and non-
iterative. These tests have been performed consider-
ing the Sobel energy map. All the approaches, ob-
viously, increase their computational time at increas-
ing of the image size. Although the non-iterative ap-
proach considerably outperforms the others, it shows
lower performance in terms of artifact generation as
pointed out in previous section. On the contrary,
the optimized iterative approaches (considering both
neighborhood and multiscale) sensitively reduce the
execution time with respect to the original iterative
approach without affecting the visual quality of the
resized image. From the analysis performed through
the quantitative tests on a standard PC, and consider-
ing the limitations in terms of CPU clock frequency
and memory storage of the mobile devices, only the
non-iterative approach has been considered for final
implementation on the Nokia N900 platform (Adams
et al., 2010; Battiato et al., 2012). Figure 4 shows the
execution time of the non-iterative algorithm running
on a Nokia N900 at increasing of the image size. Sev-
eral energy maps have been compared. Considering
all the optimizations both in terms of energy map (So-
bel) and algorithm (non-iterative), the approach takes
about 5 seconds on a Nokia N900 to resize an image
of about 5 megapixels.
4 CONCLUSIONS
In this paper we have proposed several strategies to
allow content-aware image resizing on mobile de-
vices. All the involved steps have been considered
and optimized in order to find a good trade-off be-
tween visual quality and computational complexity.
Specifically, several energy maps (Itti et al., simpli-
fied Itti et al., Gaussian, Sobel) and different versions
of the line removal algorithm (iterative, iterative with
neighborhood, iterative with neighborhood and mul-
tiscale, non-iterative) have been exploited. Moreover,
to properly compare the proposed solutions in terms
of visual quality and computational load, several tests
on a standard PC and on a Nokia N900 mobile phone
have been also performed. Future works will be de-
voted to find additional smart strategies useful to cut
down the computational time in order to employ the
iterative approaches on mobile devices.
REFERENCES
Adams, A., Talvala, E.-V., Park, S. H., Jacobs, D. E., Ajdin,
B., Gelfand, N., Dolson, J., Vaquero, D., Baek, J.,
Tico, M., Lensch, H. P. A., Matusik, W., Pulli, K.,
Horowitz, M., and Levoy, M. (2010). The franken-
camera: an experimental platform for computational
photography. ACM Trans. Graph., 29:29:1–29:12.
Avidan, S. and Shamir, A. (2007). Seam carving for
content-aware image resizing. ACM Transaction on
Graphics, 26(3).
Battiato, S., Farinella, G. M., Messina, E., Puglisi, G., Rav
`
ı,
D., Capra, A., and Tomaselli, V. (2012). On the per-
formances of computer vision algorithms on mobile
platforms. In SPIE Electronic Imaging - Digital Pho-
tography VIII.
Cho, T., Butman, M., Avidan, S., and Freeman, W. (2008).
The patch transform and its applications to image edit-
ing. In IEEE International Conference on Computer
Vision and Pattern Recognition (CVPR).
Gallea, R., Ardizzone, E., and Pirrone, R. (2010). Real-time
content-aware resizing using reduced linear model. In
IEEE International Conference on Image Processing.
Itti, L., Koch, C., and Niebur, E. (1998). A model of
saliency-based visual attention for rapid scene anal-
ysis. IEEE Transactions on Pattern Analysis and Ma-
chine Intelligence, 20(11).
Rubinstein, M., Shamir, A., and Avidan, S. (2009). Multi-
operator media retargeting. ACM Transaction on
Graphics, 28(3).
VISAPP 2012 - International Conference on Computer Vision Theory and Applications
90