CONTENT-BASED IMAGE RESIZING ON MOBILE DEVICES

Sebastiano Battiato, Giovanni Maria Farinella, Nicol

o Grippaldi and Giovanni Puglisi

Image Processing Laboratory, Department of Mathematics and Computer Science, University of Catania, Catania, Italy

Keywords:

Content-aware Image Resizing, Computer Vision on Mobile Devices.

Abstract:

Content-aware image resizing are effective algorithms that allow to take into account the visual content of

images during the resizing process. Despite the technological advances in the context of mobile devices,

content-aware image resizing algorithms are still far to be used on a hand held device due to the computational

resources needed during the resizing. In this paper we afford this problem employing a method which has

linear complexity with respect to the number of lines (rows/columns) to be reduced/augmented. The method

has been tested, both qualitatively and quantitatively, on a mobile platform.

1 INTRODUCTION

The extensive use of different display devices with

different resolution increases the demand of image re-

sizing techniques which consider the visual content

during the resizing process. Standard resizing tech-

niques, such as scaling, can be used only to change

the size of an image of a ﬁxed proportion with respect

to the two dimensions. Scaling does not take into ac-

count the visual importance of pixels during image re-

sizing (i.e., a resizing with respect to only one of the

two dimensions introduces artifacts and distortions).

Other standard operations, such as cropping, in which

outer parts of an image are removed, could produce

images with loss of semantic information. In the last

ﬁve years, different techniques for content-aware im-

age resizing have been proposed (Avidan and Shamir,

2007; Cho et al., 2008; Rubinstein et al., 2009; Gal-

lea et al., 2010). The main aim of a content-aware

image resizing is the preservation of relevant visual

information into the resized image. Seam Carving

(Avidan and Shamir, 2007) reduces or expands the

image by removing or duplicating connected path of

pixels (called seams) having low-energy in the energy

map corresponding to the image to be resized. Ru-

binstein et al. (Rubinstein et al., 2009) presented an

image resizing algorithm to perform combination of

Bi-cubic scaling, cropping and Seam Carving. With

this approach better results are obtained in terms of

visual quality, but the computational complexity in-

creases due to the use of different resizing opera-

tors. Among others, patch-based methods have been

also proposed for image retargeting or summariza-

tion. Cho et al. (Cho et al., 2008) suggested an al-

gorithm to ﬁnd an arrangement of patches of the orig-

inal image that well ﬁt in the resized image. Although

the techniques above produce impressive results, the

corresponding algorithms have a high computational

complexity to be employed in consumer mobile de-

vices. Gallea et al. (Gallea et al., 2010) proposed a

fast method for image retargeting based on the solu-

tion of a linear system. This model aims to ﬁnd shift

values for each line (row/column) preserving the dis-

tance among the relevant ones. The linearity of the

considered model allows them to elaborate even large

images in reasonable computational time. Building

on the technique proposed by Gallea et al. (Gallea

et al., 2010), in this paper we propose a method to be

employed for content-aware image resizing on mo-

bile devices. The method has linear complexity with

respect to the number of lines (rows or columns) to be

reduced/augmented. The linear complexity makes the

approach attractive for mobile devices environment.

In this paper we have performed a set of experiments

on Nokia N900 mobile platform (Adams et al., 2010)

to evaluate both qualitative and quantitative results of

this last method. The paper is organized as follows:

Section 2 presents the image resizing method. In Sec-

tion 3 the experimental phase and the results are de-

tailed. Finally, conclusions are given in Section 4.

2 PROPOSED APPROACH

As already stated in the previous section, although the

recent improvement both in terms of memory stor-

age and computational capability of mobile devices,

Battiato S., Farinella G., Grippaldi N. and Puglisi G..

CONTENT-BASED IMAGE RESIZING ON MOBILE DEVICES.

DOI: 10.5220/0003843500870090

In Proceedings of the International Conference on Computer Vision Theory and Applications (VISAPP-2012), pages 87-90

ISBN: 978-989-8565-04-4

 2012 SCITEPRESS (Science and Technology Publications, Lda.)

currently, only fast and simpliﬁed strategies can be

actually employed on mobile devices. The proposed

approach has been hence derived from Gallea et al.

(Gallea et al., 2010) through some simpliﬁcations and

a properly tuning of the involved steps.

The proposed model considers the image as a set

of lines L = [l

, l

, . . . , l

] where l

represents a single

line and n is the dimension of the rows or columns of

the original image to be resized. Our approach aims to

ﬁnd a novel set of lines L

= [l

, l

, . . . , l

], where n

the desiderate ﬁnal image dimension, obtained from

the original set L by removing (or adding in case of

image enlargement) some lines without introducing,

if possible, image distortions.

The selection of the lines to be removed is hence

a fundamental step of our approach. A careful se-

lection, considering non-informative regions, should

preserve the overall quality of the ﬁnal image. A

signiﬁcance map is then built using a measure based

on visual salience (Itti et al., 1998) and gradient in-

formation. A weight w

is associated to each line

through the projection along the considered line

by using simple operators (i.e., mean, max, min).

Note that in our tests (see Section 3) several strategies

have been considered and compared to properly ﬁnd

a trade-off between ﬁnal image quality and complex-

ity of the approach. Starting from the set of weights

W = [w

, w

, . . . w

] two different strategies of line re-

moval (insertion) have been designed. The simplest

one removes (or replicates) the |n

− n| lines corre-

sponding to the lowest weights (hence less “signiﬁ-

cant” in terms of salience and gradient information).

The second strategy considers the removal (replica-

tion) step as an iterative process. For each iteration

it removes (or replicates) the less “signiﬁcant” line

based on its weight value and then updates the whole

map of importance of pixels and the set of weights W.

Experiments results (see Section 3) show the differ-

ence between the above mentioned strategies both in

terms of visual quality (artifacts generation) and com-

putational time on a mobile device.

3 EXPERIMENTAL RESULTS

The experiments performed to assess both qualita-

tive and quantitative results are reported in the fol-

lowing subsections. Speciﬁcally, the qualitative per-

formances have been tested by employing the pro-

posed algorithms and several energy maps on a PC

with a processor Intel Core 2 Duo T5750 2 Ghz and

3 GB RAM. Quantitative performances in terms of

computational time have been obtained testing the

approaches on a mobile phone Nokia N900. The

(a) Original Image

(b) Non-Iterative approach (c) Iterative approach

Figure 1: Results obtained considering both non-iterative

(Figure 1(b)) and iterative (Figure 1(c)) approaches. The

original image shown in Figure1(a) is reduced of 20%.

considered mobile platform has an OMAP Proces-

sor with 600 Mhz of clock frequency and 1 GB

RAM. The FCam API (Adams et al., 2010) have

been used to implement the proposed resizing tech-

niques on the Nokia N900 mobile platform. These

API are open source and can be used to develop al-

gorithms and computational photography apps (Bat-

tiato et al., 2012) taking into account the new compu-

tational cameras paradigm (Adams et al., 2010).

3.1 Qualitative Evaluation

The most efﬁcient energy map (in terms of computa-

tional costs) we have considered makes use of Sobel

ﬁlter (only). This energy map is able to take into ac-

count discontinuities (e.g., edges). With regards to the

algorithm to be used to remove a ﬁxed number of lines

(row or columns) once the energy map is computed,

we compared two different variants of the original al-

gorithm proposed in (Gallea et al., 2010):

• a non-iterative approach in which the energy map

is computed only once and then the k less signif-

icant lines, with respect to the computed energy

map, are removed (or inserted).

• an iterative approach in which at each step just

one line is removed (or inserted) and the energy

map in computed on the image obtained from the

previous iteration.

VISAPP 2012 - International Conference on Computer Vision Theory and Applications

Although the non-iterative approach provides a

relatively low computational time, it does not obtain

satisfactory results. Indeed, the resized images con-

tain artifacts as shown by the red arrows in Figure

1(b). The quality of the resized images increases con-

sidering the iterative algorithm, since at each step the

saliency map takes into account the changes made by

previous iterations (see Figure 1(c)). Of course, a

drawback of the last approach is that the computation

time grows as the number of iterations. To deal with

this problem, rather than computing the energy map

at each iteration we compute it locally only on the

area which have been changed in the previous iter-

ation (neighborhood of the line removed/added) and

retain the other part from the previous energy map.

This modiﬁcation improves the computational perfor-

mance (time and space) obtaining exactly the same

results in terms of visual quality.

Even using algorithm simpliﬁcations described

above, in the ﬁrst step of the iterative process the en-

ergy map must be calculated on the entire original im-

age and this involves a considerable slowdown in the

execution of the algorithm when the image size is very

large. To overcome this problem, rather than choos-

ing the line to be removed (added) taking into account

the energy map computed on the original image, we

proceed with a multiscale approach in which a low

resolution version of the image is created with a three

levels pyramid of Gaussian and then the choice of the

line is propagated locally from the lowest resolution

to the highest resolution energy map. In this way, in

the highest levels of the pyramid we search only in a

neighborhood.

Further tests have been done to improve the qual-

ity of the resized image in the neighborhood of the

removed/added lines. In particular, we compared two

version of removal: i) a brutal removal of the less im-

portant line chosen by the algorithm and ii) the one

on which the two lines adjacent to the one to be re-

moved are averaged with the removed line in the re-

sized image. A visual comparison between these two

approaches is reported in Figure 2. As one can see

from the ﬁgure, the second approach reduces the num-

ber of artifacts in the ﬁnal resized image.

3.2 Quantitative Evaluation

In order to properly evaluate the portability of the

aforementioned alternatives on mobile devices, sev-

eral quantitative tests to study their impact in terms

of computational time have been performed. Figure 3

shows the computational time on a standard PC of the

different versions of the resizing algorithm: iterative,

iterative considering neighborhood, iterative consid-

(a) Original Image

(b) Brutal removal.

(d) Gallea et al. (Gallea et al., 2010)

Figure 2: Results obtained by reducing the image in (a) at

80% of the original horizontal size. In (b) and (c) the output

obtained considering the two versions of proposed method

for line removal. In (d) the ﬁnal image obtained employing

the method proposed by Gallea et al. (Gallea et al., 2010).

CONTENT-BASED IMAGE RESIZING ON MOBILE DEVICES

145200 187500 480000 1228800 2666000 5668704

Sobel Energy Map

Number of pixels

Computational time (min)

Non−Iterative

Iterative considering neighborhood and multiscale

Iterative considering neighborhood

Iterative

Figure 3: Execution time on a standard PC of the proposed

approaches vs. image size. Sobel has been chosen as energy

map.

145200 187500 480000 1228800 2666000 5668704

Non−Iterative Approach

Number of Pixels

Computational time (min)

Itti et al. Energy Map

Simplified Itti et al. Energy Map

Gaussian Energy Map

Sobel Energy Map

Figure 4: Execution time on a Nokia N900 of the non-

iterative approach vs. image size. Each line represents the

results obtained with different energy map: Itti et al. (blue),

simpliﬁed Itti et al. (red), Gaussian (green), Sobel (black).

ering both neighborhood and multiscale, and non-

iterative. These tests have been performed consider-

ing the Sobel energy map. All the approaches, ob-

viously, increase their computational time at increas-

ing of the image size. Although the non-iterative ap-

proach considerably outperforms the others, it shows

lower performance in terms of artifact generation as

pointed out in previous section. On the contrary,

the optimized iterative approaches (considering both

neighborhood and multiscale) sensitively reduce the

execution time with respect to the original iterative

approach without affecting the visual quality of the

resized image. From the analysis performed through

the quantitative tests on a standard PC, and consider-

ing the limitations in terms of CPU clock frequency

and memory storage of the mobile devices, only the

non-iterative approach has been considered for ﬁnal

implementation on the Nokia N900 platform (Adams

et al., 2010; Battiato et al., 2012). Figure 4 shows the

execution time of the non-iterative algorithm running

on a Nokia N900 at increasing of the image size. Sev-

eral energy maps have been compared. Considering

all the optimizations both in terms of energy map (So-

bel) and algorithm (non-iterative), the approach takes

about 5 seconds on a Nokia N900 to resize an image

of about 5 megapixels.

4 CONCLUSIONS

In this paper we have proposed several strategies to

allow content-aware image resizing on mobile de-

vices. All the involved steps have been considered

and optimized in order to ﬁnd a good trade-off be-

tween visual quality and computational complexity.

Speciﬁcally, several energy maps (Itti et al., simpli-

ﬁed Itti et al., Gaussian, Sobel) and different versions

of the line removal algorithm (iterative, iterative with

neighborhood, iterative with neighborhood and mul-

tiscale, non-iterative) have been exploited. Moreover,

to properly compare the proposed solutions in terms

of visual quality and computational load, several tests

on a standard PC and on a Nokia N900 mobile phone

have been also performed. Future works will be de-

voted to ﬁnd additional smart strategies useful to cut

down the computational time in order to employ the

iterative approaches on mobile devices.

REFERENCES

Adams, A., Talvala, E.-V., Park, S. H., Jacobs, D. E., Ajdin,

B., Gelfand, N., Dolson, J., Vaquero, D., Baek, J.,

Tico, M., Lensch, H. P. A., Matusik, W., Pulli, K.,

Horowitz, M., and Levoy, M. (2010). The franken-

camera: an experimental platform for computational

photography. ACM Trans. Graph., 29:29:1–29:12.

Avidan, S. and Shamir, A. (2007). Seam carving for

content-aware image resizing. ACM Transaction on

Graphics, 26(3).

Battiato, S., Farinella, G. M., Messina, E., Puglisi, G., Rav

ı,

D., Capra, A., and Tomaselli, V. (2012). On the per-

formances of computer vision algorithms on mobile

platforms. In SPIE Electronic Imaging - Digital Pho-

tography VIII.

Cho, T., Butman, M., Avidan, S., and Freeman, W. (2008).

The patch transform and its applications to image edit-

ing. In IEEE International Conference on Computer

Vision and Pattern Recognition (CVPR).

Gallea, R., Ardizzone, E., and Pirrone, R. (2010). Real-time

content-aware resizing using reduced linear model. In

IEEE International Conference on Image Processing.

Itti, L., Koch, C., and Niebur, E. (1998). A model of

saliency-based visual attention for rapid scene anal-

ysis. IEEE Transactions on Pattern Analysis and Ma-

chine Intelligence, 20(11).

Rubinstein, M., Shamir, A., and Avidan, S. (2009). Multi-

operator media retargeting. ACM Transaction on

Graphics, 28(3).

VISAPP 2012 - International Conference on Computer Vision Theory and Applications