Super Resolution for Smartphones

Seiichi Gohshi

, Sakae Inoue

, Isao Masuda

, Takashi Ichinose

and Yoshikatsu Tatsumi

Kogakuin University, 1-24-2, Nishi-Shinjuku, Shinjuku-ku, Tokyo, Japan

Fujitsu Connected Technologies Limited, 1-1, Kamikodanaka 4-chome, Nakahara-ku, Kawasaki, Japan

Keywords:

Smartphone, Real-time Video Processing, Nyquist Frequency, Super Resolution.

Abstract:

Smartphones were developed as an advanced communication tool. Currently they are used in various applica-

tions. The display is one of the most important features in smartphones. Compared with television (TV) and

cinema screens the display size of a smartphone is small. However, TV and ﬁlm content is commonly enjoyed

on smartphone screens. Currently, the smartphone display is one of the most used displays for various kinds

of content. In the past it was thought that it would be difﬁcult to recognize the resolution differences on small

displays. However, this is no longer the case. The resolution of smartphones have been steadily improving,

and high-deﬁnition television (HDTV) (1, 920× 1, 080 pixels) viewing resolution support is common. Signal

processing is another way to improve resolution. Super resolution (SR) has become an interesting research

ﬁeld and is applied to images and videos. SR is a technology for improving display resolution. Consequently,

SR is mainly studied for application to TV screens and computer displays. SR technology algorithms are com-

plex and a heavy load for a smartphone’s central or graphics processing unit (CPU/GPU). It is very difﬁcult

to apply SR for real-time videos on smartphones. Consequently, there have been no reports in SR for smart-

phones. This paper proposes a method for implementing real-time SR in smartphones. This method works for

real-time videos on a smartphone GPU with the developed software.

1 INTRODUCTION

Communication environments and devices have dra-

matically changed in the last two decades. Smart-

phones have become major devices on the mobile

phone market, and new models are introduced every

year. Smartphonesare all-in-one small computers and

come equipped with various functions. The display

on a smartphone is used as an input terminal as well

as a conventional display for video content. Smart-

phones have thus become important devices for en-

joying television (TV) and cinema content as well as

games. Smartphone manufacturers are constantly de-

veloping new products and trying to stand out from

the competition. The resolutions of smartphone mon-

itors are increasing, and high-deﬁnition TV (HDTV)

resolution (1, 920 × 1, 080 pixels) is now common.

Some of the latest smartphones now come with 4K

resolution. However the display sizes are approxi-

mately ﬁve inches. If there were an obviousresolution

quality difference between a small ﬁve-inch HDTV

display and a 4K one, it would be worthwhile to invest

further in 4K smartphone technology. However, there

has been no discussion or subjective assessments of

using 4K displays on smartphones. Moreover, none

of the content for smartphones on the Internet has

4K resolution, and some do not have HDTV resolu-

tion. Despite the resolution of the Internet content,

only the number of pixels of a display is increasing

for marketing. When the resolution of the content is

not sufﬁcient for the display, as with video graphics

array (VGA) or quarter VGA (QVGA), the content is

specially interpolated to be ﬁxed with the resolution

of the smartphone display. Interpolated images and

videos are blurry and cannot take full advantage of the

performance of a smartphone display. Although the

content has HDTV pixels, HDTV resolution is not al-

ways guaranteed because the focus is not always ﬁne.

Clearly, smartphone users prefer high-resolution im-

ages and videos. However, high-resolution displays

(HDTV/4K) do not always provide high-resolution

images and videos. Resolution of the content is more

important than the number of pixels in displays.

Improving image resolution has been a highly ac-

tive ﬁeld of research for many years. Unsharp mask-

ing (USM) also referred to as edge enhancement, has

been the only method for enhancing video in real-time

systems (Schreiber, 1970)(Lee, 1980)(Pratt, 2001).

Although USM is a simple and cost-effective method,

it does not actually improve resolution; it provides

106

Gohshi, S., Inoue, S., Masuda, I., Ichinose, T. and Tatsumi, Y.

Super Resolution for Smartphones.

DOI: 10.5220/0005991301060112

In Proceedings of the 13th International Joint Conference on e-Business and Telecommunications (ICETE 2016) - Volume 5: SIGMAP, pages 106-112

ISBN: 978-989-758-196-0

a better image quality using either a band-pass ﬁlter

(BPF) or a high-pass ﬁlter (HPF). However, USM can

introduce noise and edges to images.

Super resolution (SR) technology, which has been

studied for approximately two decades (Elad and

Feuer, 1996), creates a high-resolution image (HRI)

from low resolution images (LRIs). Various SR

technologies have been proposed during the past ten

years. However, most of these are proposed for still

images and are difﬁcult to apply to videos owing to

because of their complex algorithms (Farsiu et al.,

2004)(Park et al., 2003)(Katsaggelos et al., 2010)(van

Eekeren et al., 2010)(Panda et al., 2011)(Glasner

et al., 2009)(Park et al., 2003) (Sun et al., 2008)(Dong

et al., 2014). Recently, super resolution with non-

linear processing (SRNP) has been proposed by one

of the authors. SRNP can process video in real

time.This paper proposes SRNP for smartphones for

real-time video processing. Our method can work

with software over the central or graphic processing

unit (CPU/GPU) of a smartphone. It shows good re-

sults and improves videos to ﬁne quality on the dis-

play of a smartphone.

2 SUPER RESOLUTION FOR

SMARTPHONES

SR is a technology for improving image and video

resolution. As discussed in the previous section there

have been many methods and proposals (Farsiu et al.,

2004)(Park et al., 2003)(Katsaggelos et al., 2010)(van

Eekeren et al., 2010)(Panda et al., 2011)(Glasner

et al., 2009)(Park et al., 2003) (Sun et al., 2008)(Dong

et al., 2014). The size of the monitor becomes an im-

portant factor in seeing SR-processed image results.

This point has not been discussed in SR research.

SR studies freely select their processed image sizes

to recognize the resolution improvement. Personal

computer (PC) monitors are used to check image res-

olution. Although commercial HDTV sets with SR

functions are also available (Tos, 2009), the sizes of

HDTV screens are 40 inches or larger. It would be

difﬁcult to recognize improvementwith SR on a small

smartphone monitor. If we are to implement SR tech-

nology, resolution improvement must be recognizable

on smartphone displays.

Smartphones are sophisticated devices, but it is

impossible to add devices to a smartphone to use SR.

There are two difﬁculties in implementing SR on a

smartphone with limited resources. The ﬁrst is the

complexity of SR algorithms. Many SR algorithms

have been proposed (Farsiu et al., 2004)(Park et al.,

2003)(Katsaggelos et al., 2010)(van Eekeren et al.,

Figure 1: SRNP algorithm.

2010)(Panda et al., 2011)(Glasner et al., 2009)(Park

et al., 2003) (Sun et al., 2008)(Dong et al., 2014).

Super resolution image reconstruction (SRR) and

learning-based super resolution (LBSR) are typical

SR technologies, and many others have been pro-

posed (Farsiu et al., 2004)(Park et al., 2003)(Kat-

saggelos et al., 2010)(van Eekeren et al., 2010)(Dong

et al., 2014). However all SR algorithms includ-

ing SRR and LBSR are difﬁcult to use in real time

for video because they require iteration to create a

high-resolution image. Iteration is very time con-

suming and difﬁcult to execute on the CPU/GPU of

a smartphone. Although a non-iterative SRR algo-

rithm for HDTV has been proposed (Matsumoto and

Ida, 2010), its resolution is lower than that of a con-

ventional HDTV, and an additional device is required

to use the SRR algorithm.

The second difﬁculty is that SR on smartphones

must work on the CPU/GPU of a smartphone. It is

difﬁcult for smartphones to handle additional devices

required for SR to work, because of the space and

power consumption. There is almost no space to im-

plement additional parts on a smartphone, and new

parts shorten battery duration owing to higher power

consumption. If we try to make SR work on a smart-

phone, SR would have to work with the CPU/GPU

and its resources on the smartphone. The CPU/GPU

executes many tasks, and resources such as the mem-

ory bandwidth are limited. If sufﬁcient CPU/GPU

power and resources are not provided for the SR pro-

cess, video cannot be processed in real time, frame

drops can occur, and in the worst case, the video

will freeze. To overcome these difﬁculties, an SR al-

gorithm for a smartphone must be simple and sufﬁ-

ciently light to work on CPU/GPU power and limited

resources.

3 SRNP

SRNP was developed for upconversion from HDTV

Super Resolution for Smartphones

107

(a) 4K image enlarged from HD (b) Figure 2(a) with SRNP

Figure 2: Image processed with real-time SRNP hardware.

to 4K. Figure. Figure 1 shows the signal ﬂow of the

proposed method. The input video has two paths.

The ﬁrst path consists of a high-pass ﬁlter (HPF), a

non-linear function (NLF), and a limiter (LMT). This

path creates high-frequency elements that the origi-

nal video does not have. The edges in the video are

detected with the HPF. The detected edges are then

processed with the NLF. An example of an NLF is

a cubic function y = x

. The NLF generates har-

monic waves from the edges. It is well known that

images and videos can be expanded with Fourier se-

ries. Fourier series consist of sine and cosine waves.

Using the cubic function, the sin and cos functions

are changed to (sinθ)

and (cosθ)

. (sinθ)

can be

changed to (sin3θ)and (cosθ)

can be changed to

(cos3θ). (sin3θ) and (cos3θ) are harmonic waves,

and the harmonic waves have higher frequency ele-

ments that the original video did not have. The cubic

function is just an example of a nonlinear function,

and the NLF is used to create the high-frequency el-

ements by the harmonic waves. The harmonic waves

are generated only from the edges detected with the

HPF. There are no harmonic waves in ﬂat areas since

there are no edges in ﬂat areas. The LMT satu-

rates these large values to ﬁt the harmonic waves to

the video. The second path is from the input and

is directly connected to the adder (ADD). The ADD

adds the LMT-processed harmonic waves to the orig-

inal video. This process is conducted pixel by pixel.

The output of the ADD thus has high-frequency el-

ements that the original video did not have. This

video processing method can improve the resolution

and even create high-frequency elements that exceed

the Nyquist frequency of the original video. It is a

simple algorithm and real-time SRNP hardware for it

has been developed.

Figure 2 shows an image processed with SRNP

hardware. Figure 2(a) is an enlargement from HDTV

to 4K. Figure 2(b) shows the SRNP processed result

of Figure 2(a). Although Figure 2(a) is blurry, Figure

2(b) is clearly superior to Figure 2(a). Figures 2(c)

and 2(d) are the two dimensional fast Fourier trans-

form (2D-FFT) results of Figures 2(a) and 2(a) re-

spectively. Figures 2(a) and 2(a) show the frequency

characteristics in the frequency domain. The horizon-

tal and vertical axes are the horizontal and vertical

frequencies of the image. Note that Figure 2(d) has

horizontal and vertical high-frequency elements that

Figure 2(c) does not possess. This means that SRNP

improves resolution.

However, this has not worked for video in real-

time over CPU/GPU with software. As discussed in

the previous section, the resources in a smartphone

are limited, which makes it difﬁcult for SR to work in

real-time. Even though the SRNP algorithm is simple,

there is no guaranteethat it will work for video in real-

time on a smartphone.

SIGMAP 2016 - International Conference on Signal Processing and Multimedia Applications

108

4 SYSTEM ARCHITECTURE OF

SRNP FOR SMARTPHONES

Smartphones are very much the same as compact

computers equipped with the latest technologies in

order to provide users with various services. How-

ever, there are not sufﬁcient resources and power in

a smartphone to execute all of the tasks of a PC. Al-

though SRNP is a simple algorithm, it might not work

when there are insufﬁcient resources. Constructing an

experimental SRNP system on a smartphone platform

and optimizing the system is the only way to make it

practical.

The speciﬁcations for the experimental hardware

are shown in Table 1. The smartphone has a

MSM8992 CPU and an Adreno 418 GPU , The op-

erating system is Android 5.1. SRNP signal ﬂow over

the hardware is shown in Figure 3. The video is coded

with MPEG-4 H.264 and stored in the SD memory.

The MPEG-4 stream is decoded using a hardware de-

coder. Although the most common video is HDTV

(1920 × 1080), there are other formats available on

the Internet. The SRNP process is performed with

the original formats, such as HDTV, QVGA and so

on. The GPU conducts the SRNP process to im-

prove resolution depending on the input video for-

mat including HDTV. The output video format of the

GPU is the same size as that of the input video. The

GPU processes the video by frames, because the En-

large/Shrink unit after SRNP requires an entire frame

to change the video format.The GPU is controlled by

the CPU and shares memory and other resources with

other units. The GPU also controls its timing when

delivering frames to the Enlarge/Shrink unit in 33.3

ms (30 Hz), in order not to freeze the display. The

liquid crystal display (LCD) can display video in real

time, if the all signal processing works at 30 Hz with-

out delay.

A potential bottleneck of the system is the GPU

because it must ﬁnish one frame of the SRNP process-

ing within 33.3 ms. There are two difﬁculties in over-

coming this bottleneck. First, the GPU calculation of

the SRNP process itself must ﬁnish in 33.3 ms. Sec-

ond, the memory bandwidth must be adequate. The

input output video frame of the GPU is stored in 3 GB

of RAM, shown in Table 1. The RAM is also used by

other applications and is accessed by the CPU. Since

the RAM for the GPU is limited, arbitration between

the CPU and GPU is required. Two tunings are neces-

sary to arrange the tasks for dual and quad CPU cores

to access RAM and improve performance of the GPU

for SRNP. The difﬁculty of the tuning is proportional

to the higher frame rate and larger screen. The high

frame rates require short periods of SRNP, which di-

Figure 3: Signal ﬂow of a smartphone.

rectly corresponds to the capability of the GPU. Large

screens require more memory. Both of these are re-

lated to the memory bandwidth. Currently, the frame

rate of smartphones is 30 Hz. The GPU must ﬁnish

SRNP in 33.3 m for a frame, because smartphones

display videos with 30 Hz (33.3 ms).

The bottleneck of the system is the processing

time of SRNP in the GPU. The processing time of

SRNP in the GPU increases in proportion to the

screen size. Tuning the GPU to program SRNP can

work up to the ultra-HD (UHD) 3, 940 × 2, 160 dis-

play resolution. Table 2 shows the relationship be-

tween the screen sizes and the processing times of

the GPU. Currently, 4K is among the biggest prac-

tical displays available in the market. The processing

time for 4K is 14.1 ms. This means that SRNP can

work in real time for 4K. Moreover, SRNP can work

at 60 Hz (16.7 ms) for 4K because the SRNP time for

4K is 14.1 ms, which is shorter than 16.7 ms. The

simple SRNP algorithm embodies a short processing

time. If other parts of the hardware, such as the H.264

Table 1: Speciﬁcation of experimental smartphone.

Size 154 (H) x 75 (W) x 7.9 (D) mm

Mass 174 g

CPU (system LSI) QualcommMSM8992

1.8 GHz Dual Core1.4 GHz QuadCore

GPU (on the chip) QualcommAdreno 418 600 MHz

Memory RAM: 3GB, ROM: 32GB

Battery 3390 mAh

OS Android 5.1

LCD 5.4inch WQHD1440×2560

Table 2: GPU processing time (30 frame/s).

Size of image GPU processing time (ms/frame)

UHD (3940 × 2160) 14.1

WQHD (2560× 1440) 7.2

Full HD (1920× 1080) 4.6

(1280× 702) 3.9

Super Resolution for Smartphones

109

(a) Input image (b) Image processed with the developed smart-

phone

Figure 4: Image processed with an SRNP smartphone (software processed).

Figure 5: Developed Smartphone with NLSP.

decoder become faster, then SRNP can process even

4K 3,840× 2160;60 Hz).

5 RESULTS

Figures 6(a) to 7(b) show the resolution improvement

of SRNP with a smartphone. They are not still images

but frames of videos. Figures 6(a), and 7(a) are input

images and Figures 6(b), and 7(b) are processed with

SRNP. The resolution of the SRNP-processed images

is better than that of the input images. Just comparing

the input images and the SRNP-processed images is a

subjective assessment.It is sometimes difﬁcult to rec-

ognize the resolution improvement between the SR-

processed images and the images processed with a

conventional enhancer. An enhancer just ampliﬁes

the edges in the images. Our SRNP is completely

different from the conventional enhancers. SRNP

can create higher frequency elements that the orig-

inal image does not possess. As discussed in Sec-

tions 3 and 4, SRNP creates higher frequency ele-

ments that the input image does not possess. Owing

to space limitations, all of the 2D-FFT results cannot

be shown. However, all of the SRNP-processed im-

ages are improved in their resolution. These videos

have distortions caused by MPEG-4 because they

were taken using a commercial video camera. How-

ever, the distortions did not affect the image quality

and the resolution improvement from SRNP is con-

spicuous. These videos had high-frequency elements,

which made MPEG-4 decoding heavy and time con-

suming. However, all of them were displayed at 30

Hz without any frame drops or freezing. This shows

that SRNP can work in real-time on a smartphone.

Although our smartphone with SR was developed for

playing real-time video, we used still images, includ-

SIGMAP 2016 - International Conference on Signal Processing and Multimedia Applications

110

(a) Original image (b) Image processed with the developed smartphone

Figure 6: Image No. 1.

(a) Original image (b) Image processed with the developed smartphone

Figure 7: Image No. 2.

ing documents in our assessments. Observers can

control the SR switch (on/off) to compare the image

with SR on and off. We conducted subjective as-

sessments. Due to space limitations, the details of

the assessments cannot be explained here because it

takes several pages to explain everything, including

the analysis results of the assessments. After the as-

sessments, many observers said that the letters in the

documents became easier to read when SR was on

because the contrast of the LCD was enhanced. This

means that we can make the backlight of the LCD

lower and reduce the power consumption. SR also

has the potential to extend the battery life of a device.

6 CONCLUSION

Many SR technologies have been proposed, but to

date, the signal-processing load was heavy, and SR

could not run on a smartphone. We proposed an SR

technology that can work with smartphone hardware.

It works for 60 Hz video in real-time without ad-

ditional devices. The developed smartphone shows

higher-frequency elements that the original video did

not possess. 2D-FFT results were presented to prove

that it works. Although displays of smartphones are

small, the resolution improvement can be detected

easily by a smartphone user. SR for the smartphone

was developed for real-time video. However, it comes

with a side beneﬁt. SR enhances the contrast, and the

letters in documents become easier to read. The de-

veloped smartphone is equipped with wide quad HD

(WQHD), 1440×2560 pixels display. 4K displays are

beginning to be implemented on smartphones. Larger

displays require high loads to cope with real-time sys-

tems because the system components must process

more pixels. SR for a smartphone with a 4K display

is the next step.

REFERENCES

(2009). Toshiba press release.

Candes, E. J. and Fernandez-Granda, C. (2014). Towards

a mathematical theory of super resolution. Communi-

cations on Pure and Applied Mathematics, 67.6:906–

956.

Dong, C., Loy, C. C., He, K., and Tang, X. (2014). Learn-

ing a deep convolutional network for image super-

resolution. Computer Vision-ECCV, pages 184–199.

Elad, M. and Feuer, A. (1996). Super-resolution of continu-

ous image sequence. IEEE Trans. on Pattern Analysis

and Machine Intelligence.

Farsiu, S., Robinson, M., Elad, M., and Milanfar, P.

(2004). Fast and robust multi frame super reso-

lution. IEEE Transactions on Image Processing,

13(10):1327–1344.

Super Resolution for Smartphones

111

Glasner, D., Bagon, S., , and Irani, M. (Oct. 2009). Inter-

national conference on computer vision (iccv). Super-

Resolution from a Single Image.

Katsaggelos, A., Molina, R., and Mateos, J. (2010). Super

Resolution of Images and Video: Synthesis Lectures

on Images, Video and Multimedia Processing. Morgan

and Clayppo Publishers, La Vergne TN USA.

Lee, J. S. (March 1980). Ieee trans. on pattern analysis and

machine intelligence 2:165-168. Digital Image En-

hancement and Noise Filtering by Use of Local Statis-

tics.

Matsumoto, N. and Ida, T. (2010). Reconstruction-based

super-resolution using self-congruency around image

edges. IEICE transactions on information and systems

(in Japanese), J93-D(2)(2):118–126.

Panda, S., Prasad, R., and Jena, G. (Sept. 2011). Pocs

based super-resolution image reconstruction using an

adaptive regularization parameter. IJCSI International

Journal of Computer Science Issues, Vol. 8, Issue 5,

No. 2, ISSN (Online), 1694-0814.

Park, S. C., Park, M. K., and Kang, M. G. (2003).

Super-resolution image reconstruction: A technical

overview. IEEE Signal Processing Magazine, 1053-

5888/03:21–36.

Pratt, W. (2001). Digital Image Processing (3rd Ed): New

York. John Wiley and Sons.

Schreiber, W. F. (1970). Wirephoto quality improvement by

unsharp masking. J. Pattern Recognition, 2:111-121.

Sun, J., Sun, J., Xu, Z., and Shum, H. (2008). Image super-

resolution using gradient proﬁle prior. CVPR 2008,

pages 1–8.

van Eekeren, A. W. M., Schutte, K., and van Vliet, L. J.

(2010). Multiframe super-resolution reconstruction of

small moving objects. IEEE Transactions on Image

Processing, 19(11):2901–2912.

SIGMAP 2016 - International Conference on Signal Processing and Multimedia Applications

112