Real-time Super Resolution Algorithm for Security Cameras

Seiichi Gohshi

Kogakuin University, 1-24-2 Nishi-Shinjuku, Shinjuku-ku, Tokyo, 163-8677, Japan

Keywords:

Security Camera, Real-time Video Processing, Nyquist Frequency, Super Resolution.

Abstract:

Security is one of the most important things in our daily lives. Security camera systems have been introduced

to keep us safe in shops, airports, downtowns, and other public spaces. Security cameras have infrared imag-

ing modes for low-light conditions. However, infrared imaging sensitivity is low, and the quality of images

recorded in low-light conditions is often poor as they do not always possess sufﬁcient contrast and resolution;

thus, infrared imaging devices produce blurry monochrome images and videos. A real-time nonlinear signal

processing technique that improves the contrast and resolution of low-contrast infrared images and video is

proposed. The proposed algorithm can be installed in a ﬁeld programmable array.

1 INTRODUCTION

Because of the increasing demand for security and the

decreasing cost of security cameras, the security cam-

era systems are expected to be worth four billion USD

in 2015. Analog security cameras are being replaced

by digital devices, and Internet protocol (IP) security

camera systems are becoming increasingly common.

Security camera systems are not stand alone systems;

they usually work in conjunction with network tech-

nology. In TV detective dramas, investigators some-

times have special imaging technologies that can cre-

ate high-resolution images (HRI) from blurry images.

Unfortunately, in reality such technologies do not ex-

ist; therefore, increasing the resolution of the imaging

devices is the only method to obtain HRIs.

In recent times, high-resolution 4K cameras have

become more affordable, and networked 4K security

camera systems have also become available. Such

systems record videos onto servers. IP security sys-

tems are commonly employed with central building

control systems and at large-scale events. Although

high resolution security camera systems are becoming

increasingly convenient and are widely used, theystill

do not function effectively in low-light conditions.

During the night, lighting conditions are insufﬁcient

and security cameras are switched to infrared mode,

which provides only low-resolution and low-contrast

images. Several methods have been proposed to im-

prove infrared images (Lee et al., 2012; Ibekwe et al.,

2012; Pﬂugfelder et al., 2005). However, they are un-

able to produce images of sufﬁciently high quality for

their intended application.

Interest in super resolution (SR) technology,

which attempts to improve image resolution, has

grown rapidly in recent times (Farsiu et al., 2004;

Park et al., 2003; Katsaggelos et al., 2010; van Eek-

eren et al., 2010). However, most SR technologies

consider only still images, and such studies have fo-

cused primarily on the signal processing of color im-

ages shot under adequate lighting conditions. Al-

though the algorithms for still images have been ap-

plied to infrared images (Farsiu et al., 2004; van Eek-

eren et al., 2010), the image quality produced by such

methods remains insufﬁcient.

Another important requirement for security cam-

eras is real-time signal processing (note that most SR

algorithms are complex and non-real time systems).

At a crime scene, time is very valuable; thus, sig-

nal processing speed is very important. SR algo-

rithms that perform iterations or require several low-

resolution images (LRI) cannot process video in real

time; thus, the time required to perform these pro-

cesses can impede criminal investigations. Therefore,

to address this problem, SR processing should be per-

formed in real time.

An SR algorithm for infrared images that satis-

ﬁes both image quality and processing speed has not

been reported to date. In recent times, a processing

method that uses nonlinear signal processing (NLSP)

to improve resolution by creating higher frequencyel-

ements has been proposed (Gohshi, 2014). However,

this method does not satisfy the requirements for in-

frared images. In this study, we have attempted to

modify the NLSP method (Gohshi, 2014) to signiﬁ-

cantly improve the quality of infrared images.

Gohshi S..

Real-time Super Resolution Algorithm for Security Cameras.

DOI: 10.5220/0005559800920097

In Proceedings of the 12th International Conference on Signal Processing and Multimedia Applications (SIGMAP-2015), pages 92-97

ISBN: 978-989-758-118-2

 2015 SCITEPRESS (Science and Technology Publications, Lda.)

The remainder of this paper is organized as fol-

lows. In Section 2, the deﬁnition and limitations of

SR technologies are discussed. Section 3 introduces

the NLSP algorithm. In Section 4, a real-time SR al-

gorithm and expanded NLSP for a security camera

system are proposed. Section 5 presents experimen-

tal results, and conclusions and suggestions for future

work are given in Section 6.

2 SUPER RESOLUTION

Improving image resolution from analog TV to

HDTV has been a highly active ﬁeld of research for

more than two decades. Note that many images can-

not be recorded under reasonable conditions because

of optical limitations, such as different lenses, light-

ing conditions, and focus. As a result, such images

do not always have sufﬁcient resolution. For years,

unsharp masking (USM; also referred to as edge en-

hancement) has been the only method to enhance

video in real-time systems. Although USM is a sim-

ple and cost effective method, it does not essentially

improve resolution; it provides better image quality

using either a band-pass ﬁlter (BPF) or a high-pass

ﬁlter (HPF). However, USM can introduce noise and

edges to images.

SR technology, which has been researched for ap-

proximately two decades (Elad and Feuer, 1996), cre-

ates an HRI from LRIs. Many SR methods have been

proposed (Farsiu et al., 2004; Park et al., 2003; van

Eekeren et al., 2010; Matsumoto and Ida, 2008) ;

however, these methods are primarily applied to still

images, where processing time is not of paramount

importance. Such algorithms perform iterations to

construct an HRI from LRIs. These iterations con-

tinue until the value calculated by a constraint con-

dition converges to the minimum value. Minimiz-

ing this constraint condition is simply a method of

weighted least squares, and the reached point is gen-

erally not exact. Furthermore, there is no guarantee

that the minimum point can create the highest resolu-

tion. In such methods, the number of iterations can

become greater than 200 (Sugie and Gohshi, 2013) ,

and the iterations are sometimes discontinued before

they converge to the minimum point. Even though

such methods can reach the highest resolution, the re-

quired iterations create a bottleneck for real-time sig-

nal processing. Moreover, to assess results, previous

studies have only visually compared the original and

the SR processed images. However, this is a subjec-

tive quality assessment.

It is almost impossible to recognize the difference

between the proposed SR technologies and previous

image enhancement technologies by simply assessing

the pre- and post-processed images because the va-

lidities of image processing methods differ depend-

ing on the images. In recent times, some manufac-

turers have stated that their commercial products are

equipped with SR functions. However, these manu-

facturers have not disclosed the SR algorithms they

employ, and there is no evidence to support their

claims. Therefore, an objective assessment method

is required.

SR can be deﬁned as creating high-frequency

elements that the original image does not possess.

Here we propose to compare the two-dimensional fast

Fourier transform (2D-FFT) of the preprocessed im-

age with that of the post-processed image. If the 2D-

FFT results are shown, we can compare the spectra

of the pre- and post-processed images. If the spectra

of the SR post-processed image have high-frequency

elements that the preprocessed image does not pos-

sess, it can be concluded that the signal processing

method has expanded the spectra. If a proposed SR

technology could create high-frequency elements that

the original image does not possess, the 2D-FFT re-

sults should be demonstrated. Thus, we propose 2D-

FFT results as an objective assessment criteria for SR.

Figure 1 shows a crisp image and Figure 2 shows

a blurred image of Figure 1. Figure 3 is the 2D-FFT

result of Figure 1, and Figure 4 is the 2D-FFT result

of Figure 2.

By comparing Figures 3 and 4, it can be seen that

the resolution of Figures 1 and 2 can be assessed

quantitatively. Since images have different charac-

teristics, the validity of SR differs depending on the

images. Thus, only comparing the pre- and post-

processed images is an ineffective way to assess the

validity of SR because this is a qualitative subjec-

tive assessment; thus, scores may vary for different

observers. However, the proposed 2D-FFT criterion

provides an objective assessment that yields quantita-

tive scores.

3 NONLINEAR SIGNAL

PROCESSING

NLSP has been proposed for the up-conversion of

4K video to 8K video (Gohshi, 2014). Although the

NLSP algorithm (Figure 6) is quite simple, it can cre-

ate higher frequency elements that the original image

does not possess.

The input image is distributed to a HPF and an

adder (ADD). The HPF detects edges in the image,

and the edges are processed with a nonlinear function

(NLF). The NLF is a symmetry function with respect

Real-timeSuperResolutionAlgorithmforSecurityCameras

to the original point of the coordinate where the or-

dinate and abscissa cross. The cubic function y = x

is an example that satisﬁes this condition. The edges

detected with the HPF are processed with this cubic

function.

Figure 1: Original image.

Figure 2: Blurred image of Figure 1.

Figure 3: 2D-FFT result of Figure 1.

Figure 4: 2D-FFT result of Figure 2.

The cubic function can create high-frequency el-

ements that the input image does not possess. Note

that an image expanded by a Fourier series comprises

sine and cosine functions with the fundamental fre-

quency of the image ω

. Here edges are represented

with sin(nω

) and cos(nω

) functions. Here, n is an

integer number (n = 0, ±1, ±2, · · ·). Here y = x

gen-

erates sin

(nω

) and cos

(nω

). Note that sin

(nω

)

can be rewritten with sin(3nω

)and cos

(nω

) can be

rewritten with cos(3nω

). This implies that frequency

elements that havea frequency three times higher than

that of the original image can be generated, and these

high-frequency elements are edges that the original

image does not possess. The edges are added to the

input image by the ADD, and the resulting HRI is ob-

tained. Although there are many SR technologies that

use LRIs, NLSP uses only a single LRI as input.

If an even function is selected, such as y = x

, the

positive or negative sign information is lost, and the

NLF output becomes positive. However, edges are ei-

ther positive or negative. The most signiﬁcant bit is

separated from the edge information before the NLF

is performed and is then restored after the NLF. With

this method, we can use even NLFs. In this method,

for x ≥ 0, y = x

is selected, and for x < 0, y = −x

is selected. This method allows much more ﬂexibil-

ity than the algorithm shown in Figure 6 because the

choice of NLF is expanded.

Figure 5 shows an image processed with NLSP.

Figure 5(a) is an enlargement from HDTV to 4K. Fig-

ure 5(b) shows the NLSP processed result of Figure

5(a). Although Figure 5(a) is blurry, Figure 5(b) is

clearly superior to Figure 5(a). Figures 5(c) and 5(d)

are the 2D-FFT results of Figures 5(a) and 5(a) re-

spectively. Note that Figure 5(d) has horizontal and

vertical high-frequencyelements that Figure 5(c) does

not possess. Although NLSP was applied to infrared

images, the signal processing shown in Figure 6 does

not produce an infrared image of sufﬁciently high

quality. To improve the infrared image quality, a new

NLSP method is proposed in the next section.

4 SIGNAL PROCESSING

TECHNIQUE FOR SECURITY

CAMERAS

Infrared images are low-contrastmonochromeimages

that do not have high-frequency elements. Figures 8

and 10 are typical infrared images. Note that they

lack contrast and do not demonstrate high-frequency

elements. Simply adjusting the characteristics of the

HPF (Figure 6) cannot generate edges.

Thus, an algorithm for infrared images (Figure 7

) is proposed to handle low-contrast images and edge

detection. Note that the contrast should be adjusted

SIGMAP2015-InternationalConferenceonSignalProcessingandMultimediaApplications

(a) 4K image enlarged from HD (b) Figure 5(a) with NLSP

Figure 5: Image processed with real-time NLSP.

Figure 6: NLSP algorithm.

Figure 7: Block diagram of signal processing for infrared

image.

prior to performing NLSP. In Figure 7 three NLFs

(NLF1, NLF2 and NLF3) are employed. In the up-

per path, NLF1 changes the contrast to create more

high-frequency elements than the algorithm shown in

Figure 6. Using NLF1, edges at low-luminance levels

are ampliﬁed so that the HPF can easily detect them.

NLF3 has nonlinear characteristics and functions that

are similar to the NLF shown in Figure 6. NLF2 is

employed for infrared images to create high contrast

for a low-contrast image. NLF1, NLF2 and NLF3

produce high-contrast images. To increase contrast,

NLF1 and NLF2 should be convex downward. By

using several characteristics of NLFs for NLF1 and

NLF2, y = x

0.3

was selected for our experiments. In

addition, y = x

was selected for NLF3 for the exper-

iments. Here, x is the input, and y is the output of

NLF1, NLF2 and NLF3. However, more experiments

for NLF1, NLF2 and NLF3 are required to produce

improved infrared image quality.

5 EXPERIMENT AND

DISCUSSION

As is shown in Figures 9 and 11, the algorithm shown

in Figure 7 to process the input image shown in Fig-

ures 8 and 10 improves the contrast and details of the

image. Although these infrared images were shot out-

side under very similar lighting conditions, the pro-

cessed images show much more information. Images

processed with (Gohshi, 2014) are shown in Figures

12 and 13 for the comparison between the conven-

tional method and the proposed method.

Real-timeSuperResolutionAlgorithmforSecurityCameras

Figure 8: House (before processing). Figure 9: House (after processing).

Figure 10: Deer (before processing). Figure 11: House (after processing).

Figure 12: House (Figure 8 Processed with (Gohshi, 2014). Figure 13: House (Figure 10 Processed with (Gohshi, 2014).

In these experiments, NLF1 and NLF2 were set to

y = x

0.3

and NLF3 was set to y = x

to assess the basic

functionality of the proposed method. The processed

results shown in Figures 9 and 11 can be improved

by modifying the NLFs. It was observed that the pro-

posed technique can improve infrared image quality.

However, to produce a practical system, further re-

search is required to compare the NLFs of the algo-

rithm, i.e., NLF1 and NLF2. Although NLF3 must go

through the original point in the coordinates, NLF2

can employ y = x

+ a a to adjust brightness. Here,

the luminance signal is positive, and the parameter a

can make the image clearer with appropriate adjust-

ment. It is also necessary to adjust other character-

istics, such as the HPF characteristics (i.e., ﬁlter tap

length and coefﬁcients).

Our eyes function as an LPF in dark environments

and as a BPF in bright environments. Research results

SIGMAP2015-InternationalConferenceonSignalProcessingandMultimediaApplications

on NLSP techniques are limited despite the fact that

our eyes work as a nonlinear system. Further research

of such NLFs is required to improve image quality.

The proposed method can allow security cameras to

capture important images with high quality using sim-

ple real-time hardware. In recent times, the security

camera industry changes stand-alone security camera

and its recording system and IP camera systems have

been introduced. We expect that the proposed method

can be used to develop more cost-effective and useful

IP camera systems.

6 CONCLUSIONS

A single image SR technique for infrared images has

been proposed. Many SR technologies have been

proposed; however, to date, no objective SR assess-

ment method has been available. Thus, we have pro-

posed 2D-FFT to evaluate SR technologies objec-

tively. Infrared images are low-resolution and have

low-contrast. Although the proposed algorithm is

simple, it can improve the quality of such images. The

proposed SR technique can improve contrast and can

create higher frequency elements that the original im-

age does not possess. We expect that the proposed

algorithm can be used to develop real-time capable

hardware for video and image content. The proposed

method increases the potential of security systems at

night ; however, further research into the proposed al-

gorithm and nonlinear functions is required for a prac-

tical security camera system.

REFERENCES

Elad, M. and Feuer, A. (1996). Super-resolution of continu-

ous image sequence. IEEE Trans. on Pattern Analysis

and Machine Intelligence.

Farsiu, S., Robinson, M., Elad, M., and Milanfar, P.

(2004). Fast and robust multi frame super reso-

lution. IEEE Transactions on Image Processing,

13(10):1327–1344.

Gohshi, S. (2014). Real-time super resolution equipment

for 8k video. SIGMAP 2014, pages 149–157.

Ibekwe, M., Vitek, S., Klima, M., and Dostal, P. (2012).

Modeling and evaluation of image quality in wire-

less surveillance networks. In Proc. IEEE ICCST 12,

pages 345–352, Boston, USA.

Katsaggelos, A., Molina, R., and Mateos, J. (2010). Super

Resolution of Images and Video :Synthesis Lectures on

Images, Video and Multimedia Processing. Morgan

and Clayppo Publishers, La Vergne TN USA.

Lee, Y., Kim, J., Member, S., and Kyung, C.-M. (2012).

Energy-aware video encoding for image quality im-

provement in battery-operated surveillance camera.

IEEE Transactions on Very Large Scale Integration

(VLSI) SYSTEMS, 20(2):310–318.

Matsumoto, N. and Ida, T. (Apr. 2008). A study on one

frame reconstruction-based super-resolution using im-

age segmentation. IEICE Technical Report SIP2008-

6,IE2008-6 (in Japanese).

Park, S. C., Park, M. K., and Kang, M. G. (2003).

Super-resolution image reconstruction: A technical

overview. IEEE Signal Processing Magazine, 1053-

5888/03:21–36.

Pﬂugfelder, R., Bischof, H., Dom´ınguez, G. F., N¨olle, M.,

and Schwabach, H. (2005). Inﬂuence of camera prop-

erties on image analysis in visual tunnel surveillance.

In Proc. IEEE ITS ’05, pages 868–873, Vienna, Aus-

tria.

Sugie, M. and Gohshi, S. (2013). Performance veriﬁcation

of super-resolution image reconstruction. 2013 Inter-

national Symposium on Intelligent Signal Processing

and Communication Systems (ISPACS 2013), TA1-C-

4:547–552.

van Eekeren, A. W. M., Schutte, K., and van Vliet, L. J.

(2010). Multiframe super-resolution reconstruction of

small moving objects. IEEE Transactions on Image

Processing, 19(11):2901–2912.

Real-timeSuperResolutionAlgorithmforSecurityCameras