A Tensor-based Technique for Structure-aware Image Inpainting

Adib Akl and Charles Yaacoub

Faculty of Engineering, Holy Spirit University of Kaslik (USEK), Jounieh, Lebanon

{adibakl, charlesyaacoub}@usek.edu.lb

Keywords: Image Inpainting, Orientation, Second-moment Matrix, Exemplar.

Abstract: Image inpainting is an active area of study in computer graphics, computer vision and image processing.

Different image inpainting algorithms have been recently proposed. Most of them have shown their

efficiency with different image types. However, failure cases still exist, especially when dealing with local

image variations. This paper presents an image inpainting approach based on structure layer modeling,

where this latter is represented by the second-moment matrix, also known as the structure tensor. The

structure layer of the image is first inpainted using the non-parametric synthesis algorithm of Wei and

Levoy, then the inpainted field of second-moment matrices is used to constrain the inpainting of the image

itself. Results show that using the structural information, relevant local patterns can be better inpainted

comparing to the standard intensity-based approach.

1 INTRODUCTION

Image inpainting is a dynamic research field with

different applications. It is used in video animations,

video completion, frames merging, image

restoration, image extrapolation, image editing and

video compression (Kwatra et al., 2003, Bargteil et

al., 2006, Yamauchi et al., 2003, Winkenbach and

Salesin, 1994). It is also used to describe the

geometry of a surface, to remove undesired objects

from images and videos and to fill missing regions

(Bertalmio et al., 2000).

In the past decades, several image inpainting

algorithms have been proposed. For instance, the

image synthesis method of Paget and Longstaff

(Paget and Longstaff, 1998) captures the local

characteristics of an image into a statistical model

describing the interaction between the pixels of this

image. The Efros and Leung (Efros and Leung,

1999) approach generates the inpainted image by

directly sampling new values from the input sample.

The exemplar-based algorithms in (Criminisi et al.,

2004) and (Aujol et al., 2009) consist in directly

copying patches from the exemplar image. A

method based on the graph cut technique, used to

determine the patch region without choosing its size

a-priori, is proposed in (Kwatra et al., 2003). Portilla

and Simoncelli (Portilla and Simoncelli, 2000) rely

on the wavelet transform used to parameterize the

image by a set of statistics, at adjacent scales and

locations. A total variation inpainting model is

proposed in (Chan and Shen, 2001). It is based on

the theory of Euler-Lagrange and on anisotropic

diffusion. The non-parametric image synthesis

algorithm of Wei and Levoy (Wei and Levoy, 2000)

models the image as a realization of a local and

stationary random process.

It has been demonstrated that taking into

consideration the structural information of an image

can help in the synthesis of this image, especially in

the case of local structural variations (Akl et al.,

2014, Akl et al., 2015).

This paper presents a structure-based inpainting

algorithm where the structure layer of the image,

represented by the second-moment matrix field, is

first inpainted, then the obtained structure field is

used to help the image inpainting process. The

proposed approach consists in adapting non-

parametric image synthesis methods to the

specificities of the second-moment matrix. More

precisely, the algorithm of Wei and Levoy (Wei and

Levoy, 2000) is used in the inpainting of the

structure layer stage and in the inpainting process of

the image itself.

The remainder of this paper is organized as

follows: the second-moment matrix field

computation is first reviewed in section 2. The

proposed inpainting method is then detailed in

section 3. Results are shown and discussed in section

4, and section 5 finally presents conclusions and

perspectives of future work.

Akl, A. and Yaacoub, C.

A Tensor-based Technique for Structure-aware Image Inpainting.

DOI: 10.5220/0006214605990605

In Proceedings of the 6th International Conference on Pattern Recognition Applications and Methods (ICPRAM 2017), pages 599-605

ISBN: 978-989-758-222-6

599

2 THE SECOND-MOMENT

MATRIX

The second-moment matrix, also referred as the

structure tensor, is a gradient-based matrix whose

first eigenvector points in the direction of the

greatest rate of increase of the scalar field (Bigun

and Granlund, 1987, Akl and Iskandar, 2015, Akl

and Iskandar, 2016). A second-moment matrix

SM(z) at image position z summarizes the dominant

directions of the gradient in the neighbourhood of z.

Therefore, it can be used to represent and to describe

edges. In image processing, the second-moment

matrix represents partial derivatives and it is

commonly used to describe local patterns (Bigun

and Granlund, 1987).

The second-moment matrix field SM of an image

A is defined as the field of local covariance matrices

of the partial derivatives of A, built using the

gradient fields

[],

with:

**,,

xxy y

AAG A AG

(1)

where ‘*’ represents convolution, G

and G

are

isotropic Gaussian derivatives kernels.

The second-moment matrix field is computed as:

†

xx xy

xy y

sxy xy

sx sxy

sx y

SAAAM

AAA

MSM

SM SM A A





























(2)

where [.]

†

is the transpose operator and



is a

weighting function – usually Gaussian – used to

smooth the gradient fields, which makes them more

robust to noise.

The second-moment matrix can be represented by an

ellipse with its principle orientation, ranging

between -π/2 and π/2, computed as:

()

-1

SM z

tan











(3)

where U

= [U

] is the first eigenvector of

matrix SM(z).

3 PROPOSED ALGORITHM

This section details the proposed image inpainting

algorithm which consists of two stages; structure

layer inpainting and image inpainting using the

inpainted structure, i.e. the second-moment matrix

field.

For concision, we denote the missing area to be

inpainted as “MA”, the reference from which the

intensities are copied to the MA as “exemplar”, the

image showing the MA and the exemplar as “input

image” and the obtained image after inpainting as

“output image” (Fig. 1).

Figure 1: Illustration of the inpainting principle. Left:

input image showing the MA (missing area to be

inpainted) marked as a black surface, and the exemplar

(reference image from which the intensities are copied)

contoured in red. Right: output image.

3.1 Structure Layer Inpainting

The inpainting process of the structure layer starts

by computing the second-moment matrix field from

the luminance component of the input image (cf.

section 2). The luminance component is calculated

as in (ITU-R, 2011). To ensure that the inpainted

MA is locally similar to the neighbouring regions of

the input image, the algorithm of Wei and Levoy is

adapted to the specificities of the second-moment

matrix as follows: the MA of the second-moment

matrix field is first initialized as a random noise, i.e.

second-moment matrices chosen randomly from the

second-moment matrix field of the exemplar, then

the neighbourhood (vector of matrices) of each

second-moment matrix of the MA is captured, the

neighbourhood of the second-moment matrix field

of the exemplar having the best similarity with the

current neighbourhood is determined, and its central

structure tensor is copied to the current position in

the MA, as illustrated in Fig. 2. In this latter, the

second-moment matrix field is represented by its

orientation image. The palette of orientations is

shown on the right. Note that this palette is used for

all the results that follow.

The similarity between two second-moment

matrices at positions z

and z

is calculated using the

square of the Euclidean distance as follows:

ICPRAM 2017 - 6th International Conference on Pattern Recognition Applications and Methods

600





12 1 2

12 12

(), ( ) () ( )

() ( ) (2)(),

xx xx

yy yy xy xy

SM z SM z SM z SM z









 

(4)

and the similarity between two second-moment

matrix neighbourhoods SM

and SM

is calculated

using the Sum of Second-Moment Matrix

Dissimilarity (SSMD):

   



12 1 2

,,,

SSMD SM SM SM n SM n









(5)

where SM

(n) is the n

second-moment matrix

within the neighbourhood SM

- i ∈ {1,2} and N is

the number of second-moment matrices in each

neighbourhood.

Figure 2: Second-moment matrix field inpainting. The

most similar neighbourhood (yellow square) of the current

neighbourhood (blue square) is searched for in the

exemplar (contoured in red), and the corresponding matrix

is copied to the target position in the MA. The palette used

to represent the second-moment matrix field orientations

is shown on the right.

Note that this synthesis process can be repeated

iteratively in order to obtain an inpainted MA which

is coherent with the remaining second-moment

matrices of the input image, especially the

neighbouring ones.

It is trivial that the neighbouring system

(neighbour size and shape) and the scan type used in

the inpainting process directly influence the quality

of the inpainted MA. In this paper, the inpanting

process starts by filling the outer border of the MA

and ends at its center while using a square

neighbourhood of size 9×9. However, a random scan

could avoid verbatim copies (Xu et al., 2000) i.e.

when the inpainted MA seems more regular than

neighbouring second-moment matrices of the input

image.

3.2 Image Inpainting

The structure layer being inpainted, it will be used to

help the inpainting of the output image. The

inpainting process remains the same as the algorithm

of Wei and Levoy, except that the neighbourhoods

take into consideration the additional information

provided by the inpainted second-moment matrix

field.

More precisely, the image inpainting algorithm

takes as inputs, the exemplar, its second-moment

matrix field, the inpainted second-moment matrix

field (cf. section 3.1) and the output image with its

MA initialized by random noise (i.e. intensity values

chosen randomly from the reference image). Then

intensity values of the missing pixels of the MA are

updated iteratively in order to ensure their local

similarity with neighbouring pixels in the rest of the

input image: the neighbourhood of every missing

pixel of the MA is captured, the most similar

neighbourhood is searched for in the exemplar and

copied entirely to the target position in the MA.

However, the underlying neighbourhoods have two

components: an intensity component in the exemplar

) and the MA of the output image (A

), and a

second-moment matrix component in the structure

layers of the exemplar (SM

) and the output image

MA (SM

) as shown in Fig. 3.

Note that the MA pixels update is patch-based

(i.e. the most similar neighbourhood is entirely

copied to the target position) and not pixel by pixel

(i.e. the center value of the most similar

neighbourhood is copied to the target position) as it

is the case in the second-moment matrix field

inpainting stage, in order to reduce the

computational load of the inpainting process. In fact,

this can be achieved without any blockiness effect,

thanks to the additional structural information

provided by the inpainted second-moment matrix

field.

To measure the similarity between two

neighbourhoods, we propose to combine the Sum of

Second-Moment Matrix Dissimilarity (SSMD), used

for the second-moment matrix component (cf.

equation (5)), and the Sum-square Distance (SD),

used for the intensity component:





  

1,,

SD A A

SSMD SM SM



 

 

(6)

where SM

and SM

are respectively the second-

moment matrix components of the neighbourhoods

in the inpainted second-moment matrix field of the

MA and in the second-moment matrix field of the

A Tensor-based Technique for Structure-aware Image Inpainting

601

exemplar, A

and A

are respectively the intensity

components of the neighbourhoods in the MA and in

the exemplar (Fig. 3), α is a weight factor

(0 ≤ α ≤ 1 since SD and SSMD are normalized), and

the Sum-square Distance (SD) is given by:

   



12 1 2

SD A A A n A n









(7)

Figure 3: Illustration of the image inpainting process: for

each current neighbourhood (A

in MA and SM

in its

inpainted second-moment matrix field), the most similar

neighbourhood (A

in the exemplar and SM

in its second-

moment matrix field) is searched for, and the

corresponding intensity component (A

) is entirely copied

to the target position (A

Note that, when α = 1, the second-moment matrix

information is deactivated and a pure Wei and

Levoy inpainting process is applied. On the contrary,

when α = 0, the intensity information is deactivated

and the choice of the best similarity depends only on

the second-moment matrix information.

The pseudocode of the whole inpainting

algorithm is presented in Fig. 4.

It is important to mention that in both inpainting

stages of the proposed algorithm, the Wei and Levoy

method is used due to its versatility and its pixel

based principle which proved its efficiency in the

synthesis of different types of images. However,

other image synthesis algorithms could also

incorporate this structure-based approach.

4 RESULTS

In this section, some practical results are presented,

evaluated and analyzed subjectively and objectively.

4.1 Qualitative Evaluation

Fig. 5 presents inpainting results obtained using the

proposed algorithm on three different input images.

………………………………………………………

Structure Layer Inpainting

SECOND-MOMENTMATRIXCALCULATION (A)

SMNOISEINITIALIZATION (SM)

loop through all positions

zo of MA

argmax{BESTSIMILARITY(SM

vs SM

)}

(zo) SM

(z)

endloop

………………………………………………….……

Image Inpainting

MA NOISEINITIALIZATION (A)

loop through all positions

zo of MA

argmax{BESTSIMILARITY(SM

, A

vs SM

, A

)}

endloop

………………………………………………….……

Figure 4: Pseudocode of the proposed inpainting

algorithm.

Each result shows, from first to seventh row, the

input image showing the missing area (marked as a

black surface) and the exemplar (contoured in red),

the structure layer of the input image, the inpainted

structure layer obtained by the proposed approach,

the output image obtained using the proposed

algorithm with α = 1 (pure Wei and Levoy

inpainting), the output image obtained using the

proposed approach with α = 0.5, nearer view of the

output images obtained with α = 1 and

α = 0.5. As mentioned in section 3.1, square

neighbourhoods of size 9×9 are used.

It can be seen in the first result that the proposed

approach succeeds in well reproducing the structural

information of the exemplar. The obtained

orientation field of the structure layer does not

present distortions nor edge effects. Therefore, the

output image obtained with α = 0.5 is of good

quality and visually better than the one obtained

with the intensity-based inpainting approach of Wei

and Levoy.



ICPRAM 2017 - 6th International Conference on Pattern Recognition Applications and Methods

602

A B C

Figure 5: Inpainting results obtained by the proposed algorithm. For each result (1

to 7

row): input image, its structure

layer orientation field, inpainted structure layer orientation field, output image obtained using the proposed approach while

deactivating the structural information (α = 1, pure Wei and Levoy inpainting), output image obtained using the proposed

algorithm with α = 0.5, nearer view of the output images obtained with α = 1 and α = 0.5.

The inpainted missing area obtained by this latter

looks over smoothed and presents dynamics

degradation. The same applies for the second result

where the inpainted area obtained with α = 1 appears

more regular than the exemplar, i.e. repeated

periodic patterns that do not exist in the exemplar

are present. In the third result, both output images

are of acceptable quality.

4.2 Quantitative Evaluation

Besides the subjective qualitative evaluation

presented in section 4.1, this section presents a

A Tensor-based Technique for Structure-aware Image Inpainting

603

quantitative analysis of the results. It consists in

comparing the histograms of intensity and

orientations of the exemplar and the inpainted area

by computing the Kullback and Leibler (Kullback

and Leibler, 1951) divergence between them as

follows:

()

(||) ()log

()

()log ,

()

MA exp MA

exp

KL H H H i







(8)

where H

and H

exp

are respectively the histograms

of intensity (or orientation) of the missing area and

the exemplar.

The Kullback and Leibler difference values obtained

on the histograms of the output images of Fig. 5 are

shown in Table 1.

Table 1: Objective results obtained on the images of Fig.

Imag

Intensity

Histograms

Orientation

Histograms

α = 1 α = 0.5 α = 1 α = 0.5

0.412 0.201 0.399 0.297

0.308 0.31 0.402 0.289

0.398 0.356 0.3 0.306

The objective evaluation of Table 1 generally

confirms our subjective analysis. In result A, both

intensity and orientation histogram differences are

higher with α = 1 than with α = 0.5, which verifies

that the dynamics of the inpainted missing area are

distorted with the pure Wei and Levoy inpainting.

The high orientation histogram difference (0.402) in

result B is due to the undesired repetitive patterns

shown in the inpainted area with α = 1. Finally, the

success of both, Wei and Levoy’s algorithm and the

proposed approach, in leading to output images of

roughly similar quality, is verified in the last row of

Table 1.

5 CONCLUSIONS

We have proposed an image inpainting algorithm

which consists in first inpainting the structure layer

of the image, then using it to constrain the inpainting

process of the image. The proposed approach relies

on adapting the algorithm of Wei and Levoy to the

specificities of the second-moment matrix. The

obtained results quality was highly encouraging, in

terms of dynamics and structures preservation, and

proved that using the structure layer in the inpainting

process could be advantageous comparing to pure

intensity-based approaches.

However, using other non-parametric methods

than Wei and Levoy, and evaluating their efficiency

in the structure and image inpainting processes, is of

our interest. We also aim at comparing the

performance of the proposed algorithm with several

existing inpainting methods, using a large database.

In addition, we aim at reinforcing the use of the

proposed approach with different inpainting scan

types, different neighbourhood shapes and size.

Finally, it is necessary to consolidate the proposed

objective evaluation using second order statistics,

such as the autocorrelation, for example.

ACKNOWLEDGMENT

This work has been partly supported by a research

grant from the Higher Center for Research at the

Holy Spirit University of Kaslik (USEK), Lebanon.

REFERENCES

Kwatra, V., Schödl, A., Essa, I., Turk, G., Bobick, A.,

2003. "Graphcut Textures: Image and Video Synthesis

Using Graph Cuts," Proc. of ACM SIGGRAPH, pp.

277-286.

Bargteil, A. W., Sin, F., Michaels, J. E., Goktekin, T. G.,

O’Brien, J. F., 2006. "A Texture Synthesis Method for

Liquid Animations," Proc. ACM

SIGGRAPH/Eurographics Symposium on Computer

Animation, Vienna, Austria, September 2-4.

Yamauchi, H., Haber, J., Seidel, H-P., 2003. "Image

restoration using multiresolution texture synthesis and

image inpainting,” Proc. Int. Conf. Comput. Graph.

Winkenbach, G., Salesin, D. H., 1994. “Computer-

generated pen-and-ink illustration,” Proc. of

SIGGRAPH 94, pp. 91–100, Orlando, Florida.

Bertalmio, M., Sapiro, G., Caselles, V., Ballester, C.,

2000. “Image inpainting,” Proc. of the 27th annual

conference on Computer graphics and interactive

techniques, pp. 417-424.

Akl, A., Yaacoub, C., Donias, M., Da Costa, J.-P.,

Germain, C., 2015. Texture Synthesis Using the

Structure Tensor. IEEE Transactions on Image

Processing, 24 (11), art. no. 7163318, pp. 4082-4095.

Paget, R., Longstaff, I.D., 1998. "Texture synthesis via a

non causal nonparametric multiscale markov random

field," IEEE Trans. on Image Processing, vol. 7(6),

pp. 925–931.

Efros, A., Leung, T., 1999. “Texture synthesis by non-

parametric sampling,” International Conference on

Computer Vision, vol. 2, pp. 1033–1038.

ICPRAM 2017 - 6th International Conference on Pattern Recognition Applications and Methods

604

Chan, T., Shen, J., 2001. " Non-texture inpainting by

curvature-driven diffusions," J. Visual Comm. Image

Rep.

Wei, L.-Y., Levoy, M., 2000. "Fast texture synthesis using

tree-structured vector quantization," Proc. of ACM

SIGGRAPH 2000, pp. 479-488.

Portilla, J., Simoncelli, E.P., 2000. "A Parametric Texture

Model based on Joint Statistics of Complex Wavelet

Coefficients," Int'l Journal of Computer Vision,

vol.40(1), pp. 49-71.

Akl, A., Yaacoub, C., Donias, M., Da Costa, J.-P.,

Germain, C., 2015. Two-stage color texture synthesis

using the structure tensor field. GRAPP 2015 - 10th

International Conference on Computer Graphics

Theory and Applications; VISIGRAPP, Proceedings,

pp. 182-188.

Bigun, J., Granlund, G., 1987. “Optimal Orientation

Detection of Linear Symmetry,” International

Conference on Computer Vision, ICCV, (London).

Piscataway: IEEE Computer Society Press,

Piscataway. pp. 433-438.

Aujol, J., Ladjal, S., Masnou, S., 2009. "Exemplar-based

inpainting from a variational point of view," SIAM

Journal on Mathematical Analysis.

Akl, A., Yaacoub, C., Donias, M., Da Costa, J.-P.,

Germain, C., 2014. Structure tensor based synthesis of

directional textures for virtual material design. 2014

IEEE International Conference on Image Processing,

ICIP 2014, art. no. 7025986, pp. 4867-4871.

ITU-R Recommendation BT.601-7, “Studio encoding

parameters of digital television for standard 4:3 and

wide screen 16:9 aspect ratios,” ITU-R, Mar. 2011.

Xu, Y.Q., Guo, B., Shum, H., 2000. “Chaos mosaic: Fast

and memory efficient texture synthesis,” In Tech. Rep.

MSRTR-2000-32, Microsoft Research.

Akl, A., Iskandar, J., 2015. Structure tensor regularization

for texture analysis. 5th International Conference on

Image Processing, Theory, Tools and Applications

2015, IPTA 2015, art. no. 7367217, pp. 592-596.

Criminisi, A., Pérez, P., Toyama, K., 2004. "Region filling

and object removal by exemplar-based image

inpainting," Microsoft Research, Cambridge (UK) and

Redmond (US).

Kullback, S., Leibler, R.A.., 1951. “On information and

sufficiency,” Ann. Math. Statist., vol. 22(1), pp. 79–

86.

Akl, A., Iskandar, J., 2016. Second-moment matrix

adaptation for local orientation estimation.

International Conference on Systems, Signals, and

Image Processing, 2016-June.

A Tensor-based Technique for Structure-aware Image Inpainting

605