Affine Invariant Self-similarity for Exemplar-based Inpainting

Vadim Fedorov, Pablo Arias, Gabriele Facciolo, Coloma Ballester

2016

Abstract

This paper presents a new method for exemplar-based image inpainting using transformed patches. We build upon a recent affine invariant self-similarity measure which automatically transforms patches to compare them in an appropriate manner. As a consequence, it intrinsically extends the set of available source patches to copy information from. When comparing two patches, instead of searching for the appropriate patch transformation in a highly dimensional parameter space, our approach allows us to determine a single transformation from the texture content in both patches. We incorporate the affine invariant similarity measure in a variational formulation for inpainting and present an algorithm together with experimental results illustrating this approach.

References

  1. Arias, P., Facciolo, G., Caselles, V., and Sapiro, G. (2011). A variational framework for exemplar-based image inpainting. International Journal of Computer Vision, 93:319-347.
  2. Aujol, J.-F., Ladjal, S., and Masnou, S. (2010). Exemplarbased inpainting from a variational point of view. SIAM Journal on Mathematical Analysis, 42(3):1246- 1285.
  3. Ballester, C., Bertalmío, M., Caselles, V., Sapiro, G., and Verdera, J. (2001). Filling-in by joint interpolation of vector fields and gray levels. IEEE Transactions on Image Processing, 10(8):1200-1211.
  4. Ballester, C., Calderero, F., Caselles, V., and Facciolo, G. (2014). Multiscale analysis of similarities between images on riemannian manifolds. SIAM Journal Multiscale Modeling and Simulation, 12(2):616-649.
  5. Ballester, C. and Gonzalez, M. (1998). Affine invariant texture segmentation and shape from texture by variational methods. Journal of Mathematical Imaging and Vision, 9(2):141-171.
  6. Barnes, C., Shechtman, E., Finkelstein, A., and Goldman, D. B. (2009). PatchMatch: a randomized correspondence algorithm for structural image editing. In ACM SIGGRAPH 2009 Papers, pages 1-11. ACM.
  7. Barnes, C., Shechtman, E., Goldman, D. B., and Finkelstein, A. (2010). The generalized PatchMatch correspondence algorithm. In European Conference on Computer Vision.
  8. Bertalmío, M., Sapiro, G., Caselles, V., and Ballester, C. (2000). Image inpainting. In Proceedings of SIGGRAPH,, pages 417-424.
  9. Buades, A., Coll, B., and Morel, J.-M. (2005). A non local algorithm for image denoising. In Proceedings of the IEEE Conference on CVPR, volume 2, pages 60-65.
  10. Cao, F., Gousseau, Y., Masnou, S., and Prez, P. (2011). Geometrically guided exemplar-based inpainting. SIAM Journal on Imaging Sciences, 4(4):1143-1179.
  11. Chan, T. and Shen, J. H. (2001a). Mathematical models for local nontexture inpaintings. SIAM J. App. Math., 62(3):1019-43.
  12. Chan, T. and Shen, J. H. (2001b). Nontexture inpainting by curvature-driven diffusions. Journal of Visual Communication and Image Representation, 12(4):436- 449.
  13. Criminisi, A., Pérez, P., and Toyama, K. (2004). Region filling and object removal by exemplar-based inpainting. IEEE Trans. on IP, 13(9):1200-1212.
  14. Demanet, L., Song, B., and Chan, T. (2003). Image inpainting by correspondence maps: a deterministic approach. Applied and Computational Mathematics, 1100:217-50.
  15. Drori, I., Cohen-Or, D., and Yeshurun, H. (2003). Fragment-based image completion. In ACM SIGGRAPH 2003 Papers, volume 22, pages 303-12.
  16. Efros, A. A. and Leung, T. K. (1999). Texture synthesis by non-parametric sampling. In Proceedings of the IEEE ICCV, pages 1033-38.
  17. Fedorov, V., Arias, P., Sadek, R., Facciolo, G., and Ballester, C. (2015). Linear multiscale analysis of similarities between images on riemannian manifolds: Practical formula and affine covariant metrics. SIAM Journal on Imaging Sciences, 8(3):2021-2069.
  18. Foi, A. and Boracchi, G. (2012). Foveated self-similarity in nonlocal image filtering. Human Vision and Electronic Imaging XVII, 8291(1):829110.
  19. Garding, J. and Lindeberg, T. (1994). Direct estimation of local surface shape in a fixating binocular vision system. In Eklundh, Lecture Notes in Computer Science, pages 365-376.
  20. Gilboa, G. and Osher, S. J. (2008). Nonlocal operators with applications to image processing. Multiscale Modeling and Simulation, 7(3):1005-1028.
  21. Ga°rding, J. (1992). Shape from texture for smooth curved surfaces in perspective projection. Journal of Mathematical Imaging and Vision, 2(4):327-350.
  22. Ga°rding, J. and Lindeberg, T. (1996). Direct computation of shape cues using scale-adapted spatial derivative operators. International Journal of Computer Vision, 17(2):163-191.
  23. Hays, J. and Efros, A. (2007). Scene completion using millions of photographs. In SIGGRAPH, New York, NY, USA. ACM.
  24. Huang, J.-. B., Kopf, J., Ahuja, N., and Kang, S. B. (2013). Transformation guided image completion. In International Conference on Computational Photography, pages 1-9.
  25. Huang, J. B., Kang, S. B., Ahuja, N., and Kopf, J. (2014). Image completion using planar structure guidance. ACM Transactions on Graphics (Proceedings of SIGGRAPH 2014), 33(4):129:1-129:10.
  26. Kawai, N., Sato, T., and Yokoya, N. (2009). Image inpainting considering brightness change and spatial locality of textures and its evaluation. In Advances in Image and Video Technology, pages 271-282.
  27. Lowe, D. (2004). Distinctive image features from scaleinvariant keypoints. International Journal of Computer Vision, 60(2):91-110.
  28. Mansfield, A., Prasad, M., Rother, C., Sharp, T., Kohli, P., and van Gool, L. (2011). Transforming image completion. In Proceedings of BMVC, pages 121.1-121.11.
  29. Masnou, S. (2002). Disocclusion: a variational approach using level lines. IEEE Transactions on Image Processing, 11(2):68-76.
  30. Masnou, S. and Morel, J.-M. (1998). Level lines based disocclusion. In Proceedings of IEEE ICIP, volume 3, pages 259-263.
  31. Matas, J., Chum, O., Urban, M., and Pajdla, T. (2004). Robust wide-baseline stereo from maximally stable extremal regions. Image and Vision Computing, 22(10):761-767.
  32. Mikolajczyk, K. and Schmid, C. (2004). Scale & affine invariant interest point detectors. International Journal of Computer Vision, 60(1):63-86.
  33. Pavic, D., Schonefeld, V., and Kobbelt, L. (2006). Interactive image completion with perspective correction. The Visual Computer, 22(9-11):671-681.
  34. Peyré, G. (2009). Manifold models for signals and images. Computer Vision and Image Understanding, 113(2):249-260.
  35. Pizarro, L., Mrázek, P., Didas, S., Grewenig, S., and Weickert, J. (2010). Generalised nonlocal image smoothing. International Journal of Computer Vision, 90:62-87.
  36. Wang, Z. (2008). Image affine inpainting. InImage Analysis and Recognition, volume 5112 of Lecture Notes in Computer Science, pages 1061-1070.
  37. Wexler, Y., Shechtman, E., and Irani, M. (2007). Space-time completion of video. IEEE Transactions on PAMI, 29(3):463-476.
Download


Paper Citation


in Harvard Style

Fedorov V., Arias P., Facciolo G. and Ballester C. (2016). Affine Invariant Self-similarity for Exemplar-based Inpainting . In Proceedings of the 11th Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 3: VISAPP, (VISIGRAPP 2016) ISBN 978-989-758-175-5, pages 48-58. DOI: 10.5220/0005728100480058


in Bibtex Style

@conference{visapp16,
author={Vadim Fedorov and Pablo Arias and Gabriele Facciolo and Coloma Ballester},
title={Affine Invariant Self-similarity for Exemplar-based Inpainting},
booktitle={Proceedings of the 11th Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 3: VISAPP, (VISIGRAPP 2016)},
year={2016},
pages={48-58},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005728100480058},
isbn={978-989-758-175-5},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 11th Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 3: VISAPP, (VISIGRAPP 2016)
TI - Affine Invariant Self-similarity for Exemplar-based Inpainting
SN - 978-989-758-175-5
AU - Fedorov V.
AU - Arias P.
AU - Facciolo G.
AU - Ballester C.
PY - 2016
SP - 48
EP - 58
DO - 10.5220/0005728100480058