REAL-TIME VIDEO MATTING FOR MIXED REALITY USING DEPTH GENERATED TRIMAPS

Nicholas Beato, Remo Pillat, Charles E. Hughes

2012

Abstract

In Mixed Reality scenarios, background replacement is a common way to immerse a user in a synthetic environment. Properly identifying the background pixels in an image or video is a difficult problem known as matting. Proper alpha mattes usually come from human guidance, special hardware setups, or color dependent algorithms. This is a consequence of the under-constrained nature of the per pixel alpha blending equation. While the field of natural image matting has made progress finding a least squares solution for an alpha matte, the generation of trimaps, indicating regions of known foreground and background pixels, normally requires human interaction or offline computation. We overcome these limitations by combining a low fidelity depth image that segments the original video signal with a real-time parallel natural image matting technique that favors objects with similar colors in the foreground and background. This allows us to provide real-time alpha mattes for Mixed Reality scenarios that take place in relatively controlled environments. As a consequence, while monochromatic backdrops (such as green screens or retro-reflective material) aid the algorithm’s accuracy, they are not an explicit requirement.

References

  1. Beato, N., Zhang, Y., Colbert, M., Yamazawa, K., and Hughes, C. E. (2009). Interactive chroma keying for mixed reality. Comput. Animat. Virtual Worlds, 20:405-415.
  2. Bergh, F. V. D. and Lalioti, V. (1999). Software chroma keying in an immersive virtual environment. South African Computer Journal, 24:155-162.
  3. Chuang, Y.-Y., Curless, B., Salesin, D. H., and Szeliski, R. (2001). A bayesian approach to digital matting. In Proceedings of IEEE CVPR 2001, volume 2, pages 264-271. IEEE Computer Society.
  4. Gastal, E. S. L. and Oliveira, M. M. (2010). Shared sampling for real-time alpha matting. Computer Graphics Forum, 29(2):575-584. Proceedings of Eurographics.
  5. Gvili, R., Kaplan, A., Ofek, E., Yahav, Giora, ., and Benton, S. A. (2003). Depth keying. Stereoscopic Displays and Virtual Reality Systems X, 5006:564-574.
  6. Joshi, N., Matusik, W., and Avidan, S. (2006). Natural video matting using camera arrays. ACM Trans. Graph., 25:779-786.
  7. Levin, A., Lischinski, D., and Weiss, Y. (2006). A closed form solution to natural image matting. In Proceedings of the 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Volume 1, pages 61-68, Washington, DC, USA. IEEE Computer Society.
  8. Levin, A., Rav-acha, A., and Lischinski, D. (2008). Spectral matting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30:1699-1712.
  9. McGuire, M., Matusik, W., and Yerazunis, W. (2006). Practical, real-time studio matting using dual imagers. In Rendering Techniques 2006 (Proceedings of the 17th Eurographics Symposium on Rendering). Eurographics, Eurographics Association.
  10. Microsoft (last accessed May 2011). Kinect full body interaction. http://www.xbox.com/kinect.
  11. Mishima, Y. (1992). A software chromakeyer using polyhedric slice. NICOGRAPH'92, pages 44-52.
  12. Pitie and, F. and Kokaram, A. (2010). Matting with a depth map. In Image Processing (ICIP), 2010 17th IEEE International Conference on, pages 21-24.
  13. Rhemann, C., Rother, C., Wang, J., Gelautz, M., Kohli, P., and Rott, P. (2009). A perceptually motivated online benchmark for image matting. In Computer Vision and Pattern Recognition, 2009. CVPR 2009. IEEE Conference on, pages 1826-1833.
  14. Ruzon, M. and Tomasi, C. (2000). Alpha estimation in natural images. In Computer Vision and Pattern Recognition, 2000. Proceedings. IEEE Conference on, volume 1, pages 18-25.
  15. Shi, J. and Malik, J. (2000). Normalized cuts and image segmentation. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI).
  16. Smith, A. R. and Blinn, J. F. (1996). Blue screen matting. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, SIGGRAPH 7896, pages 259-268, New York, NY, USA. ACM.
  17. Sun, J., Li, Y., Bing, S., and yeung Shum, K. H. (2006). Flash matting. In In Proceedings of ACM SIGGRAPH, pages 772-778. ACM Press.
  18. Vlahos, P. (July 11, 1978). Comprehensive Electronic Compositing System. U.S. Patent 4,100,569. Expired.
  19. Wang, J. and Cohen, M. (2007). Optimized color sampling for robust matting. In Computer Vision and Pattern Recognition, 2007. CVPR 7807. IEEE Conference on, pages 1-8.
  20. Wang, O., Finger, J., Yang, Q., Davis, J., and Yang, R. (2007). Automatic natural video matting with depth. In Computer Graphics and Applications, 2007. PG 7807. 15th Pacific Conference on, pages 469-472.
Download


Paper Citation


in Harvard Style

Beato N., Pillat R. and E. Hughes C. (2012). REAL-TIME VIDEO MATTING FOR MIXED REALITY USING DEPTH GENERATED TRIMAPS . In Proceedings of the International Conference on Computer Graphics Theory and Applications and International Conference on Information Visualization Theory and Applications - Volume 1: GRAPP, (VISIGRAPP 2012) ISBN 978-989-8565-02-0, pages 280-288. DOI: 10.5220/0003870402800288


in Bibtex Style

@conference{grapp12,
author={Nicholas Beato and Remo Pillat and Charles E. Hughes},
title={REAL-TIME VIDEO MATTING FOR MIXED REALITY USING DEPTH GENERATED TRIMAPS},
booktitle={Proceedings of the International Conference on Computer Graphics Theory and Applications and International Conference on Information Visualization Theory and Applications - Volume 1: GRAPP, (VISIGRAPP 2012)},
year={2012},
pages={280-288},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0003870402800288},
isbn={978-989-8565-02-0},
}


in EndNote Style

TY - CONF
JO - Proceedings of the International Conference on Computer Graphics Theory and Applications and International Conference on Information Visualization Theory and Applications - Volume 1: GRAPP, (VISIGRAPP 2012)
TI - REAL-TIME VIDEO MATTING FOR MIXED REALITY USING DEPTH GENERATED TRIMAPS
SN - 978-989-8565-02-0
AU - Beato N.
AU - Pillat R.
AU - E. Hughes C.
PY - 2012
SP - 280
EP - 288
DO - 10.5220/0003870402800288