A PROTOTYPE FOR PRACTICAL EYE-GAZE CORRECTED VIDEO CHAT ON GRAPHICS HARDWARE

Maarten Dumont, Steven Maesen, Sammy Rogmans, Philippe Bekaert

2008

Abstract

We present a fully functional prototype to convincingly restore eye contact between two video chat participants, with a minimal amount of constraints. The proposed six-fold camera setup is easily integrated into the monitor frame, and is used to interpolate an image as if its virtual camera captured the image through a transparent screen. The peer user has a large freedom of movement, resulting in system specifications that enable genuine practical usage. Our software framework thereby harnesses the powerful computational resources inside graphics hardware, to achieve real-time performance up to 30 frames per second for 800 × 600 resolution images. Furthermore, an optimal set of finetuned parameters are presented, that optimizes the end-to-end performance of the application, and therefore is still able to achieve high subjective visual quality.

References

  1. Baker, H. H., Bhatti, N. T., Tanguay, D., Sobel, I., Gelb, D., Goss, M. E., Culbertson, W. B., and Malzbender, T. (2002). The coliseum immersive teleconferencing system. In Proceedings of the International Workshop on Immersive Telepresence, Juan-les-Pins, France.
  2. Brown, D. C. (1966). Decentering distortion of lenses. Photometric Engineering, 32(3):444-462.
  3. Chien, S.-Y., Yu, S.-H., Ding, L.-F., Huang, Y.-N., and Chen, L.-G. (2003). Efficient stereo video coding system for immersive teleconference with two-stage hybrid disparity estimation algorithm. In ICIP 2003: Proceedings of the 2003 International Conference on Image Processing, pages 749-752.
  4. Criminisi, A., Shotton, J., Blake, A., and Torr, P. H. S. (2003). Gaze manipulation for one-to-one teleconferencing. In ICCV 7803: Proceedings of the Ninth IEEE International Conference on Computer Vision, page 191, Washington, DC, USA. IEEE Computer Society.
  5. Gemmell, J., Toyama, K., Zitnick, C. L., Kang, T., and Seitz, S. (2000). Gaze awareness for videoconferencing: A software approach. IEEE MultiMedia, 7(4):26-35.
  6. Geys, I., Koninckx, T. P., and Van Gool, L. (2004). Fast interpolated cameras by combining a gpu based plane sweep with a max-flow regularisation algorithm. In 3DPVT 7804: Proceedings of the 3D Data Processing, Visualization, and Transmission, 2nd International Symposium, pages 534-541, Washington, DC, USA. IEEE Computer Society.
  7. Geys, I. and Van Gool, L. (2004). Extended view interpolation by parallel use of the gpu and the cpu. In Videometrics VIII: Proceedings of the Society of PhotoOptical Instrumentation Engineers (SPIE) Conference, volume 5665, pages 96-107.
  8. Guo, X., Gao, W., and Zhao, D. (2005). Motion vector prediction in multiview video coding. In ICIP 2005: Proceedings of the 2005 International Conference on Image Processing.
  9. Hsu, R.-L., Abdel-Mottaleb, M., and Jain, A. K. (2002). Face detection in color images. IEEE Transactions on Pattern Analysis and Machine Intelligence, 24(5):696-706.
  10. Lei, B. J. and Hendriks, E. A. (2002). Real-time multi-step view reconstruction for a virtual teleconference system. EURASIP Journal on Applied Signal Processing, 2002(1):1067-1087.
  11. Magnor, M., Pollefeys, M., Cheung, G., Matusik, W., and Theobalt, C. (2005). Video-based rendering. In SIGGRAPH 7805: ACM SIGGRAPH 2005 Courses, New York, NY, USA.
  12. Middlebury (2001). Middlebury stereo vision page. www.middlebury.edu/stereo.
  13. Nozick, V., Michelin, S., and Arquès, D. (2006). Real-time plane-sweep with local strategy. In Journal of WSCG, volume 14.
  14. Owens, J. D., Luebke, D., Govindaraju, N., Harris, M., Krger, J., Lefohn, A. E., and Purcell, T. J. (2007). A survey of general-purpose computation on graphics hardware. Computer Graphics Forum, 26(1):80-113.
  15. Rogmans, S., Lu, J., and Lafruit, G. (2008). A scalable end-to-end optimized real-time image-based rendering framework on graphics hardware. In Proceedings of 3DTV-CON, The True Vision, Capture, Transmission, and Display of 3D Video, pages 129-132, Istanbul, Turkey.
  16. Scharstein, D. and Szeliski, R. (2002). A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision, 47(1-3):7-42.
  17. Schreer, O., Brandenburg, N., Askar, S., and Trucco, M. (2001). A virtual 3d video-conference system providing semi-immersive telepresence: A real-time solution in hardware and software. In Proceedings of the eBusiness-eWork Conference, pages 184-190, Venice, Italy.
  18. Svoboda, T., Martinec, D., and Pajdla, T. (2005). A convenient multi-camera self-calibration for virtual environments. PRESENCE: Teleoperators and Virtual Environments, 14(4):407-422.
  19. Yang, R. and Pollefeys, M. (2003). Multi-resolution realtime stereo on commodity graphics hardware. In 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 211-220, Madison, WI, USA. IEEE Computer Society.
  20. Yang, R. and Welch, G. (2002). Fast image segmentation and smoothing using commodity graphics hardware. Journal of Graphics Tools, 7(4):91-100.
  21. Yang, R., Welch, G., and Bishop, G. (2002). Real-time consensus-based scene reconstruction using commodity graphics hardware. In PG 7802: Proceedings of the 10th Pacific Conference on Computer Graphics and Applications, page 225, Washington, DC, USA. IEEE Computer Society.
Download


Paper Citation


in Harvard Style

Dumont M., Maesen S., Rogmans S. and Bekaert P. (2008). A PROTOTYPE FOR PRACTICAL EYE-GAZE CORRECTED VIDEO CHAT ON GRAPHICS HARDWARE . In Proceedings of the International Conference on Signal Processing and Multimedia Applications - Volume 1: SIGMAP, (ICETE 2008) ISBN 978-989-8111-60-9, pages 236-243. DOI: 10.5220/0001932402360243


in Bibtex Style

@conference{sigmap08,
author={Maarten Dumont and Steven Maesen and Sammy Rogmans and Philippe Bekaert},
title={A PROTOTYPE FOR PRACTICAL EYE-GAZE CORRECTED VIDEO CHAT ON GRAPHICS HARDWARE},
booktitle={Proceedings of the International Conference on Signal Processing and Multimedia Applications - Volume 1: SIGMAP, (ICETE 2008)},
year={2008},
pages={236-243},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0001932402360243},
isbn={978-989-8111-60-9},
}


in EndNote Style

TY - CONF
JO - Proceedings of the International Conference on Signal Processing and Multimedia Applications - Volume 1: SIGMAP, (ICETE 2008)
TI - A PROTOTYPE FOR PRACTICAL EYE-GAZE CORRECTED VIDEO CHAT ON GRAPHICS HARDWARE
SN - 978-989-8111-60-9
AU - Dumont M.
AU - Maesen S.
AU - Rogmans S.
AU - Bekaert P.
PY - 2008
SP - 236
EP - 243
DO - 10.5220/0001932402360243