Real-time Video-based View Interpolation of Soccer Events using Depth-selective Plane Sweeping

Patrik Goorts, Cosmin Ancuti, Maarten Dumont, Sammy Rogmans, Philippe Bekaert

2013

Abstract

In this paper we present a novel technique to synthesize virtual camera viewpoints for soccer events. Our real-time video-based rendering technique does not require a precise estimation of the scene geometry. We initially segment the dynamic parts of the scene to consequently estimate a depth map of the filtered foreground regions using a plane sweep strategy. The depth map is indicatively segmented to depth information per player. A consecutive plane sweep is used, where the depth sweep is limited to the depth range of each player individually, effectively removing major ghosting artifacts, such as third legs or ghost players. The background and shadows are interpolated independently. For maximum performance our technique is implemented using a combination of NVIDIA’s shaders language Cg and NVIDIA’s general purpose computing framework CUDA. The rendered results of an actual soccer game demonstrate the usability and accuracy of our framework.

References

  1. Dumont, M., Rogmans, S., Maesen, S., and Bekaert, P. (2009). Optimized two-party video chat with restored eye contact using graphics hardware. e-Business and Telecommunications, pages 358-372.
  2. Germann, M., Hornung, A., Keiser, R., Ziegler, R., Wü rmlin, S., and Gross, M. (2010). Articulated billboards for video-based rendering.
  3. Goorts, P., Rogmans, S., and Bekaert, P. (2009). Optimal Data Distribution for Versatile Finite Impulse Response Filtering on Next-Generation Graphics Hardware using CUDA . In Proc. of The Fifteenth Intl. Conference on Parallel and Distributed Systems, pages 300-307.
  4. Grau, O., Hilton, A., Kilner, J., Miller, G., Sargeant, T., and Starck, J. (2007). A Free-Viewpoint Video System for Visualization of Sport Scenes. SMPTE motion imaging journal, 116(5/6):213.
  5. Hartley, R. and Zisserman, A. (2003). Multiple view geometry in computer vision, volume 2. Cambridge Univ Press.
  6. Hayashi, K. and Saito, H. (2006). Synthesizing FreeViewpoing Images from Multiple View Videos in Soccer Stadium. In Intl. Conf. on Computer Graphics, Imaging and Visualisation, pages 220-225.
  7. Hilton, A., Guillemaut, J., Kilner, J., Grau, O., and Thomas, G. (2011). 3d-tv production from conventional cameras for sports broadcast. IEEE Transactions on Broadcasting, (99):1-1.
  8. Hirakawa, K. and Parks, T. (2005). Adaptive homogeneitydirected demosaicing algorithm. Image Processing, IEEE Transactions on, 14(3):360-369.
  9. Inamoto, N. and Saito, H. (2007). Virtual viewpoint replay for a soccer match by view interpolation from multiple cameras. IEEE Transactions on Multimedia, 9(6):1155-1166.
  10. Kanade, T., Rander, P., and Narayanan, P. (1997). Virtualized reality: Constructing virtual worlds from real scenes. Multimedia, IEEE, 4(1):34-47.
  11. Kutulakos, K. and Seitz, S. (2000). A theory of shape by space carving. Intl. Journal of Computer Vision, 38(3):199-218.
  12. Lowe, D. (2004). Distinctive image features from scaleinvariant keypoints. Intl. journal of computer vision, 60(2):91-110.
  13. Matusik, W., Buehler, C., Raskar, R., Gortler, S., and McMillan, L. (2000). Image-based visual hulls.
  14. Miller, G., Hilton, A., and Starck, J. (2005). Interactive free-viewpoint video.
  15. Ohta, Y., Kitahara, I., Kameda, Y., Ishikawa, H., and Koyama, T. (2007). Live 3D Video in Soccer Stadium. Intl. Journal of Computer Vision, 75(1):173-187.
  16. Rogmans, S., Dumont, M., Cuypers, T., Lafruit, G., and Bekaert, P. (2009a). Complexity reduction of realtime depth scanning on graphics hardware.
  17. Rogmans, S., Dumont, M., Lafruit, G., and Bekaert, P. (2009b). Migrating real-time depth image-based rendering from traditional to next-gen gpgpu.
  18. Seitz, S., Curless, B., Diebel, J., Scharstein, D., and Szeliski, R. (2006). A comparison and evaluation of multi-view stereo reconstruction algorithms.
  19. Seitz, S. and Dyer, C. (1999). Photorealistic scene reconstruction by voxel coloring. Intl. Journal of Computer Vision, 35(2):151-173.
  20. Svoboda, T., Martinec, D., and Pajdla, T. (2005). A convenient multicamera self-calibration for virtual environments. Presence: Teleoperators & Virtual Environments, 14(4):407-422.
  21. Yang, R., Pollefeys, M., Yang, H., and Welch, G. (2004). A unified approach to real-time, multi-resolution, multibaseline 2d view synthesis and 3d depth estimation using commodity graphics hardware. Intl. Journal of Image and Graphics, 4(4):627-651.
  22. Yang, R. and Welch, G. (2002). Fast image segmentation and smoothing using commodity graphics hardware. Journal of Graphics Tools, 7(4):91-100.
  23. Yang, R., Welch, G., and Bishop, G. (2003). Real-time consensus-based scene reconstruction using commodity graphics hardware.
  24. Ziegler, G. and Rasmusson, A. (2010). Efficient volume segmentation on the gpu.
  25. Zitnick, C., Kang, S., Uyttendaele, M., Winder, S., and Szeliski, R. (2004). High-quality video view interpolation using a layered representation.
Download


Paper Citation


in Harvard Style

Goorts P., Ancuti C., Dumont M., Rogmans S. and Bekaert P. (2013). Real-time Video-based View Interpolation of Soccer Events using Depth-selective Plane Sweeping . In Proceedings of the International Conference on Computer Vision Theory and Applications - Volume 2: VISAPP, (VISIGRAPP 2013) ISBN 978-989-8565-48-8, pages 131-137. DOI: 10.5220/0004217301310137


in Bibtex Style

@conference{visapp13,
author={Patrik Goorts and Cosmin Ancuti and Maarten Dumont and Sammy Rogmans and Philippe Bekaert},
title={Real-time Video-based View Interpolation of Soccer Events using Depth-selective Plane Sweeping},
booktitle={Proceedings of the International Conference on Computer Vision Theory and Applications - Volume 2: VISAPP, (VISIGRAPP 2013)},
year={2013},
pages={131-137},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0004217301310137},
isbn={978-989-8565-48-8},
}


in EndNote Style

TY - CONF
JO - Proceedings of the International Conference on Computer Vision Theory and Applications - Volume 2: VISAPP, (VISIGRAPP 2013)
TI - Real-time Video-based View Interpolation of Soccer Events using Depth-selective Plane Sweeping
SN - 978-989-8565-48-8
AU - Goorts P.
AU - Ancuti C.
AU - Dumont M.
AU - Rogmans S.
AU - Bekaert P.
PY - 2013
SP - 131
EP - 137
DO - 10.5220/0004217301310137