Free Viewpoint Video for Soccer using Histogram-based Validity Maps in Plane Sweeping

Patrik Goorts, Steven Maesen, Maarten Dumont, Sammy Rogmans, Philippe Bekaert

2014

Abstract

In this paper, we present a method to accomplish free viewpoint video for soccer scenes. This will allow the rendering of a virtual camera, such as a virtual rail camera, or a camera moving around a frozen scene. We use 7 static cameras in a wide baseline setup (10 meters apart from each other). After debayering and segmentation, a crude depth map is created using a plane sweep approach. Next, this depth map is filtered and used in a second, depth-selective plane sweep by creating validity maps per depth. The complete method employs NVIDIA CUDA and traditional GPU shaders, resulting in a fast and scalable solution. The results, using real images, show the effective removal of artifacts, yielding high quality images for a virtual camera.

References

  1. Baillard, C. and Zisserman, A. (2000). A plane-sweep strategy for the 3d reconstruction of buildings from multiple images. International Archives of Photogrammetry and Remote Sensing, 33(B2; PART 2):56-62.
  2. Dumont, M., Rogmans, S., Maesen, S., and Bekaert, P. (2009). Optimized two-party video chat with restored eye contact using graphics hardware. e-Business and Telecommunications, pages 358-372.
  3. Eisemann, M., De Decker, B., Magnor, M., Bekaert, P., De Aguiar, E., Ahmed, N., Theobalt, C., and Sellent, A. (2008). Floating textures. Computer Graphics Forum, 27(2):409-418.
  4. Germann, M., Hornung, A., Keiser, R., Ziegler, R., Würmlin, S., and Gross, M. (2010). Articulated billboards for video-based rendering. Computer Graphics Forum, 29(2):585-594.
  5. Glasbey, C. A. and Mardia, K. V. (1998). A review of image-warping methods. Journal of applied statistics, 25(2):155-171.
  6. Goorts, P., Ancuti, C., Dumont, M., and Bekaert, P. (2013a). Real-time video-based view interpolation of soccer events using depth-selective plane sweeping. In Proceedings of the Eight International Conference on Computer Vision Theory and Applications (VISAPP 2013). INSTICC.
  7. Goorts, P., Dumont, M., Rogmans, S., and Bekaert, P. (2012a). An end-to-end system for free viewpoint video for smooth camera transitions. In Proceedings of the Second International Conference on 3D Imaging (IC3D 2012). 3D Stereo Media.
  8. Goorts, P., Maesen, S., Dumont, M., Rogmans, S., and Bekaert, P. (2013b). Optimization of free viewpoint interpolation by applying adaptive depth plane distributions in plane sweeping. In Proceedings of the Eleventh International Conference on Signal Processing and Multimedia Applications (SIGMAP 2013).
  9. Grau, O., Hilton, A., Kilner, J., Miller, G., Sargeant, T., and Starck, J. (2007). A Free-Viewpoint Video System for Visualization of Sport Scenes. SMPTE motion imaging journal, 116(5/6):213.
  10. Hartley, R. and Zisserman, A. (2003). Multiple view geometry in computer vision, volume 2. Cambridge Univ Press.
  11. Hayashi, K. and Saito, H. (2006). Synthesizing FreeViewpoing Images from Multiple View Videos in Soccer Stadium. In Proceedings of the International Conference on Computer Graphics, Imaging and Visualisation, pages 220-225.
  12. Hilton, A., Guillemaut, J., Kilner, J., Grau, O., and Thomas, G. (2011). 3d-tv production from conventional cameras for sports broadcast. IEEE Transactions on Broadcasting, (99):1-1.
  13. Kutulakos, K. and Seitz, S. (2000). A theory of shape by space carving. International Journal of Computer Vision, 38(3):199-218.
  14. Levoy, M. and Hanrahan, P. (1996). Light field rendering. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques, pages 31- 42. ACM.
  15. Lowe, D. (2004). Distinctive image features from scaleinvariant keypoints. International Journal of Computer Vision, 60(2):91-110.
  16. Malvar, H., He, L., and Cutler, R. (2004). High-quality linear interpolation for demosaicing of bayer-patterned color images. In Proceedings of the International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2004), volume 3, pages 485-488. IEEE.
  17. Matas, J., Chum, O., Urban, M., and Pajdla, T. (2002). Robust wide baseline stereo from maximally stable extremal regions. In Proceedings of the British machine vision conference, volume 1, pages 384-393.
  18. Matusik, W., Buehler, C., Raskar, R., Gortler, S., and McMillan, L. (2000). Image-based visual hulls. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 369- 374. ACM Press/Addison-Wesley Publishing Co.
  19. McMillan, L. and Bishop, G. (1995). Plenoptic modeling: An image-based rendering system. In Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, pages 39-46. ACM.
  20. Miller, G., Hilton, A., and Starck, J. (2005). Interactive free-viewpoint video. In Proceedings of the International Conference on Visual Media Production (CVMP 2005), pages 50-59. Citeseer.
  21. Ohta, Y., Kitahara, I., Kameda, Y., Ishikawa, H., and Koyama, T. (2007). Live 3D Video in Soccer Stadium. International Journal of Computer Vision, 75(1):173- 187.
  22. Rogmans, S., Dumont, M., Cuypers, T., Lafruit, G., and Bekaert, P. (2009). Complexity reduction of real-time depth scanning on graphics hardware. In Proceedings of the International Conference on Computer Vision Theory and Applications (VISAPP 2009).
  23. Scharstein, D. and Szeliski, R. (2002). A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision, 47(1):7-42.
  24. Seitz, S. and Dyer, C. (1999). Photorealistic scene reconstruction by voxel coloring. Inernational. Journal of Computer Vision, 35(2):151-173.
  25. Svoboda, T., Martinec, D., and Pajdla, T. (2005). A convenient multicamera self-calibration for virtual environments. Presence: Teleoperators & Virtual Environments, 14(4):407-422.
  26. Tuytelaars, T. and Van Gool, L. (2000). Wide baseline stereo matching based on local, affinely invariant regions. In Proceedings of the British machine vision conference, volume 2, page 4.
  27. Wang, L., Jin, H., Yang, R., and Gong, M. (2008). Stereoscopic inpainting: Joint color and depth completion from stereo images. In Proceedings of the International Conference on Computer Vision and Pattern Recognition (CVPR 2008), pages 1-8. IEEE.
  28. Yang, R., Pollefeys, M., Yang, H., and Welch, G. (2004). A unified approach to real-time, multi-resolution, multibaseline 2d view synthesis and 3d depth estimation using commodity graphics hardware. International Journal of Image and Graphics, 4(4):627-651.
  29. Yang, R. and Welch, G. (2002). Fast image segmentation and smoothing using commodity graphics hardware. Journal of Graphics Tools, 7(4):91-100.
  30. Yang, R., Welch, G., and Bishop, G. (2003). Real-time consensus-based scene reconstruction using commodity graphics hardware. Computer Graphics Forum, 22(2):207-216.
Download


Paper Citation


in Harvard Style

Goorts P., Maesen S., Dumont M., Rogmans S. and Bekaert P. (2014). Free Viewpoint Video for Soccer using Histogram-based Validity Maps in Plane Sweeping . In Proceedings of the 9th International Conference on Computer Vision Theory and Applications - Volume 3: VISAPP, (VISIGRAPP 2014) ISBN 978-989-758-009-3, pages 378-386. DOI: 10.5220/0004730003780386


in Bibtex Style

@conference{visapp14,
author={Patrik Goorts and Steven Maesen and Maarten Dumont and Sammy Rogmans and Philippe Bekaert},
title={Free Viewpoint Video for Soccer using Histogram-based Validity Maps in Plane Sweeping},
booktitle={Proceedings of the 9th International Conference on Computer Vision Theory and Applications - Volume 3: VISAPP, (VISIGRAPP 2014)},
year={2014},
pages={378-386},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0004730003780386},
isbn={978-989-758-009-3},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 9th International Conference on Computer Vision Theory and Applications - Volume 3: VISAPP, (VISIGRAPP 2014)
TI - Free Viewpoint Video for Soccer using Histogram-based Validity Maps in Plane Sweeping
SN - 978-989-758-009-3
AU - Goorts P.
AU - Maesen S.
AU - Dumont M.
AU - Rogmans S.
AU - Bekaert P.
PY - 2014
SP - 378
EP - 386
DO - 10.5220/0004730003780386