RGB-D and Thermal Sensor Fusion - Application in Person Tracking

Ignacio Rocco Spremolla, Michel Antunes, Djamila Aouada, Björn Ottersten

2016

Abstract

Many systems combine RGB cameras with other sensor modalities for fusing visual data with complementary environmental information in order to achieve improved sensing capabilities. This article explores the possibility of fusing a commodity RGB-D camera and a thermal sensor. We show that using traditional methods, it is possible to accurately calibrate the complete system and register the three RGB-D-T data sources. We propose a simple person tracking algorithm based on particle filters, and show how to combine the mapped pixel information from the RGB-D-T data. Furthermore, we use depth information to adaptively scale the tracked target area when radial displacements from the camera occur. Experimental results provide evidence that this allows for a significant tracking performance improvement in situations with large radial displacements, when compared to using only a tracker based on RGB or RGB-T data.

References

  1. Bouguet, J.-Y. (2004). Camera calibration toolbox for matlab.
  2. Choi, C. and Christensen, H. I. (2013). Rgb-d object tracking: A particle filter approach on gpu. In Intelligent Robots and Systems (IROS), 2013 IEEE/RSJ International Conference on, pages 1084-1091. IEEE.
  3. Endres, F., Hess, J., Engelhard, N., Sturm, J., Cremers, D., and Burgard, W. (2012). An evaluation of the rgbd slam system. In Robotics and Automation (ICRA), 2012 IEEE International Conference on.
  4. Everingham, M., Van Gool, L., Williams, C. K., Winn, J., and Zisserman, A. (2010). The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-338.
  5. Jafari, O. H., Mitzel, D., and Leibe, B. (2014). Real-time rgb-d based people detection and tracking for mobile robots and head-worn cameras. In Robotics and Automation (ICRA), 2014 IEEE International Conference on, pages 5636-5643. IEEE.
  6. Kumar, S., Marks, T. K., and Jones, M. (2014). Improving person tracking using an inexpensive thermal infrared sensor. In Computer Vision and Pattern Recognition Workshops (CVPRW), 2014 IEEE Conference on, pages 217-224. IEEE.
  7. Luber, M., Spinello, L., and Arras, K. O. (2011). People tracking in rgb-d data with on-line boosted target models. In Proc. of The International Conference on Intelligent Robots and Systems (IROS).
  8. Nakagawa, W., Matsumoto, K., de Sorbier, F., Sugimoto, M., Saito, H., Senda, S., Shibata, T., and Iketani, A. (2014). Visualization of temperature change using rgb-d camera and thermal camera. In Computer Vision-ECCV 2014 Workshops, pages 386-400. Springer.
  9. Nummiaro, K., Koller-Meier, E., and Van Gool, L. (2002). Object tracking with an adaptive color-based particle filter. In Pattern Recognition, pages 353-360. Springer.
  10. Nummiaro, K., Koller-Meier, E., and Van Gool, L. (2003). An adaptive color-based particle filter. Image and Vision Computing, 21(1):99-110.
  11. Pérez, P., Hue, C., Vermaak, J., and Gangnet, M. (2002). Color-based probabilistic tracking. In Computer visionECCV 2002, pages 661-675. Springer.
  12. Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R., Kipman, A., and Blake, A. (2011). Real-time human pose recognition in parts from single depth images. In Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition.
  13. Stolkin, R., Rees, D., Talha, M., and Florescu, I. (2012). Bayesian fusion of thermal and visible spectra camera data for region based tracking with rapid background adaptation. In Multisensor Fusion and Integration for Intelligent Systems (MFI), 2012 IEEE Conference on, pages 192-199. IEEE.
  14. Susperregi, L., Martínez-Otzeta, J. M., Ansuategui, A., Ibarguren, A., and Sierra, B. (2013). Rgb-d, laser and thermal sensor fusion for people following in a mobile robot. Int. J. Adv. Robot. Syst.
  15. Talha, M. and Stolkin, R. (2012). Adaptive fusion of infrared and visible spectra camera data for particle filter tracking of moving targets. In Sensors, 2012 IEEE, pages 1-4. IEEE.
  16. Vemulapalli, R., Arrate, F., and Chellappa, R. (2014). Human action recognition by representing 3d skeletons as points in a lie group. In Computer Vision and Pattern Recognition (CVPR), 2014 IEEE Conference on.
  17. Vidas, S., Lakemond, R., Denman, S., Fookes, C., Sridharan, S., and Wark, T. (2012). A mask-based approach for the geometric calibration of thermal-infrared cameras. Instrumentation and Measurement, IEEE Transactions on, 61(6):1625-1635.
  18. Vidas, S., Moghadam, P., and Bosse, M. (2013). 3d thermal mapping of building interiors using an rgb-d and thermal camera. In Robotics and Automation (ICRA), 2013 IEEE International Conference on, pages 2311- 2318. IEEE.
Download


Paper Citation


in Harvard Style

Rocco Spremolla I., Antunes M., Aouada D. and Ottersten B. (2016). RGB-D and Thermal Sensor Fusion - Application in Person Tracking . In Proceedings of the 11th Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 3: VISAPP, (VISIGRAPP 2016) ISBN 978-989-758-175-5, pages 610-617. DOI: 10.5220/0005717706100617


in Bibtex Style

@conference{visapp16,
author={Ignacio Rocco Spremolla and Michel Antunes and Djamila Aouada and Björn Ottersten},
title={RGB-D and Thermal Sensor Fusion - Application in Person Tracking},
booktitle={Proceedings of the 11th Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 3: VISAPP, (VISIGRAPP 2016)},
year={2016},
pages={610-617},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0005717706100617},
isbn={978-989-758-175-5},
}


in EndNote Style

TY - CONF
JO - Proceedings of the 11th Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 3: VISAPP, (VISIGRAPP 2016)
TI - RGB-D and Thermal Sensor Fusion - Application in Person Tracking
SN - 978-989-758-175-5
AU - Rocco Spremolla I.
AU - Antunes M.
AU - Aouada D.
AU - Ottersten B.
PY - 2016
SP - 610
EP - 617
DO - 10.5220/0005717706100617