Learning 3D Human UV with Loose Clothing from Monocular Video

Meng-Yu Kuo, Jingfan Guo, Ryo Kawahara

2024

Abstract

We introduce a novel method for recovering a consistent and dense 3D geometry and appearance of a dressed person from a monocular video. Existing methods mainly focus on tight clothing and recover human geometry as a single representation. Our key idea is to regress the holistic 3D shape and appearance as a canonical displacement and albedo maps in the UV space, while fitting the visual observations across frames. Specifically, we represent the naked body shape by a UV-space SMPL model, and represent the other geometric details, including the clothing, as a shape displacement UV map. We obtain the temporally coherent overall shape by leveraging a differential mask loss and a pose regularization. The surface details in UV space are jointly learned in the course of non-rigid deformation with the differentiable neural rendering. Meanwhile, the skinning deformation in the garment region is updated periodically to adjust its residual non-rigid motion in each frame. We additionally enforce the temporal consistency of surface details by utilizing the optical flow. Experimental results on monocular videos demonstrate the effectiveness of the method. Our UV representation allows for simple and accurate dense 3D correspondence tracking of a person wearing loose clothing. We believe our work would benefit applications including VR/AR content creation.

Paper Citation

in Harvard Style

Kuo M., Guo J. and Kawahara R. (2024). Learning 3D Human UV with Loose Clothing from Monocular Video. In Proceedings of the 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP; ISBN 978-989-758-679-8, SciTePress, pages 122-129. DOI: 10.5220/0012414500003660

in Bibtex Style

@conference{visapp24,
author={Meng-Yu Kuo and Jingfan Guo and Ryo Kawahara},
title={Learning 3D Human UV with Loose Clothing from Monocular Video},
booktitle={Proceedings of the 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP},
year={2024},
pages={122-129},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012414500003660},
isbn={978-989-758-679-8},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 19th International Joint Conference on Computer Vision, Imaging and Computer Graphics Theory and Applications - Volume 4: VISAPP
TI - Learning 3D Human UV with Loose Clothing from Monocular Video
SN - 978-989-758-679-8
AU - Kuo M.
AU - Guo J.
AU - Kawahara R.
PY - 2024
SP - 122
EP - 129
DO - 10.5220/0012414500003660
PB - SciTePress