Batch Contrastive Regularization for Deep Neural Network

Muhammad Tanveer, Hung Khoon Tan, Hui Fuang Ng, Maylor Karhang Leung, Joon Huang Chuah

2020

Abstract

As neural network becomes deeper, it becomes more capable of generating more powerful representation for a wide variety of tasks. However, deep neural network has a large number of parameters and easy to overfit the training samples. In this paper, we present a new regularization technique, called batch contrastive regularization. Regularization is performed by comparing samples collectively via contrastive loss which encourages intra-class compactness and inter-class separability in an embedded Euclidean space. To facilitate learning of embedding features for contrastive loss, a two-headed neural network architecture is used to decouple regularization classification. During inference, the regularization head is discarded and the network operates like any conventional classification network. We also introduce bag sampling to ensure sufficient positive samples for the classes in each batch. The performance of the proposed architecture is evaluated on CIFAR-10 and CIFAR-100 databases. Our experiments show that features regularized by contrastive loss has strong generalization performance, yielding over 8% improvement on ResNet50 for CIFAR-100 when trained from scratch.

Paper Citation

in Harvard Style

Tanveer M., Tan H., Ng H., Leung M. and Chuah J. (2020). Batch Contrastive Regularization for Deep Neural Network. In Proceedings of the 12th International Joint Conference on Computational Intelligence (IJCCI 2020) - Volume 1: NCTA; ISBN 978-989-758-475-6, SciTePress, pages 368-377. DOI: 10.5220/0010135303680377

in Bibtex Style

@conference{ncta20,
author={Muhammad Tanveer and Hung Khoon Tan and Hui Fuang Ng and Maylor Karhang Leung and Joon Huang Chuah},
title={Batch Contrastive Regularization for Deep Neural Network},
booktitle={Proceedings of the 12th International Joint Conference on Computational Intelligence (IJCCI 2020) - Volume 1: NCTA},
year={2020},
pages={368-377},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0010135303680377},
isbn={978-989-758-475-6},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 12th International Joint Conference on Computational Intelligence (IJCCI 2020) - Volume 1: NCTA
TI - Batch Contrastive Regularization for Deep Neural Network
SN - 978-989-758-475-6
AU - Tanveer M.
AU - Tan H.
AU - Ng H.
AU - Leung M.
AU - Chuah J.
PY - 2020
SP - 368
EP - 377
DO - 10.5220/0010135303680377
PB - SciTePress