VP-DARTS: Validated Pruning Differentiable Architecture Search

Tai-Che Feng, Sheng-De Wang

2024

Abstract

Recently Differentiable Architecture Search (DARTS) has gained increasing attention due to its simplicity and efficient search capability. However, such search methods have a significant chance of encountering overfitting, which can result in the performance collapse problem of the discovered models. In this paper, we proposed VP-DARTS, a validated pruning-based differentiable architecture search method using soft pruning, to address this issue. Firstly, unlike previous search methods, we consider the differentiable architecture search process as a model pruning problem. It prunes or removes unimportant operations from the supernet that contains all possible architectures to obtain the final model. We also show that the traditional hard pruning method would gradually reduce the capacity of the search space during training, leading to local optimal results. To get better architectures than hard pruning, we proposed using a parameterized soft pruning approach in our training process. Secondly, the original DARTS method selects the operation with the maximum architecture parameter on each edge to form the final architecture after training. But we found that this approach cannot truly reflect their importance. Therefore, we estimate the impact on the supernet of each candidate operation by using a subset of the validation set to evaluate its degree of importance. Finally, we implement our method on the NAS-Bench-201 search space, and the experimental results show that VP-DARTS is a robust search method that can obtain architectures with good performance and stable results.

Paper Citation

in Harvard Style

Feng T. and Wang S. (2024). VP-DARTS: Validated Pruning Differentiable Architecture Search. In Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART; ISBN 978-989-758-680-4, SciTePress, pages 47-57. DOI: 10.5220/0012296700003636

in Bibtex Style

@conference{icaart24,
author={Tai-Che Feng and Sheng-De Wang},
title={VP-DARTS: Validated Pruning Differentiable Architecture Search},
booktitle={Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART},
year={2024},
pages={47-57},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0012296700003636},
isbn={978-989-758-680-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 16th International Conference on Agents and Artificial Intelligence - Volume 2: ICAART
TI - VP-DARTS: Validated Pruning Differentiable Architecture Search
SN - 978-989-758-680-4
AU - Feng T.
AU - Wang S.
PY - 2024
SP - 47
EP - 57
DO - 10.5220/0012296700003636
PB - SciTePress