Authors:
Tai-Che Feng
and
Sheng-De Wang
Affiliation:
Department of Electrical Engineering, National Taiwan University, Taipei, Taiwan
Keyword(s):
Deep Learning, Model Optimization, Neural Architecture Search, Differentiable Architecture Search, Model Compression, Model Pruning, Soft Pruning.
Abstract:
Recently Differentiable Architecture Search (DARTS) has gained increasing attention due to its simplicity and efficient search capability. However, such search methods have a significant chance of encountering overfitting, which can result in the performance collapse problem of the discovered models. In this paper, we proposed VP-DARTS, a validated pruning-based differentiable architecture search method using soft pruning, to address this issue. Firstly, unlike previous search methods, we consider the differentiable architecture search process as a model pruning problem. It prunes or removes unimportant operations from the supernet that contains all possible architectures to obtain the final model. We also show that the traditional hard pruning method would gradually reduce the capacity of the search space during training, leading to local optimal results. To get better architectures than hard pruning, we proposed using a parameterized soft pruning approach in our training process. S
econdly, the original DARTS method selects the operation with the maximum architecture parameter on each edge to form the final architecture after training. But we found that this approach cannot truly reflect their importance. Therefore, we estimate the impact on the supernet of each candidate operation by using a subset of the validation set to evaluate its degree of importance. Finally, we implement our method on the NAS-Bench-201 search space, and the experimental results show that VP-DARTS is a robust search method that can obtain architectures with good performance and stable results.
(More)