TND-NAS: Towards Non-Differentiable Objectives in Differentiable Neural Architecture Search

Bo Lyu, Shiping Wen

2022

Abstract

Differentiable architecture search has gradually become the mainstream research topic in the field of Neural Architecture Search (NAS) for its high efficiency compared with the early heuristic NAS (EA-based, RL-based) methods. Differentiable NAS improves the search efficiency, but no longer naturally capable of tackling the non-differentiable objectives. Researches in the multi-objective NAS field target this point but requires vast computational resources cause of the individual training of each candidate architecture. We propose the TND-NAS, which discretely sample architectures based on architecture parameter α (without sampling controller), and directly optimize α by policy gradient algorithm. Our representative experiment takes two objectives (Parameters, Accuracy) as an example, we achieve a series of high-performance compact architectures on CIFAR10 (1.09M/3.3%, 2.4M/2.95%, 9.57M/2.54%) and CIFAR100 (2.46M/18.3%, 5.46M/16.73%, 12.88M/15.20%) datasets.

Paper Citation

in Harvard Style

Lyu B. and Wen S. (2022). TND-NAS: Towards Non-Differentiable Objectives in Differentiable Neural Architecture Search. In Proceedings of the 3rd International Symposium on Automation, Information and Computing - Volume 1: ISAIC; ISBN 978-989-758-622-4, SciTePress, pages 177-181. DOI: 10.5220/0011917300003612

in Bibtex Style

@conference{isaic22,
author={Bo Lyu and Shiping Wen},
title={TND-NAS: Towards Non-Differentiable Objectives in Differentiable Neural Architecture Search},
booktitle={Proceedings of the 3rd International Symposium on Automation, Information and Computing - Volume 1: ISAIC},
year={2022},
pages={177-181},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011917300003612},
isbn={978-989-758-622-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 3rd International Symposium on Automation, Information and Computing - Volume 1: ISAIC
TI - TND-NAS: Towards Non-Differentiable Objectives in Differentiable Neural Architecture Search
SN - 978-989-758-622-4
AU - Lyu B.
AU - Wen S.
PY - 2022
SP - 177
EP - 181
DO - 10.5220/0011917300003612
PB - SciTePress