Authors:
Devon Tao
and
Lucas Bang
Affiliation:
Computer Science Department, Harvey Mudd College, Claremont, CA, U.S.A.
Keyword(s):
Neural Architecture Search, Fitness Landscape, Neural Networks.
Abstract:
Neural Architecture Search (NAS) research has historically faced issues of reproducibility and comparability of algorithms. To address these problems, researchers have created NAS benchmarks for NAS algorithm evaluation. However, NAS search spaces themselves are not yet well understood. To contribute to an understanding of NAS search spaces, we use the framework of fitness landscape analysis to analyze the topology search space of NATS-Bench, a popular cell-based NAS benchmark. We examine features of density of states, local optima, fitness distance correlation (FDC), fitness distance rank correlations, basins of attraction, neutral networks, and autocorrelation in order to characterize the difficulty and describe the shape of the NATS-Bench topology search space on CIFAR-10, CIFAR-100, and ImageNet16-120 image classification problems. Our analyses show that the difficulties associated with each fitness landscape could correspond to the difficulties of the image classification proble
ms themselves. Furthermore, we demonstrate the importance of using multiple metrics for a nuanced understanding of an NAS fitness landscape.
(More)