DDSAS: Dynamic and Differentiable Space-Architecture Search

Longxing Yang (Institute of Computing Technology, Chinese Academy of Sciences); Yu Hu (Institute of Computing Technology, Chinese Academy of Sciences)*; Shun Lu (Institute of Computing Technology, Chinese Academy of Sciences); Zihao Sun (Institute of Computing Technology, Chinese Academy of Sciences); Jilin Mei (Institute of Computing Technology, Chinese Academy of Sciences); Yiming Zeng (Tecent ADlab); Zhiping Shi (Capital Normal University); Yinhe Han (Institute of Computing Technology Chinese Academy of Sciences); Xiaowei Li (Institute Of Computing Technology Chinese Academy Of Sciences)
PMLR Page

Abstract

Neural Architecture Search (NAS) has made remarkable progress in automatically designing neural networks. However, existing differentiable NAS and stochastic NAS methods are either biased towards exploitation and thus may converge to a local minimum, or biased towards exploration and thus converge slowly. In this work, we propose a Dynamic and Differentiable Space-Architecture Search (DDSAS) method to address the exploration-exploitation dilemma. DDSAS dynamically samples space, searches architectures in the sampled subspace with gradient descent, and leverages the Upper Confidence Bound (UCB) to balance exploitation and exploration. The whole search space is elastic, offering flexibility to evolve and to consider resource constraints. Experiments on image classification datasets demonstrate that with only 4GB memory and 3 hours for searching, DDSAS achieves 2.39% test error on CIFAR10, 16.26% test error on CIFAR100, and 23.9% test error when transferring to ImageNet. When directly searching on ImageNet, DDSAS achieves comparable accuracy with more than 6.5 times speedup over state-of-the-art methods.