Editorial Board

Modern deep neural networks have numerous types of different layers with varied connections between layers. For example, skip-connection and its sub-modules are used to promote convergence of models. Currently, most deep neural network architectures are developed based on human experience, through a long and tedious process of trial and error. With the growing interest in neural architecture search (NAS), attempts have been made to automate the process of identifying effective architectures for a given deep learning problem. NAS can be defined as a technique for automating the design of artificial neural networks (ANN) [1]. The early concept of NAS included defining a search space, search strategy, and performance estimation strategy. Fig. 1 shows the concept of NAS, which is a gradient-based method for identifying good architectures [2]. The concept is based on the observation that the structure and connectivity of a neural network can be typically specified by a variable-length string.

Fig. 1. 
Concept of NAS [2].

NAS is computationally expensive and time-consuming, and uses 450 GPUs for up to 3–4 days. Meanwhile, efficient neural architecture search (ENAS) has been proposed, which is a fast and inexpensive approach for automatic model design [3]. The main contribution of ENAS was to improve the efficiency of NAS by sharing parameters among child models, using much fewer GPU-hours than existing automatic model design approaches. The best existing architecture search algorithms are computationally demanding despite their outstanding performance.

For instance, obtaining a state-of-the-art architecture for CIFAR-10 and ImageNet required 2000 GPU days of reinforcement learning (RL), or 3150 GPU days of evolution [4,5]. Moreover, in all the popular existing approaches, architecture search is treated as a black-box optimization problem over a discrete domain, which requires a large number of architecture evaluations. Therefore, differentiable architecture search (DARTS) was a method proposed for efficiently searching architecture, which tackled the problem differently. Its main contribution was to relax the search space to be continuous, instead of searching over a discrete set of candidates, such that the architecture could be optimized with respect to its validation set performance by gradient descent [6]. Fig. 2 shows the concept of DARTS.

Fig. 2. 
Overview of DARTS [6] ; (a) Operations on the edges are initially unknown, (b) Continuous relaxation of the search by placing a mixture of candidate operations on each edge, (c) Joint optimization of mixing probabilities, and network weights by solving a bilevel optimization problem, and (d) Inducing the final architecture from the learned mixing probabilities.

All the aforementioned architectures are highly successful in computer vision tasks, and image classification has always served as a fundamental task for NAS. Hence, networks designed and pre-trained on classification tasks are commonly used. However, the object-detection task requires both localization and classification information for each instance, and here, the backbone architecture influences the performance. Ultimately, the architecture designed for image classification may not perform well for detection. To overcome this problem, some existing works proposed ways to improve the backbone architecture [7-10].

Pre-training is an unavoidable and costly procedure. However, training from the beginning on the target task takes numerous iterations when compared with fine-tuning from a pre-trained one. As ImageNet pre-training has been a standard rule for many computer vision tasks, there are plenty of trained models available [11]. To take advantage of these models, an NAS model was proposed based on the parameter adaptation paradigm. The proposed NAS could adapt the parameters of the manually designed base network, which is pre-trained on ImageNet or MobileNetV2 [12]. The base network is expanded to the super network, which is essentially the search space in the proposed NAS. Thus, this architecture search could obtain the optimal target architecture for detection. With this method, there is no need for pre-training on a large-scale dataset. The effectiveness of the proposed NAS is tested via experiments on detection tasks.

MobileNetV2, the most commonly used network for designing search spaces in NAS methods, is adapted as the base model. To adapt the network for the detection task, the two architecture elements (that is, kernel size and depth) of the network model are fine-tuned. Fig. 3 shows the overall framework of the proposed NAS, wherein the base network is expanded to its super-network for searching. Further, the parameters of the base network are added to the final architecture. The framework is composed of two important aspects: (1) remapping the parameters of the base model, and (2) network adaptation.

Fig. 3. 
Framework of proposed NAS.

An ImageNet pre-trained MobileNetV2 model is used for the proposed NAS, and an object detection task is performed. The experiment is conducted on the MS-COCO dataset [14]. In the search process of architecture adaptation, 50% of the data is randomly sampled from the original set for validation. The input images are resized to 320 × 320. A standard RMSProp optimizer with a weight decay of 4 × 10^-5 is used for operation weights in the search process. For the first 500 iterations, the learning rate is set from 0 to 0.03. After that, the learning rate is decayed by 0.1 at 22 epochs. The architecture optimization begins at 20 epochs. λ is set to 0.2, and τ is set to 10 for the loss function, wherein λ denotes the cross-entropy loss and τ term controls the multiplication and addition arithmetics. The search process takes 24 epochs in total, that is, 76 h on a single TITAN Xp. Table 1 lists the results on the COCO dataset, with a comparison to the literature DetNas [10] in terms of a number of parameters, λ, MAdds, and detection accuracy. The proposed NAS proves to be lightweight, using only 4.6M parameters, which is far less than those of DetNas. However, the mAP is lower than the existing one. The MAdds also exhibits comparable results, reducing the operations to 1B only.

Table 2 shows the comparison of computational cost for the object detection task. The experiments using the proposed NAS are conducted on a single TITAN Xp GPU. Here GDs denotes the number of GPU days. The MMDetection framework is used which is an open source object detection toolbox based on PyTorch.

Fig. 4 shows the architecture found after the search process, where Kx_Ey denotes the kernel size of the depthwise convolution is ‘x’ and expansion ratio is ‘y’.

Fig. 4. 
Obtained architecture.

Fig. 5 shows the visualization of the searched architecture on the SSDLite framework, where MB denotes the inverted residual block proposed in MobileNetV2.

Fig. 5. 
Proposed NAS on SSDLite.

Fig. 6 shows the visualization of object detection results on the MS-COCO validation dataset.

Fig. 6. 
Detection on MS-COCO validation dataset.

In this study, the existing popular NAS methods are reviewed, and an efficient NAS method for object detection tasks is proposed. However, the method is limited, and is not explored in different frameworks. Our main contribution is taking complete advantage of the models pre-trained on ImageNet. The manually designed network is adopted as the base model. The strategy of remapping the parameters fully utilizes the base model, which, in turn, accelerates the overall network efficiently. With this method, researchers can quickly adapt other manually designed networks to various other tasks. In future, it could be more beneficial to perform other computer vision tasks by introducing various adaptation techniques.