Depth-Adaptive Dynamic Neural Networks: A Survey

Yi Sun; Jian Li; Xin Xu; Yuru Wang

doi:10.3788/LOP202259.1415008

Laser & Optoelectronics Progress, Volume. 59, Issue 14, 1415008(2022)

Depth-Adaptive Dynamic Neural Networks: A Survey

Yi Sun, Jian Li^*, Xin Xu^**, and Yuru Wang

College of Intelligence Science and Technology, National University of Defense Technology, Changsha 410000, Hunan , China

show less

Abstract Get PDF(in Chinese)

Figures & Tables(13)

Fig. 1. Depth adaptive neural networks for automatically adjusting inference depth based on the input complexity. (a) Network structure for processing simple input; (b) network structure for processing complex input

Download full size

Fig. 2. Typical structures of depth-adaptive neural networks. (a) Multi-exit neural network; (b) skip-connection network

Download full size

Fig. 3. Information exchange scheme of output module

Download full size

Fig. 4. Network structure MSDNet based on multi-scale down sampling^[34]

Download full size

Fig. 5. Classification accuracy of MSDNet and Ensemble-ResNets on ImageNet dataset^[34]

Download full size

Fig. 6. Basic structure of Gate Module

Download full size

Fig. 7. Samples with different complexity. (a) Samples with relatively simple texture and background; (b) complex samples

Download full size

Fig. 8. Shared parameters $θ$ receives conflict gradients from different exits. (a) Conflicted gradients have negative cosine similarity value; (b) level of gradient conflict in the training stage

Download full size

Fig. 9. Network structure with dense connection

Download full size

Table 1. Overview about the depth-adaptive neural networks

View table

Table 1. Overview about the depth-adaptive neural networks

Method

Network structure

Depth-adaptive policy

（input-complexity estimation）

Training method

Multi-exit

Independent output branches^［33-36］；

Additive/geometric ensemble^［37-38］；

Multi-scale feature fusion^［39］；

Multi-scale receptive field^{［34，38］}

Confidence-based early exiting^{［33-37，40-42］}；

Mutual information estimation early exiting^［43-44］；

Learning policy networks for early exiting^［45-47］

Weighted gradient descent^{［28，33，48］}；

Knowledge distillation^［49-51］；

Gradient adjustment^{［38，52］}

Skip-style

Centralized gate module^［53］；

Distributed gate module^［54-56］；

Randomly block dropout^［57-58］

Skipping non-linear blocks^［53-58］

Sparse regularization^［56］；

Reinforcement-learning based^［53］

Table 2. Performance comparison of different information fusion approaches on CIFAR100 dataset
View table
Table 2. Performance comparison of different information fusion approaches on CIFAR100 dataset
Method Exit-1 Exit-2 Exit-3 Exit-4
Baseline 66.77 70.31 71.93 73.0
Additive-ensemble 66.04 70.70 72.49 73.23
Geometric-ensemble 63.91 70.35 72.67 73.01
Multi-scale feature fusion 66.60 70.53 72.75 73.05

Table 3. Performance comparison of multi-exit networks trained by knowledge distillation
View table
Table 3. Performance comparison of multi-exit networks trained by knowledge distillation
Method Exit-1 Exit-2 Exit-3 Exit-4 Exit-5
MSDNet^［34］ 79.25 86.46 89.15 89.83 90.75
IMPR^［38］ 80.15 87.89 90.52 91.33 91.74
DBT^［50］ 80.80 86.92 88.82 89.15 89.73
H-DBT^［49］ 83.06 87.12 90.85 91.9 92.04

Table 4. Performance comparison of multi-exit networks after using different gradient adjustment approaches on ImageNet dataset
View table
Table 4. Performance comparison of multi-exit networks after using different gradient adjustment approaches on ImageNet dataset
Method Exit-1 Exit-2 Exit-3 Exit-4 Exit-5
MSDNet^［34］ 58.48 65.96 68.66 69.48 71.03
IMPR-GE^［38］ 57.75 65.54 69.24 70.27 71.89
PCgrad+GE^［52］ 57.62 64.87 68.93 71.05 72.45

Tools

Get Citation

Copy Citation Text

Yi Sun, Jian Li, Xin Xu, Yuru Wang. Depth-Adaptive Dynamic Neural Networks: A Survey[J]. Laser & Optoelectronics Progress, 2022, 59(14): 1415008

Download Citation

EndNote(RIS)BibTex Plain Text

Set citation alerts for article

Save article for my favorites

Paper Information

Category: Machine Vision

Received: Apr. 12, 2022

Accepted: May. 23, 2022

Published Online: Jul. 1, 2022

The Author Email: Li Jian (lijian@nudt.edu.cn), Xu Xin (xinxu@nudt.edu.cn)

DOI:10.3788/LOP202259.1415008

Topics

laser devices and laser physics

Lasers and Laser Optics

Laser physics

laser manufacturing

Instrumentation, Measurement and Metrology

Table 1. Overview about the depth-adaptive neural networks

Table 1. Overview about the depth-adaptive neural networks

Table 2. Performance comparison of different information fusion approaches on CIFAR100 dataset

Table 2. Performance comparison of different information fusion approaches on CIFAR100 dataset

Table 3. Performance comparison of multi-exit networks trained by knowledge distillation

Table 3. Performance comparison of multi-exit networks trained by knowledge distillation

Table 4. Performance comparison of multi-exit networks after using different gradient adjustment approaches on ImageNet dataset

Table 4. Performance comparison of multi-exit networks after using different gradient adjustment approaches on ImageNet dataset