Lightweight Network Based on Multiregion Fusion for Facial Expression Recognition

Hong Tang; Junling Xiang; Haitao Chen; Lü Rongcheng; Zehao Xia

doi:10.3788/LOP213204

Laser & Optoelectronics Progress, Volume. 60, Issue 6, 0610006(2023)

Lightweight Network Based on Multiregion Fusion for Facial Expression Recognition

Hong Tang^1,2, Junling Xiang^1,2、*, Haitao Chen³, Lü Rongcheng¹, and Zehao Xia³

¹College of Communication and Information Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China

²Chongqing Key Laboratory of Mobile Communication Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, China

³International College, Chongqing University of Posts and Telecommunications, Chongqing 400065, China

show less

Abstract Get PDF(in Chinese)

Figures & Tables(11)

Fig. 1. Multi-area fusion network

Download full size

Fig. 2. VGG-BN-16 network structure

Download full size

Fig. 3. Key point mask generation map. (a) 68 key points; (b) 31 key points; (c) mask map

Download full size

Fig. 4. Key points and mask generation under occlusion.(a) Key points; (b) mask map

Download full size

Fig. 5. Overall framework of the model

Download full size

Fig. 6. Accuracy under different confidence thresholds

Download full size

Table 1. Performance comparison of different FER methods

View table

Table 1. Performance comparison of different FER methods

Method	Accuracy in RAF-DB /%	Accuracy in AffectNet /%
ResNet-50^［25］	82.83	54.37
VGG-16^［26］	80.96	51.11
pACNN^［11］	83.05	55.33
gACNN^［11］	85.07	58.78
E2-CapsNet^［27］	85.24
APM^［28］	85.17
DLP-CNN^［29］	84.13	54.47
Wgan^［30］	83.49
IPFR^［31］		57.40
SPA-SE^［32］		58.14
SPWFA-SE^［32］		59.23
Proposed method	85.39	58.81

Table 2. Facial expression recognition confusion matrix on RAF-DB dataset

View table

Table 2. Facial expression recognition confusion matrix on RAF-DB dataset

Expression	Angry	Disgust	Fear	Happy	Sadness	Surprise	Neutral
Angry	0.81	0.02	0.04	0.03	0.03	0.04	0.03
Disgust	0.06	0.59	0.02	0.08	0.09	0.03	0.11
Fear	0.02	0.04	0.57	0.03	0.14	0.15	0.05
Happy	0.00	0.01	0.00	0.92	0.02	0.01	0.04
Sadness	0.03	0.06	0.01	0.03	0.82	0.00	0.05
Surprise	0.04	0.01	0.01	0.02	0.01	0.85	0.06
Neutral	0.01	0.01	0.01	0.04	0.09	0.02	0.82

Table 3. Facial expression recognition confusion matrix on AffectNet dataset

View table

Table 3. Facial expression recognition confusion matrix on AffectNet dataset

Expression	Angry	Disgust	Fear	Happy	Sadness	Surprise	Neutral
Angry	0.36	0.24	0.02	0.13	0.07	0.03	0.15
Disgust	0.06	0.57	0.03	0.04	0.03	0.02	0.25
Fear	0.01	0.05	0.49	0.05	0.10	0.20	0.10
Happy	0.02	0.02	0.00	0.90	0.03	0.01	0.03
Sadness	0.03	0.06	0.02	0.04	0.56	0.02	0.27
Surprise	0.02	0.05	0.03	0.23	0.03	0.43	0.21
Neutral	0.01	0.04	0.02	0.13	0.04	0.05	0.71

Table 4. Ablation experimental results on RAF-DB dataset
View table
Table 4. Ablation experimental results on RAF-DB dataset
Method Accuracy /% Dataset
Local detail branch 42.17 RAF-DB
Global branch 78.26
Global branch+attention map 80.53
Proposed method（VGG-16） 84.59
Proposed method 85.39

Table 5. Pruned results of two methods

View table

Table 5. Pruned results of two methods

Method	Trim percentage	Accuracy /%	Parameters /10⁶	Pruned（parameter）/%	Flops /10⁹	Pruned（floating-point）/%
Proposed method	0	85.39	31.00		16.95
	40%	84.78	13.77	55.58	11.64	31.33
	50%	84.19	11.48	62.97	9.91	41.53
	60%	82.43	8.26	73.35	8.11	52.15
VGGNet	0	80.96	20.48		14.26
	40%	75.31	10.69	47.80	9.75	31.62
	50%	52.92	8.79	57.08	8.66	39.27
	60%	49.25	5.28	74.22	7.59	46.77

Tools

Get Citation

Copy Citation Text

Hong Tang, Junling Xiang, Haitao Chen, Lü Rongcheng, Zehao Xia. Lightweight Network Based on Multiregion Fusion for Facial Expression Recognition[J]. Laser & Optoelectronics Progress, 2023, 60(6): 0610006

Download Citation

EndNote(RIS)BibTex Plain Text

Set citation alerts for article

Save article for my favorites