Journal of Electronic Science and Technology, Volume. 22, Issue 3, 100278(2024)
De-biased knowledge distillation framework based on knowledge infusion and label de-biasing techniques
Figures & Tables(6)
Fig. 1. Illustration of the classical knowledge distillation (KD) and our de-biased knowledge distillation (DeBKD).
Fig. 2. Workflow of the proposed de-biased knowledge distillation framework.
Table 1. Comparison of distillation effects across different datasets.
View tableView in ArticleTable 1. Comparison of distillation effects across different datasets.
Method D&C C&H C&B Acc (%) Pre (%) Rec (%) Acc (%) Pre (%) Rec (%) Acc (%) Pre (%) Rec (%) Teacher 84.98 84.22 84.22 90.03 90.13 90.38 93.23 93.40 93.26 KD-18 82.64 82.64 83.33 88.09 89.58 97.32 91.32 90.84 91.35 KD-8 79.54 80.09 79.54 83.78 84.68 83.07 82.94 86.93 82.77 FKD-18 82.98 82.88 83.05 88.74 89.94 87.67 91.79 91.87 91.65 FKD-8 79.44 79.47 79.36 86.42 86.56 86.36 84.21 84.33 84.13 DeBKD-18 83.12 83.58 83.12 89.78 89.85 89.57 93.34 93.85 93.31 DeBKD-8 80.80 80.81 80.80 86.73 87.48 86.15 89.06 90.23 88.97
Table 2. Impact of different learning freedom on the classification accuracy.
View tableView in ArticleTable 2. Impact of different learning freedom on the classification accuracy.
Learning freedom D&C C&H C&B DeBKD-8 DeBKD-18 DeBKD-8 DeBKD-18 DeBKD-8 DeBKD-18 1/8 78.98 79.42 70.62 85.21 82.81 88.54 1/10 79.48 82.08 83.71 86.73 86.79 91.02 1/12 80.80 83.12 84.38 89.78 89.06 93.36 1/14 79.64 80.36 86.73 85.81 85.47 91.15 1/16 77.40 78.48 85.05 84.29 83.85 91.15
Table 3. Impact of knowledge infusion (KI) on model performance (
Δ indicates the change in performance).View tableView in ArticleTable 3. Impact of knowledge infusion (KI) on model performance (
Δ indicates the change in performance).Method KI Dataset D&C C&H C&B Acc (%) Δ (%) Acc (%) Δ (%) Acc (%) Δ (%) KD × 79.54 — 83.78 — 82.94 — √ 79.86 +0.32 84.64 +0.86 83.61 +0.67 FKD × 79.44 — 86.42 — 84.21 — √ 79.97 +0.53 86.77 +0.35 85.02 +0.81 DeBKD × 79.64 — 86.73 — 87.11 — √ 80.80 +1.16 87.75 +1.44 89.06 +1.95
Tools
Get Citation
Copy Citation Text
Yan Li, Tai-Kang Tian, Meng-Yu Zhuang, Yu-Ting Sun. De-biased knowledge distillation framework based on knowledge infusion and label de-biasing techniques[J]. Journal of Electronic Science and Technology, 2024, 22(3): 100278
Paper Information
Category:
Received: Jun. 6, 2024
Accepted: Aug. 15, 2024
Published Online: Oct. 11, 2024
The Author Email: Li Yan (yanli@std.uestc.edu.cn), Sun Yu-Ting (skye.sun@uq.edu.au)
© Copyright 2018-2021 | Chinese Laser Press.
All Rights Reserved 沪ICP备15018463号-20