Chinese Journal of Liquid Crystals and Displays, Volume. 38, Issue 8, 1095(2023)
Behavior recognition based on time-dependent attention
Table 0. [in Chinese]
View tableView in ArticleTable 0. [in Chinese]
算法1 ：自动帧采样算法输入：原始视频：RawVideo，帧采样率：r，有效视频帧采样数量：γ
输出：包含视频帧数据的缓存：DataBuffer
变量：开始索引：start_idx结束索引：end_idx帧有效采样计数器：sample_count采样计数器：count视频帧数量：frame_count帧采样数量：fram_sample_count
1 获取视频帧数量：frame_count=GetFrameCountFun（RawVideo）
2 初始化参数：
3 start_idx=0
4 end_idx=frame_count-1
5 sample_count=0
6 count=0
7 fram_sample_count=frame_count // r-1
8 if（frame_count>300）
9 end_idx=RandInt（300，frame_count）
10 start_idx=end_idx-300
11 fram_sample_count=301 // r-1
12 初始化数据缓存：InitFun（DataBuffer）
13 while（count<end_idx）do
14 frame=GetFrame（RawVideo）
15 if（count<start_idx）
16 count+=1
17 continue
18 if（count>end_idx）
19 break
20 if（sample_count<fram_sample_count）
21 DataBuffer［sample_count］=frame
22 sample_count+=1
23 count+=1
24 if（len（DataBuffer）<γ+2）：
25 delete DataBuffer
26 else
27 DataBuffer←随机从DataBuffer 顺序抽取
γ帧数据
28 return DataBuffer
Table 1. Parameters of network structure
View tableView in ArticleTable 1. Parameters of network structure
Slow Fast Outsize T×S2 global average pool，concate，fc # classes Input — — 64×2242 Data stride 16，12 stride 2，12 slow：4×2242
fast：32×2242
Conv1 1×72，64
stride 1，22
5×72，8
stride 1，22
slow：4×1122
fast：32×1122
Pool1 1×32，64
stride 1，22
1×32，8
stride 1，22
slow：4×562
fast：32×562
Res_TCAM1 1×12，64
1×32，64×3
TCAM，256
3×12，8
1×32，8×3
TCAM，32
slow：4×562
fast：32×562
Res_TCAM2 1×12，128
1×32，128×4
TCAM，512
3×12，16
1×32，16×4
TCAM，64
slow：4×282
fast：32×282
Res_TCAM3 3×12，256
1×32，256×6
TCAM，1 024
3×12，32
1×32，32×6
TCAM，128
slow：4×142
fast：32×142
Res_TCAM4 3×12，512
1×32，512×4
TCAM，2 048
3×12，64
1×32，64×4
TCAM，256
slow：4×72
fast：32×72
Table 2. Effect of γ on the model
View tableView in ArticleTable 2. Effect of γ on the model
Model γ Top-1/% GFlops SlowFast 64 92.68 27.99 96 93.12 41.99 128 94.55 55.98 SlowFast+TCAM 64 95.15 28.74 96 96.03 43.08 128 96.11 57.49
Table 3. Effects of different frame sampling rates on the model
View tableView in ArticleTable 3. Effects of different frame sampling rates on the model
T×τ top-1/% GFlops 2×32 92.89 14.86 4×16 95.15 28.53 8×8 96.73 56.79
Table 4. Effects of time-dependent attention and lateral connectivity on experimental results
View tableView in ArticleTable 4. Effects of time-dependent attention and lateral connectivity on experimental results
model TCAM Lateral Lateral（Ours） Top-1/% SlowFast（b+） — √ — 92.68 SlowFast — — √ 93.79 Slow Only √ — √ 94.70 Fast Only（ours） √ — √ 95.83 SlowFast √ — √ 95.15
Table 5. Comparison of recognition accuracy of TCAM with other methods
View tableView in ArticleTable 5. Comparison of recognition accuracy of TCAM with other methods
Class Method Pretraining Spatial resolution Backbone UCF101/% HMDB51/% Baseline IDT+FV［ 1 ］— — — 85.90 57.20 IDT+BOVW［ 1 ］— — — 87.90 61.10 Two-stream［ 7 ］ImageNet 224×224 ConvNets 88.00 59.40 C3D ［ 8 ］— 128×171 C3D 85.2 — No-attention Two-stream-I3D［ 9 ］— 224×224 Inception V1 93.40 66.40 Two-stream-I3D［ 9 ］ImageNet+Kinetics 224×224 Inception V1 98.00 80.70 Hidden Two-stream［ 26 ］UCF101 224×224 I3D 97.10 78.70 R（2+1）D［ 27 ］Kinetics 112×112 ResNet34 94.97 — Attention AR3D_V1［ 22 ］— 112×112 R3D 88.39 51.53 AR3D_V2［ 22 ］— 112×112 R3D 89.28 52.51 STA-LSTM［ 10 ］ImageNet 224×224 GoogLeNet 90.80 59.70 STA-LSTM［ 10 ］Kinetics 224×224 I3D 98.66 81.45 Chen［ 28 ］Kinetics 224×224 Inception-v3 96.13 74.45 Chen［ 28 ］Kinetics 224×224 BN-Inception 96.18 75.17 NST［ 29 ］ImageNet+ Kinetics 224×224 I3D 96.00 76.10 文献［ 11 ］ImageNet+ Kinetics 224×224 ResNet50 96.3 77.7 文献［ 12 ］ImageNet 128×128 InceptionV3 94.9 — 文献［ 13 ］ImageNet 299×299 InceptionV3 94.9 70.8 TCAM（Ours） — 224×224 ResNet50 95.83 76.10 TCAM（Ours） Kinetics 224×224 ResNet50 98.16 82.30
Get Citation
Copy Citation Text
Kuan LIU, Wei WANG, Hong-ting SHEN, Hong-tao HOU, Min-zhen GUO, Zi-jiang LUO. Behavior recognition based on time-dependent attention[J]. Chinese Journal of Liquid Crystals and Displays, 2023, 38(8): 1095
Category: Research Articles
Received: Oct. 11, 2022
Accepted: --
Published Online: Oct. 9, 2023
The Author Email: Zi-jiang LUO (luozijiang@mail.gufe.edu.cn)
