Optics and Precision Engineering, Volume. 31, Issue 4, 552(2023)

Combining residual shrinkage and spatio-temporal context for behavior detection network

Zhong HUANG1...2,*, Mengyuan TAO1, Min HU2, Juan LIU1 and Shengbao ZHAN1 |Show fewer author(s)
Author Affiliations
  • 1School of Electronic Engineering and Intelligent Manufacturing, Anqing Normal University, Anqing24633,China
  • 2School of Computer Science and Information Engineering, Hefei University of Technology, Hefei30009, China
  • show less
    Figures & Tables(16)
    Network structure of RS-STCBD
    Unit of 3D residual shrinkage with channel-adaptive soft thresholds (3D-RSST)
    Detection results of Cricket movement on R-C3D and RS-STCBD
    Detection results of Billiards movement on R-C3D and RS-STCBD
    Detection results of High Jump movement on R-C3D and RS-STCBD
    Influence of balance factor on mAP@0.5
    Influence of NMS threshold on mAP@0.5
    Influence of number of high-quality proposals on mAP@0.5
    • Table 1. Structure parameter of feature subnet

      View table
      View in Article

      Table 1. Structure parameter of feature subnet

      LayerStructure parameterSize of feature
      Conv1

      7×7×7,64,

      stride 1(L),2(HW

      64×L×H2×W2
      Max Pool

      3×3×3

      max pool,stride 2

      64×L2×H4×W4
      R1(3D-RSST,64)✕364×L2×H4×W4
      R2(3D-RSST,128)✕4128×L4×H8×W8
      R3(3D-RSST,256)✕6256×L8×H16×W16
    • Table 2. Parameter of multilayer convolution

      View table
      View in Article

      Table 2. Parameter of multilayer convolution

      InputSizeStridePaddingOutput
      Conv1256(3,3,3)(1,1,1)(1,1,1)512
      Conv2512(3,3,3)(1,2,2)(1,1,1)512
      Conv3512(3,3,3)(1,2,2)(1,1,1)512
    • Table 3. Behavioral datasets of THUMOS14 and ActivityNet1.2

      View table
      View in Article

      Table 3. Behavioral datasets of THUMOS14 and ActivityNet1.2

      DatasetTHUMOS14ActivityNet1.2

      Training Set

      (Behavior Proposal)

      2004 819(7 151)

      Validation Set

      (Behavior Proposal)

      200(3 007)2 383(3 582)

      Test Set

      (Behavior Proposal)

      213(3 358)2 480
      Number of Behavior20100
    • Table 4. Parameter setting of network

      View table
      View in Article

      Table 4. Parameter setting of network

      ParameterReference value
      OptimizerSGD
      Epoch8
      Learning Rate104
      Length of Frame768
      Size of Anchor24568910121416
      Threshold of NMS0.7
      K150
      Balance Factor0.4
    • Table 5. mAP comparison of different methods in THUMOS14

      View table
      View in Article

      Table 5. mAP comparison of different methods in THUMOS14

      MethodmAP@iou/%
      0.10.20.30.40.50.60.7
      R-C3D1054.551.544.835.628.9--
      MSA-Net1765.660.752.341.629.720.610.1
      SSN866.059.451.941.029.8--
      LGN9--52.343.433.023.614
      DecoupleSSAD7--49.944.435.824.313.6
      BackTAL18--54.445.536.326.214.8
      RS-STCBD53.351.947.443.536.926.415.6
    • Table 6. Comparison of AP@0.5 of each behavior under different methods in THUMOS14

      View table
      View in Article

      Table 6. Comparison of AP@0.5 of each behavior under different methods in THUMOS14

      BehaviorSMS4R-C3D10LGN9Two-stream R-C3D12RS-STCBD
      Baseball Pitch15.326.120.519.921.6
      Basketball Dunk6.454.021.855.340.1
      Billiards9.88.312.511.213.9
      Clean and Jerk44.527.930.233.246.2
      Cliff Diving16.448.250.554.061.3
      Cricket Bowling11.431.616.931.128.0
      Cricket Shot3.610.95.511.614.3
      Diving11.426.249.431.135.4
      Frisbee Catch10.720.13.321.513.0
      Golf Swing14.216.124.532.848.0
      Hammer Throw30.343.256.558.371.5
      High Jump34.530.959.437.938.5
      Javelin Throw21.447.050.147.246.6
      Long Jump40.657.469.562.178.7
      Pole Vault17.142.769.757.771.6
      Shotput17.419.426.420.015.6
      Soccer Penalty8.515.819.119.212.5
      Tennis Swing5.716.615.911.621.7
      Throw Discus31.529.243.341.053.0
      Volleyball Spiking5.35.615.911.37.0
      mAP@0.517.828.933.033.436.9
    • Table 7. mAP comparison of different methods in ActivityNet1.2

      View table
      View in Article

      Table 7. mAP comparison of different methods in ActivityNet1.2

      MethodmAP@iou/%
      0.50.750.95Average
      R-C3D1026.8---
      TwinNet2536.419.13. 722.8
      AffNet1641.224.45.725.1
      BackTAL1841.527.34.727.0
      MHCS2642.926.36.026.4
      SSN841.327.06.126.6
      RS-STCBD41.627.17.027.2
    • Table 8. Results of ablation experiments at different improvement stages

      View table
      View in Article

      Table 8. Results of ablation experiments at different improvement stages

      Strategy

      3D-

      RSST

      Multilayer

      Convolution

      Non-local Attention Mechanismiou/%
      THUMOS14ActivityNet1.2
      0.40.50.60.70.50.750.95Average
      R-C3D35.628.921.810.926.8---
      Strategy141.134.125.616.838.925.66.624.3
      Strategy239.229.722.112.935.524.24.322.3
      Strategy340.430.122.311.137.124.95.223.4
      RS-STCBD43.536.926.415.641.627.17.027.2
    Tools

    Get Citation

    Copy Citation Text

    Zhong HUANG, Mengyuan TAO, Min HU, Juan LIU, Shengbao ZHAN. Combining residual shrinkage and spatio-temporal context for behavior detection network[J]. Optics and Precision Engineering, 2023, 31(4): 552

    Download Citation

    EndNote(RIS)BibTexPlain Text
    Save article for my favorites
    Paper Information

    Category: Information Sciences

    Received: May. 16, 2022

    Accepted: --

    Published Online: Mar. 7, 2023

    The Author Email: HUANG Zhong (huangzhong3315@163.com)

    DOI:10.37188/OPE.20233104.0552

    Topics