MSP-YOLACT：Instance Segmentation Model for Multimodal PET/CT Medical Images of Lung Tumors

Fig. 2. Multi-mode feature mixer

Fig. 3. Strengthen feature fusion module

Fig. 4. Multi-scale feature fusion

Fig. 5. Parallel feature enhancement prediction head module

Fig. 6. Global and local feature enhancement module

Fig. 7. Radar map of different module instances segmentation results of MSP-YOLACT model

Fig. 8. Bar graphs of different network instances segmentation results

Fig. 9. The segmentation results

Table 1. Global and local feature enhancement module pseudo code module

Table 1. Global and local feature enhancement module pseudo code module

Input：input feature map $x$

Output：output feature map $y_{s w g c}$

1： $y_{s} = S e l f A t t i o n (x)$ /*performs self-attention on the input feature map */

2： $y_{s q} = C o n v (x)$ /*perform the convolution operation on $y_{s}$ */

3： $y_{s w q} = d e p t h w i s e c o n v (y_{s q})$ /*perform a single-channel grouped convolution */

4： $y_{s q 1} = A d a p A v g p o o l (y_{s q})$ /* $y_{s q}$ performs adaptive averaging pooling*/

5： $y_{s q p 1} = A d a p A v g p o o l (y_{s w q})$ /* $y_{s w q}$ performs adaptive average pooling*/

6： $y_{s q 2} = B N (C o n v_{1 \times 1}^{2} (δ (B N (C o n v_{1 \times 1}^{1} (y_{s q})))))$ /* $y_{s q}$ performs 1×1 convolution and normalization */

7： $y_{s q p 2} = B N (C o n v_{1 \times 1}^{2} (δ (B N (C o n v_{1 \times 1}^{1} (y_{s w q})))))$ /* $y_{s w q}$ performs 1×1 convolution and normalization */

8： $y_{s q c}$ = $y_{s q 1}$ + $y_{s q 2}$ /* $y_{s q 1}$ and $y_{s q 2}$ are added */

9： $y_{s q p c}$ = $y_{s q p 1}$ + $y_{s q p 2}$ /* $y_{s q p 1}$ and $y_{s q p 2}$ are added */

10： $y_{s q p c c} = [y_{s q c}; y_{s q p c}]$ /*concatenate $y_{s q}$ with $y_{s w q}$ */

11： $y_{s w g c}$ = $y_{s q p c c}$ + $x$ /* add $y_{s q p c c}$ to $x$ */

Table 2. Experimental designs for different modal correlations
View table
View in Article
Table 2. Experimental designs for different modal correlations
Experiment PET/CT CT PET Branch
Experiment one √ × × 1
Experiment two √ √ × 2
Experiment three √ × √ 2
Experiment four √ √ √ 3

Table 3. Ablation experiment design of different modules

Table 3. Ablation experiment design of different modules

Ablation experiment	ResNet50	FPN	Head	MFM	SFFM	MFF	DHead	GLFEM
Experiment one	√	√	√	×	×	×	×	×
Experiment two	√	√	√	√	×	×	×	×
Experiment three	√	√	√	√	√	×	×	×
Experiment four	√	√	√	√	√	√	×	×
Experiment five	√	√	×	√	√	√	√	×
Experiment six	√	√	×	√	√	√	√	√

Table 4. Different modal image correlation experiment（IoU=0.50）

Table 4. Different modal image correlation experiment（IoU=0.50）

Model	AP_det/%	AP_seg/%	AR_det/%	AR_seg/%	mAP_det/%	mAP_seg/%
Experiment one	58.31	59.78	49.08	49.16	58.25	59.45
Experiment two	57.63	59.33	48.89	48.95	57.59	59.18
Experiment three	58.39	59.62	48.97	49.08	58.31	59.32
Experiment four	58.57	59.86	49.18	49.22	58.44	59.53

Table 5. Results of ablation experiments（IoU=0.50）

Table 5. Results of ablation experiments（IoU=0.50）

Model	AP_det/%	AP_seg/%	AR_det/%	AR_seg/%	mAP_det/%	mAP_seg/%
Experiment one	58.57	59.86	49.18	49.22	58.44	59.53
Experiment two	59.73	61.83	49.30	49.34	59.59	61.70
Experiment three	60.78	62.36	49.78	50.65	60.67	62.21
Experiment four	61.85	62.68	50.02	51.05	61.12	62.33
Experiment five	62.35	63.62	50.69	51.50	62.29	63.17
Experiment six	64.55	65.53	51.47	52.58	64.37	65.41

Table 6. Comparative experimental results（IoU=0.50）