Most intravascular ultrasound (IVUS) image segmentation methods lack capture of global information, and the topological relationships of segmentation results do not conform to medical prior knowledge, which affects subsequent diagnosis and treatment. To address the above issues, an image segmentation method is proposed that combines convolutional neural networks (CNN) and Transformer dual branch backbone networks with topological forcing networks. A backbone network is constructed by juxtaposing CNN branches and Transformer branches to achieve the fusion of local and global information. The modules that make up the Transformer branch combine axial self attention mechanism and enhanced mixed feedforward neural network to adapt to small datasets. In addition, connecting the topology forcing network after the backbone network and using a bilateral filtering smoothing layer instead of a Gaussian filtering smoothing layer can further improve the segmentation accuracy while ensuring the correctness of the topology structure of the segmentation results. The experimental results show that the Jaccard measure coefficients of the lumen and media obtained by the proposed method are 0.018 and 0.016 higher than those of the baseline network, and the Hausdorff distance coefficients are 0.148 and 0.288 higher, respectively. Moreover, the accuracy of the topological structure is 100%. This method can provide accurate, reliable, and topologically correct segmentation results for IVUS images, and performs well in visualization results and various evaluation indicators.