Aiming at the problems of large parameters and low semantic segmentation accuracy of real-time semantic segmentation networks for true-color microvascular decompression (MVD) images. This paper proposes a U-shaped lightweight fast semantic segmentation network U-MVDNet (U-Shaped Microvascular Decompression Network) for MVD scenarios, which consists of encoder-decoder structure. A Light Asymmetric Bottleneck Module (LABM) is designed in the encoder to encode context features. Feature Fusion Module (FFM) is introduced in the decoder to effectively combine high-level semantic features and underlying spatial details. Experimental results show that for the MVD test set, U-MVDNet achieves 0.66 M parameters, 76.29% mIoU (mean Intersection-over-Union), and 140 frame/s speed on NVIDIA GTX 2080Ti. And when input image size is 640 × 480, the real-time (24 frame/s) semantic segmentation is realized on NVIDIA Jetson AGX Xavier embedded development board. The proposed network has no pretrained model, fewer parameters, and fast inference speed. The semantic segmentation performance is superior to other comparison methods, and a good trade-off between segmentation accuracy and speed is achieved. Furthermore, U-MVDNet can also be easily developed and applied on embedded platform with superior performance and easy deployment.