To address the issue of low recognition accuracy in scene images caused by subtle inter-class differences and ambiguous intra-class classifications, this paper proposes a novel semantic segmentation framework. By introducing deep metric learning and focusing on the semantic relationships between pixels, the model’?s recognition accuracy can be improved. Firstly, feature extraction is performed through the hollow space pyramid pooling module. Then, in the decoding process, the shallow high-resolution features and deep low resolution features are fused using a structure to better restore the details and boundaries in the image. Secondly, in the deep metric learning module, a well structured pixel semantic embedding space is learned to effectively classify pixels by maximizing the Euclidean distance between pixels of different categories and minimizing the Euclidean distance between pixels of the same category. Finally, a fusion loss function combining weighted focus loss and contrast loss is adopted to balance the importance between different samples, thereby more accurately measuring the performance of the model and improving the accuracy and robustness of scene recognition. The experimental results demonstrate that the average intersection to union ratios of the model on the publicly available datasets ADE20K and Cityscapes are 47.6% and 83.1%, respectively. Compared with the baseline of today?’?s advanced scene recognition methods, the results show that the proposed method is feasible and progressiveness.