To address the insufficient utilization of features and the significant semantic discrepancy between encoder and decoder features in existing point cloud semantic segmentation networks, a point cloud semantic segmentation method based on cross-layer attention feature fusion is proposed in this paper. Initially, a local feature encoding module is designed to refine neighboring points coordinates through polar coordinate encoding and offset updates, and the differences in relative semantic features are calculated to enhance the richness of neighborhood features, allowing the network to learn the local details of objects of different shapes and sizes. Then, an adaptive feature aggregation module is introduced to enhance the network's perception of local regional features, ensuring comprehensive utilization of neighborhood feature information. Finally, a cross-layer attention fusion network is incorporated to mitigate the semantic discrepancy between encoder and decoder layers. Experimental validations are conducted on the large-scale indoor scene point cloud dataset S3DIS and complex outdoor scene point cloud dataset Semantic3D. On the S3DIS dataset, Area 5 achieved a mean Intersection over Union of 65.2%, and a mean accuracy of 75.1%, showing improvements of 2.8% and 3.7%, respectively, compared to RandLA-Net. For the Semantic3D dataset, the mean Intersection over Union is 76.7%, and the overall accuracy is 94.6%, marking increases of 4.9% and 0.4% over RandLA-Net. The results substantiate the method's capability to extract distinctive features from complex point clouds and achieve precise segmentation across diverse scene categories.