Aiming at the parks and other similar environments application scenarios where the scene feature is unique and the global navigation satellite system (GNSS) signal is unstable, as well as the problem that the lack of semantic understanding in Lidar simultaneous localization and mapping (SLAM) results in a localization error of unmanned vehicle, a building method of three-dimensional semantic map with the data fusion of monocular camera and Lidar is proposed. This method is based on the characteristics of strong structural park environment and high dynamic variation of pedestrian in vehicles. The improved panoramic feature pyramid network (PFPN) is used for scene visual semantic segmentation, and then the pixel level fusion method is used to provide semantic information for the laser point cloud, so as to effectively remove the interference of dynamic targets in the process of normal distribution transformation (NDT) mapping, and then improve the robustness and accuracy of unmanned vehicle SLAM technology in the dynamic environment. Experimental validation is carried out on the cyclone intelligent self-driving platform and compared with the original NDT method, the experimental results show that the proposed method is able to improve the construction accuracy comprehensively, with the most significant improvement of 34.34% in positional accuracy; besides that, the number of point clouds for building the diagram is also reduced by 39.78%, which greatly improves the construction speed.