The types of surface diseases on city walls are complex and diverse, with uneven distribution. Spectral data has a low accuracy in identifying diseases with spatial similarity. Aiming at the problem that a single data cannot fully establish the classification and severity characteristics of complex surface diseases on city walls, a multi-data feature fusion method for ancient city wall disease detection is proposed. Independent feature extraction networks are constructed to extract spatial spectral features and texture color features from spectral and true color data sources, and batch normalization layers are introduced after each convolutional layer to accelerate network convergence. By using a feature fusion module to fuse multi-dimensional features of spectral data and color data, advanced feature learning is completed at the fully connected layer. The feature map is mapped to a nonlinear space through a linear rectification function, increasing the model's non-linear expression ability. Construct a joint loss function based on contrastive loss and classification loss to optimize the weights of fused data and improve the discrimination of diseases with similar spatial features. Finally, the Softmax layer is used for pixel by pixel classification to achieve quantitative evaluation and visual analysis of the damage situation of the ancient city wall. The experimental results show that the overall classification accuracy and Kappa coefficient of the proposed method are 96.46% and 94.20%, respectively. Compared with single spectral and true color data, the classification accuracy has been improved by 6.63 percentage points and 12.05 percentage points, respectively. The proposed method is of great significance for the identification of complex disease areas in city walls and the visualization of disease distribution.