To address the problems of incomplete feature extraction of architectural elements and difficulties in the recognition of similar architectural styles, we propose a salient region suppression and multi-scale feature fusion (SRSMSFF) architectural style recognition method. First, the improved Resnet18 is used to extract the initial architectural features. Next, the salient region suppression module (SRSM) is designed, which guides the network to learn the features of potential regions by hiding the most discriminative regions. And multi-scale feature fusion (MSFF) is designed, which combines multi-scale structure with salient region suppression to obtain a more complete feature of architectural elements. Then, channel attention is used to assign corresponding weights to each channel, which can highlight important channel information. Finally, the large-margin Softmax loss function (L-Softmax) is introduced through maximizing the decision boundary distance of the feature embedding space, which improves the performance of similar architectural style recognition. The experimental results show that our model achieves 64.44% and 80.21% accuracy on the 25-class and 10-class public architectural style datasets. It achieves an accuracy of 88.21% on the dataset of ancient Chinese architectural styles. Its performance is superior to current advanced method.