The accurate classification of architectural style is of great significance to the study of architectural culture and human history and civilization. Models based on convolutional neural network (CNN) has achieved good performance in the field of architectural style classification due to its powerful feature extraction ability. However, most current CNN models only extract the local features of architecture buildings. With the attention mechanism, a model based on Transformer can extract the globle features of architecture buildings. In order to improve the accuracy of architectural style classification, an architectural style classification method fusing CNN and Transformer is proposed. The core of the network is CT-Block structure. In terms of channel dimension, the structure is divided into two branches, CNN and Transformer, and the features pass through the two channels respectively and then concatenate together. This structure then concatenate together. This structure can not only fuse the local features extracted by CNN and the global features extracted by Transformer, but also alleviate the problem of model size and parameter number increase caused by the two-branch structure. The experimental results of Architectural Style Dataset and WikiChurches dataset were 79.83% and 68.41% respectively, which was better than other models in the field of architectural style classification.