In the task of lithology classification, the feature information obtained from a single data source is limited. Hence, multisource data fusion is an important means by which to improve the accuracy of lithology classification. As typical remote sensing data sources, aerial remote sensing images and digital elevation models can provide complementary spectral and elevation information. In order to improve the accuracy of lithology classification, a new lithology classification method for multisource remote sensing data is proposed. The proposed method combines the spatial attention mechanisms of channel and multiscale convolutional neural networks. Additionally, this method enhances the learning ability of convolutional neural networks on deep features of aerial remote sensing images and digital elevation models by designing a multiscale void convolutional module to better capture the spatial relationships of features and effectively eliminate the structural differences of heterogeneous data in the original data space. By designing local and global multiscale channel spatial attention modules, different weights can be assigned to spectral channels and spatial regions of multisource data in an adaptive way to both realize more targeted training of the network by using the significance of features and further improve the classification performance of the model. Finally, a basin in Sichuan province is taken as the study area to validate the proposed techniques. The experimental results show that the proposed method is significantly better than four typical machine learning methods in the overall accuracy and average accuracy, which proves that the proposed multisource data fusion method can make full use of the complementary advantages of different data sources and effectively improve the discrimination accuracy of geological lithology.