This paper proposes a wavelet-spatial high-order feature aggregation network (WHFA-Net) which can be divided into two branches: wavelet domain feature extraction and spatial domain feature extraction. Firstly, a Harr wavelet transform is embedded into convolutional neural networks (CNNs), and low-frequency components of the depth-wise convolutional features are retained as wavelet depth features. Secondly, depth feature learning is performed by the max pooling, and then the output is used as spatial depth features. In addition, the wavelet domain and spatial domain depth features are vectored, and their auto-correlation and cross-correlation high-order depth feature vectors are obtained. Feature regularization, feature aggregation, and feature normalization are then performed in sequence. Finally, a cross-entropy loss function is utilized for end-to-end network training. The experimental results on NWPU45 (NWPU-RESISC45 Dataset) and AID (Aerial Image Dataset) show that compared with that of the benchmark network (VGG-16), the accuracy of the proposed WHFA-Net in scene classification is improved by 5.13%-12.12%. Furthermore, compared with DCCNN, APDC-Net, GBNet, LCNN-BFF, MSCP, and Wavelet CNN, the accuracy of WHFA-Net in scene classification is higher. Additionally, the effectiveness and performance differences of each module and branch are verified through the ablation experiments. Therefore, WHFA-Net can effectively and stably extract the high-order aggregated features of different feature domains in remote sensing scene images and accurately perform scene classification.