Stone cultural heritage is a precious carrier of human history and culture. However, weathering problems severely threaten the long-term preservation of stone cultural heritage sites. Determining the main weathering environment is crucial for developing the corresponding protection measures. However, determining the weathering environment in a non-destructive and contract-free manner is challenging. Therefore, this study proposes a weathering-environment evaluation method based on hyperspectral imaging technology. First, the visible-near-infrared (VNIR) and short-wave infrared (SWIR) spectral data of weathered sandstones in different environments were obtained using hyperspectral imaging technology. The spectra were preprocessed using standard normal variation (SNV) and multiple scattering calibration (MSC), the spectral features were downscaled via principal component analysis, and multiple machine-learning and deep-learning algorithms were used to establish classification models for weathering environments of stone cultural heritage. The results show that the classification model established based on SNV preprocessed spectral data has a higher overall accuracy rate. The deep-learning model outperformed the conventional machine-learning model, with a maximum accuracy rate of 0.98 attained. The proposed method enables a rapid, contact-free assessment of the weathering environments of stone cultural heritage sites and provide important support for targeted cultural-heritage protection.