This study aimed to effectively monitor the changes in soil properties after Spartina alterniflora invasion on coastal wetland ecosystems. The study area is a typical Spartina alterniflora wetland in the Yancheng Wetland Rare Birds National Nature Reserve of Jiangsu Province. A total of 15 sites were identified by a stratified-random sampling method, and 45 soil samples were collected from three depth intervals (0~30, 30～60, and 60~100 cm). The visible-near infrared spectral reflectance and 10 soil physicochemical properties were measured. The performance of partial least squares regression (PLSR) and random forest (RF) was studied, spectral transformation forms' influence on prediction accuracy was analyzed, and the potential of invasion years and soil depth as auxiliary predictors were discussed. The results show that: (1) the visible-near infrared spectral reflectance can be used to predict organic carbon, inorganic carbon, total nitrogen, water content, pH, bulk density, salinity, and clay contents in soils with reasonable accuracy; (2) the method of partial least squares generally outperform random forest algorithm, the R² of prediction models developed using the PLSR method was between 0.341 and 0.979, and the biggest R² of random forest models was 0.722; (3) Differential transformation and reciprocal transformation of spectral reflectance can substantially improve the model performance. The optimal prediction model of full nitrogen can be obtained based on the original spectra (R² is 0.769 and RMSE is 0.091 g·kg^-1). In contrast, the optimal models for other soil properties are mostly based on differential or reciprocal transformation of the original spectra. (4) In general, the model performance can be improved by adding variables of invasion years and soil depth, and the prediction accuracy of organic carbon, total nitrogen, salinity, pH and bulk density models are more sensitive to the two variables. The prediction model accuracy (R²) for estimating soil organic carbonincreased from 0.794 to 0.806, the accuracy (R²) of the pH model increased from 0.838 to 0.884, and the accuracy (R²) of the salt optimal model increased from 0.978 to 0.997. To sum up, visible-near infrared spectroscopy can be applied to predict key soil physicochemical properties in Spartina alterniflora wetlands, and soil change monitoring of invaded Spartina alterniflora wetlands can be achieved through appropriate spectral transformation, and variable selection and model selection.