With the increasing development of industry and agriculture in China, heavy metal pollution in soil represented by nickel (Ni), iron (Fe), copper (Cu), chromium (Cr), lead (Pb), etc. , has a serious impact on human life. Hyperspectral technology has advantages such as being real-time, non-destructive, and fast, which provides scientific means to obtain information on soil heavy metal content efficiently and accurately. At the same time, the spectral transformation method significantly impacts the inversion accuracy of soil heavy metal content. To clarify the relationship between the spectral transformation method and the inversion accuracy, 60 soil samples were collected in the study area to determine the Ni, Fe, Cr, Cu, and Pb heavy metals content and the corresponding spectral reflectance between 350～2 500 nm. Based on the correlation coefficient (CC) analysis, the feature bands for remote sensing detection of soil heavy metals were selected by the modified discrete binary particle swarm optimization (MDBPSO) method. Finally, the inverse models of Ni, Fe, Cr, Cu and Pb contents were constructed by the random forest (RF) algorithm with the feature bands as independent variables. In this study, based on Gaussian smoothing of the original reflectance, the effects of four differential spectral transformation methods, including first-order differential (R′), first-order differential of logarithmic inverse (1/lgR)′, first-order differential of inverse (1/R)′, and first-order differential of exponential (eR)′, on the accuracy of soil heavy metal inversion were compared and analyzed. The results show that based on the CC analysis method, the MDBPSO algorithm can effectively reduce the redundancy of spectral data and improve the efficiency of the model operation. The number of feature bands sensitive to Ni, Fe, Cr, Cu and Pb in R′, (1/lgR)′, (1/R)′, (eR)′, has been reduced by at least 154, 363, 135, 744 and 889, respectively. (1/lgR)′, R′, R′, (1/R)′, and R′ spectral transformation methods were applied to the combined operation of Ni, Fe, Cr, Cu, and Pb feature bands, respectively. The accuracy of the estimated models was better than other differential transformation methods, where the coefficients of determination of the model test set were 0. 913, 0. 906, 0. 872, 0. 912, and 0. 876. The root mean square errors were 0. 743, 0. 095, 2. 588, 1. 541, and 1. 453, respectively. This study provides a scientific reference for selecting of differential spectral transformation methods when using remote sensing data to retrieve soil heavy metal content. It provides new ideas for further realizing large-area high-precision remote sensing monitoring of soil heavy metal content.