A nonlinear reconstruction method based on nonlinear spectral dictionary learning was proposed to solve the ill-posed problem of spectral image reconstruction from a single RGB image. In order to adapt to the linear and nonlinear data, the method firstly improves the nonlinear principal component analysis algorithm based on a modified self-association neural network model. It uses to learn the low-dimensional spectral dictionary from the training spectrum set, which is used in the inverse equation of spectral reconstruction to alleviate the ill condition. In addition, based on the spectral dictionary, the damped Gaussian Newton method combined with the truncated singular value decomposition regularization method is used further to alleviate the ill-posed problem of the nonlinear inversion, and the spectral image can be reconstructed from a single RGB image. In the experiment, two different spectral training sets, Munsell and Munsell+Pantone, were used to learn the spectral dictionary. Meanwhile, CAVE and UEA spectral image libraries were used for the spectral reconstruction tests. Compared with the existing methods, it is found that the average root means square error of CAVE and UEA spectral images reconstructed by this method under different spectral training sets were the lowest, which were 0.212 4, 0.255 4, 0.229 4 and 0.294 9 respectively. The standard deviations of root mean square error was close to the effect of the best method, which was 0.068 5, 0.084 7, 0.066 8 and 0.087 0 respectively. The results show that the method for reconstructing the spectral image from a single RGB image has advantages in accuracy and stability.