For the phase retrieval in fringe projection 3D shape measurements, a new fringe projection phase retrieval method based on variational mode decomposition was proposed. Firstly, the projection fringe pattern was decomposed into a background part, a fringe part and a noise part by the development of variational mode decomposition model and the minimization of the model. Then, the fringe part was processed by Hilbert and arc tangent transform to obtain a wrapping phase, and by quality guided phase unwrapping and Zernike carrier removal to acquire the final absolute phase. Simulation and experimental results show that the phase error by the proposed method is 3.14×10-4, smaller than the errors 3.30×10-4 and 6.52×10-4 that respectively obtained by Fourier transform method and continuous wavelet transform method. The proposed method is superior to the Fourier transform method and continuous wavelet transform method in the process of projection fringes with edge information, providing more accurate results, and is more effective for the application of the three dimensional measurement of objects with discontinuous and abrupt changes.