A signal processing technique based on all-phase spectral analysis was proposed to reduce the displacement measurement error of a self-mixing interferometer. Self-mixing interference principle and signal processing technique were analyzed. First, the mathematical model and signal characteristics for the self-mixing interferometer were presented based on the model of laser oscillation cavity with three mirrors. Then, the displacement principle based on self-mixing interference was introduced. A all-phase spectral analysis was used to extract the phase of self-mixing interference signal to reconstruct the displacement of a reflective target and the signal processing algorithm was investigated with arithmetic simulation. Finally,an experiment system was established and the calibration result of bimorph PZT was presented. Experimental results indicate that the displacement error can be decreased to 4.4 nm by using all-phase spectral analysis. It concludes that the all-phase spectral analysis can reduce the displacement error of the self-mixing interferometer to nanometer level without increasing the number of optical elements.