A three-dimensional (3D) wind field inversion method of a coherent Doppler wind lidar based on genetic algorithm for spectrum estimation is proposed. The method can obtain 3D wind vectors directly from the multi-directional power spectral density without estimating the radial wind velocities, which can improve the data inversion accuracy in the case of low signal-to-noise ratio (SNR). The genetic algorithm adopted is an improved genetic algorithm for the coherent lidar, which can accurately, quickly, and parallelly retrieve the wind vector. Simulation is performed by the proposed genetic algorithm, and results show that the improved genetic algorithm has a significant improvement in convergence speed and global optimization ability compared to the traditional genetic algorithm. In the low SNR signal simulation comparison, the wind field inversion result of this method is better than that of the traditional nonlinear least squares method. The method has been applied to actual lidar systems. In the comparison of the measured data of the lidar and sounding balloons, the root mean square error of the horizonal wind speed is less than 0.7 m/s, and the standard deviation of the horizontal wind direction is less than 6°. The accuracy of the wind field inversion results is verified. By comparing the results of the proposed method with the results of the least squares wind field inversion of the measured data, we find that under the atmosphere conditions at that time, the proposed method increases the detection range by about 12.3%. The comparison between simulation and measured data fully proves the ability and effectiveness of the proposed method to retrieve the 3D wind field.