In coherent optical communication system, the polarization state of the signal light is uncertain because of the influence of atmospheric turbulence, and the optical mixer is sensitive to the polarization of signal light and local oscillator light. Based on the principle of optical mixer, the relationship among the polarization state of the signal light, the intermediate frequency signal and the mixing efficiency is deduced, The amplitude of the intermediate frequency signal is used as feedback control signal to increase the mixing efficiency, and a single particle optimization algorithm for coherent optical communication systems is designed. Experimental results show that in closed loop state, the amplitude of the intermediate frequency signal increases rapidly, the mixing efficiency increases about 64%, and the fluctuation variance of amplitude of intermediate frequency signal decreases to 0.001. The polarization control in the coherent optical communication system is realized.