The Lukosz pre-correction modal algorithm can correct low-order aberrations of wavefront distortion and narrow the search range of iterative algorithms. The adaptive cosine-decay stochastic parallel gradient descent (AcSPGD) algorithm can compensate for high-order aberrations of wavefront distortion and improve the correction accuracy of iterative algorithms. In this paper, a new hybrid modal algorithm based on the pre-correction model and AcSPGD algorithm is applied to correct wavefront distortion in wavefront sensorless adaptive optics, and the feasibility of the optimization algorithm is also verified by the experiments. Experimental results show that the correction speed of the hybrid modal algorithm is two times faster than the commonly used stochastic parallel gradient descent (SPGD) algorithm, and the correction accuracy of the hybrid modal algorithm is better than the traditional Lukosz modal algorithm. Applied to wavefront sensorless adaptive optics, the optimization algorithm effectively reduces the phase fluctuation of the wavefront and improves the far-field Strehl ratio, which thus improves the signal-to-noise ratio of the atmospheric laser communication system by 2.9 dB, reduces the bit error rate to 10^?6, and improves the communication performance of the free-space optical communication system. The hybrid modal algorithm has great reference and application value.The Lukosz pre-correction modal algorithm can correct low-order aberrations of wavefront distortion and narrow the search range of iterative algorithms. The adaptive cosine-decay stochastic parallel gradient descent (AcSPGD) algorithm can compensate for high-order aberrations of wavefront distortion and improve the correction accuracy of iterative algorithms. In this paper, a new hybrid modal algorithm based on the pre-correction model and AcSPGD algorithm is applied to correct wavefront distortion in wavefront sensorless adaptive optics, and the feasibility of the optimization algorithm is also verified by the experiments. Experimental results show that the correction speed of the hybrid modal algorithm is two times faster than the commonly used stochastic parallel gradient descent (SPGD) algorithm, and the correction accuracy of the hybrid modal algorithm is better than the traditional Lukosz modal algorithm. Applied to wavefront sensorless adaptive optics, the optimization algorithm effectively reduces the phase fluctuation of the wavefront and improves the far-field Strehl ratio (SR), thus improving the communication performance of the free-space optical communication (FSO) system.