Free-Space Optical (FSO) communication, as an effective transmission technology with high speed, low latency, large bandwidth, and support for rapid link deployment, has been increasingly valued in the field of wireless communication aimed at big data transmission in recent years. However, the communication performance of FSO signal link is susceptible to weather conditions and atmospheric states (especially atmospheric turbulence), resulting in degradation of signal reception and transmission quality as well as system performance. In order to enhance the reception, transmission, and overall performance of FSO communication systems, researchers have begun to apply various advanced machine learning algorithms to optimize the signal detection and channel modeling processes in FSO communication systems. In this article, the research progress of applying typical machine learning algorithms in FSO communication systems in signal detection, channel estimation, auxiliary optical compensation, and other aspects are reviewed. We compare and analyze the application characteristics of different typical machine learning algorithms, and discuss the future development trends of applying machine learning algorithms in FSO communication systems.