In underwater wireless optical communication systems, the effects of water absorption, scattering, and turbulence make channel estimation and signal detection different, leading to increased communication bit error rates (BER) and even communication failure. To address the difficulties of channel estimation and signal detection in complex underwater channels for optical communication, a machine learning (ML)-based channel estimation and demodulation algorithm is proposed, and its performance in underwater channel estimation and signal detection in direct current biased optical-orthogonal frequency division multiplexing (DCO-OFDM)optical communication systems is studied. Firstly, based on the proposed channel estimation and demodulation algorithm (deep neural network (DNN) and unsupervised learning k-means constellation demodulator), simulation modeling of complex channel frequency response, second-order equalization, and bit error analysis are completed. Secondly, studies on the signal-to-noise ratio (SNR) gains in complex optical communication channels are conducted, comparing traditional least squares (LS), linear minimum mean square error (LMMSE) channel estimation algorithms, and minimum distance demodulation algorithms. In the simulation results, in an underwater channel with a turbulence scintillation index of 0.18 and a distance of 10 m, the proposed channel estimation algorithm provides a signal-to-noise ratio gain larger than 6 dB and 1 dB compared with the LS and LMMSE estimation for 8-order quadrature amplitude modulation (8-QAM) subcarriers at a bit error rate of 10^-5. Additionally, using the proposed signal detection algorithm, an SNR gain larger than 1 dB is achieved compared with traditional algorithms. The simulation results demonstrate that the proposed ML-based channel estimation and demodulation algorithm can impraove the performance of complex underwater optical communication channels. The research results provide a reference for the design of long-distance, high-speed complex underwater optical communication systems.