This paper introduces a hybrid modulation end-to-end communication system based on convolutional neural network (CNN) to optimize the structure and performance of the hybrid quadrature amplitude modulation (QAM) and pulse-position modulation (PPM) modulation system applied to visible light communication. This scheme used the designed loss function to train the network in multiple stages to realize QAM and PPM. Accordingly, the two modulations were combined to realize hybrid modulation. With regard to demodulation, a method for recognizing the pulse of the received signal by changing the kernel size of CNN is proposed to improve the pulse-recognition accuracy and reduce the calculation complexity. The simulation results show that under the additive white Gaussian noise and Rayleigh fading channels, the proposed technical scheme exhibits fine generalization ability for the hybrid modulation method with different pulse time slots and modulation levels. When the symbol error rate is 10^-3, the error performance improvement range is 0.4 dB?2.8 dB compared with the traditional demodulation method.