To improve the spectral efficiency of the system and solve the problem of single signal mode in traditional orthogonal frequency division multiplexing(OFDM) systems, a layered hybrid visible light communication system based on binary on-off keying(OOK) and adaptive constellation optimization(ACO-OFDM) is designed. The transmitting end of the system uses OOK signals to limit the OFDM signal, generating ACO-OFDM and non adaptive constellation optimization (NACO)-OFDM signals. Add the OOK signal as a direct current (DC) bias to the NACO-OFDM signal to ensure that the output is a unipolar real signal. The receiving end adopts signal reconstruction technology, omitting the noise estimation step and reducing the receiving complexity to about 1/5 of traditional receivers. The simulation results show that the system can effectively demodulate OFDM and OOK signals, and at the maximum transmission layer, its spectral efficiency is about twice that of conventional single-layer systems.