In order to solve the optical power limitation problem of the transmitter of the radio-over-fiber (RoF) communication system and to improve the reliability of information transmission, a LDPC-PS-64QAM scheme is proposed by combining low-density parity-check (LDPC) codes and probabilistic shaping (PS) techniques to optimise the transmission characteristics of RoF systems. The scheme uses the sign reversal method to probabilistically shape the 64-quadrature amplitude modulation (QAM) signal, while the LDPC code specified by 3GPP is used as the channel coding scheme. The simulation results show that at a symbol rate of 10 Gbaud, the bit error rate (BER) of uniform distribution (UD) and PS signals are compared, and the transmitting power of PS signal is reduced by 1.4 dB compared with UD signal, and the transmitting power of the transmitter is further reduced by using LDPC code; at the fiber length of 20 km, the optical signal-to-noise ratio (OSNR) of PS signal is reduced by 2.1 dB compared with that of UD signal at the same BER, and it has a better noise resistance. The result shows that the LDPC-PS-64QAM system has higher reliability by changing the transmission length of the fiber.