A circular quadrature amplitude modulation based on probability-geometric hybrid shaping (PS-GS-Cir-MQAM) vector terahertz (THz) signal transmission scheme is proposed to improve the transmission performance of a radio-over-fiber (ROF) transmission system. The transmission performance of the 130 GHz band PS-GS-Cir-MQAM signal in an RoF system with an intensity modulator (IM) and in-phase/quadrature (I/Q) modulator cascaded is analyzed via co-simulation in Matlab and VPI, and the signal transmission performance is compared and analyzed. The improvement effect of different shaping methods on the error code performance is evaluated. The results show that the error performance of the proposed PS-GS-Cir-MQAM THz signal is better than those of the other three shaping formats. Among them, the PS-GS-Cir-32QAM THz signal shows the most significant improvement in error performance. Compared with the other three shaping formats, the signal power gain increases by 0.88, 1.4, and 2.48 dB,respectively. The proposed hybrid shaping-based vector THz signal is advantageous in terms of transmission; specifically, it overcomes the nonlinear effects of optical devices and optical fibers.