The transimission performance of on-off keying (OOK) and orthogonal frequency division multiplexing (OFDM) signals in radio-over-fiber (ROF) system is experimentally and theoretically studied. The theoretical results show that OFDM signal is almost not affected by inter-symbol interference (ISI),only with the phase changed,where as OOK signal is seriously affected by ISI. The experiment results show that the eye diagram of 2.5 Gb/s OOK signal is closed when transmitting over 60 km in single mode fiber (SMF),and its bit error rate (BER) can only reach 10-3. The constellation of OFDM is clear when ttransmitting over 130 km with BER of 10-4. All theoretical and experimental results show that the performance of OFDM is better than that of OOK in ROF system.