In order to improve the transmission capacity of the radio over fiber (RoF) system, a bidirectional RoF transmission system is proposed based on the generation of frequency-quadrupled vector signal and wavelength reuse technique. For the downlink transmission, a stimulated Brillouin scattering-assisted narrow-band optical notch filter and a Sagnac loop are employed to generate a frequency-quadrupled vector signal directly in optical domain. At the base station, the unmodulated sideband of the downlink signal is filtered out by a polarizer and reused as the uplink optical carrier to achieve wavelength reuse. In the experiment, a quadrature phase shift keying (QPSK) signal centered at 24 GHz is generated by frequency beating. A 400 Mbit/s downlink QPSK signal at 8 GHz and a 400 Mbit/s baseband on-off keying (OOK) signal as uplink are used to evaluate the system performance over a 6.15 km fiber link. The feasibility of the proposed system is confirmed by the experimental result.