The modulation mode of radio over fiber (ROF) system in application process is analyzed. Experimental results show that single side band modulation is more suitable for the communication process of ROF system. Based on complex coupled mode theory of tilted fiber Bragg grating (TFBG), the spectrum characteristics of the grating are analyzed and tested. After special designing, TFBG can be used in ROF system to reflect or transmit double side band signals to realize single side band modulation. So output amplitude changing with distance in period is reduced and transmission efficiency and receiving sensitivity of the signals are improved.