Due to the high peak to average power ratio in coherent optical orthogonal frequency division multiplexing system, fractional Fourier transform combined with peak to average power ratio reduction algorithm was proposed. With the decrease of fractional Fourier transform order, the peak to average power ratio of signal is reduced. Fractional Fourier transform combined with the reduction algorithm can reduce the peak to average power ratio further, and improve the degradation of transmission performance caused by the reduction algorithm. The simulation analyses have three aspects including compatibility of the combination, inhibition effect of peak to average power ratio and improvement of the negative effects in the reduction algorithm. The results show that the probability of the selective mapping algorithm combined with fractional Fourier transform is 104 less than that of the single selective mapping algorithm when the rotation factor is 4 and the peak to average power ratio is higher than 6.6 dB. When the same degree of inhibition of peak to average power ratio is chosen, the time complexity of the combined algorithm decrease by 8 times than single algorithm. The simulation results using clipping algorithm combined with fractional Fourier transform have great improvement than single algorithm, which shows combined algorithm has good performance of compatibility.