A reference-based pre-equalization (RPE) technology is introduced. The technology, which is based on the self-coherent mixing frequency technology, can measure the accurate qualitative information of transfer function in self-coherent homodyne detection system and pre-emphasize the transmission signal. The RPE technology can pre-compensate all electrical and optical linear loss, including the bandwidth limitation of modulator and power amplifier and the high-frequency component loss caused by optical pre-filter. The calibration performance of RPE strongly depends on the laser linewidth of both the transmitter and the receiver. We investigate the RPE calibration performances of several laser sources which are universal in self-coherent homodyne detection system, and develop a phase correction algorithm to obtain accurate qualitative information of the transfer function. The feasibility of the algorithm is verified when we use several types of laser sources in coherent optical communication system with polarization-multiplexed quadrature phase shift keying. The wavelength division multiplexing bandwidth limitation is 25 GHz and the rate is 32 Gbit/s.