Advanced modulation technology, channel equalization technology and channel coding technology are the three key technologies to improve the performance of optical transmission system. The optical transmission experiment is carried out by using double binary(DB) modulation technology, feed-forward equalization(FFE) technology and low-density parity check (LDPC) coding technology. After 25 km standard single-mode fiber transmission, the data transmission at 50 Gb/s rate is completed on the 12 GHz optical receiver. The experimental results show that FFE can effectively improve the performance of optical transmission system, and the decoding threshold of LDPC code is around 2×10-2.