In order to suppress the nonlinear effect of the optical transmission system, an improved 32-channel quadrature pulse shaping dual-phase conjugate twin wave (QPS dual-PCTW) nonlinear suppression scheme combining high-order modulation format and nonlinear separation technology is proposed. With the joint simulation of Optisystem and Matlab, a 32-channel nonlinear separation model with or without QPS dual-PCTW scheme under 16-order quadrature amplitude modulation(16QAM) and 64QAM is built, and the system's bit error rate is analyzed through simulation under different conditions to study the suppression of nonlinear effects gained by this scheme. The simulation results show that whether the QPS dual-PCTW scheme is used or not, the four wave mixing(FWM) effect has a greater impact on the system performance, when the QPS dual-PCTW scheme is used, the main factor affecting the system performance is inter symbol interference.