The invariance of modulation transfer function (MTF invariance) is defined to be the evaluating criterion of the wavefront coding system. The rapid optimization of wavefront coding system based on the MTF invariance is proposed by means of introducing the mathematical program to normal optical design process. The interface called dynamic data exchange (DDE) interface, between mathematical program and optical design program is applied to realize the fast data exchanging and merit function calculating. The MTF invariance of the optimized system decreases to 0.0119, compared with 0.0187 of the original system. Results show that the optimization of special optical systems can be executed more conveniently and fleetly with the help of external program and dynamic data exchange.