All-optical phase modulation is an important part of all-optical signal processing, which is of great significance to the realization of all-optical networks. Semiconductor optical amplifier (SOA) is a key component of all optical phase modulation system because of its high nonlinear coefficient and small size. However, due to the concurrence of cross phase modulation and cross gain modulation in SOA, the power fluctuation of current SOA-based phase modulation signals is inevitable, which affects the signal quality. Therefore, an all-optical phase modulation scheme of cascading SOA controlled by differential balance signal is proposed, and the principle of the scheme is analyzed. Then, the all-optical phase modulation is realized through experiments, and the waveform fluctuation caused by SOA cross gain modulation is eliminated during the phase modulation process, and the principle is verified. This modulation scheme is of great significance to realize all-optical wavelength conversion for high-order coding formats and improve the flexibility and expansibility of the network.