Novel optical offset quadrature phase shift keying (OQPSK) and improved minimum-shift keying (MSK) modulation schemes to smooth optical phase hopping for high speed fiber optics transmission are proposed. Simulations have been done among the MSK, OQPSK, quadrature phase shift keying (QPSK), and binary phase-shift keying (BPSK) modulation formats at 40 Gb/s over 100-km transmission link using coherent detection. Simulation results present good performances to elevate chromatic dispersion tolerance and reduce the influence of nonlinear effects by adopting the smooth phase modulation formats. Inter-symbol interference decreases for the fast side-lobe property and tight spectrum as a result of avoding the π phase shift and reducing envelop fluctuation.