A novel 40-Gb/s constant envelope optical frequency shift keying (FSK) transmitter and the transmission characteristics are investigated both by simulation and experiment. Meanwhile, to increase the spectrum efficiency of FSK, we propose a novel optical minimum-shift keying (MSK) scheme and analyze its performance compared with other MSK schemes and other traditional modulation formats. Simulation and experimental results show that the novel FSK scheme could be a potential candidate for the future high speed transmission and label switching systems. And the novel MSK scheme deserves future deep research for its potential excellent performance.