Based on the principle of mode coupling, the calculated sidelobe height of output waveform of acousto-optic tunable filter (AOTF) is up to -9.5 dB. The experimental result is about -8 dB. The mechanism of sidelobe production is illustrated. A method is proposed to bend the spiral when we change the width of the three waveguides. The sidelobe can be reduced to -23 dB by this method.