The characteristics of an acousto optic tunable filter and its optical system are investigated. The effective solutions to improve the purity of the diffracted light in this optical system are put forward. The orthogonal polarizing method, and the combined modulation method of optics and electronics are discussed. The measurement results of several optical improvement methods are compared with that of FT IR through the near infrared spectral measurement of water. The results show that both methods can effectively reject the inference of zero order light. But the orthogonal polarizing method does at the expense of the utilization ratio of light. The combined modulation method can increase spectral resolution and signal to noise ratio of this optical system, while not decrease the utilization ratio of light.