This paper proposes an optimal design method of filter window for the holographic three-dimensional (3D) display optical system. First, based on scalar diffraction theory, the complex amplitude coding method is used to obtain the kinoform, and the spectrum distribution of the reconstructed light field of the kinoform on the focal plane of the optical system is studied. Then, the spectrum center position and frequency distribution formula of the reconstructed object light field on the focal plane of the optical system are derived and verified by experiments. Finally, an optimized design method for the filter window of the holographic display system based on the phase-only spatial light modulator is proposed. The results of theoretical simulation and optical experiment show that the optimization method can obtain kinoform with optimal encoding, and can design a high quality holographic 3D display system to obtain high quality holographic reproduced images.