A novel vibration isolation system based on Stewart platform was designed to attenuate the vibration of space optical payloads, and the general configuration characteristics of the Stewart platform were analyzed. Firstly, the theoretical model of the vibration isolation system was developed through Newton-Euler dynamic approach.On the basis of the model, the closed form formulation was derived to calculate the stiffness matrix, damping matrix, natural frequencies and principal mode shapes of the vibration isolation system. According to the research above, an optimization method was used to optimize the structure parameters and to narrow the distribution range of the first six natural frequencies of the isolation system, then to obtain an optimal configuration of this vibration isolation system. Finally, the finite element method and the analytic method were utilized to analyze the modes of this vibration isolation system and to obtain the first six natural frequencies of the system. The analysis indicates that the maximum error is 1.51% between the two methods. In order to demonstrate the isolation effect of the vibration isolation system, the complex frequency response was analyzed, and the translation frequency response curves and the rotation frequency response curves were obtained. It is shown that the vibration attenuation rate is more than 90% when the vibration frequency is higher than 10 Hz,which verifies that the vibration isolation system meets all the isolation requirements.