The performance of an optical system with sensitive line-of-sight (LOS) is influenced by rotational vibration. In view of this, a design methodology is proposed for a passive vibration isolation system in an optical system with sensitive LOS. Rotational vibration is attributed to two sources: transmitted from the mounting base and generated by modal coupling. Therefore, the elimination of the rotational vibration caused by coupling becomes an important part of the design of the isolation system. Additionally, the decoupling conditions of the system can be obtained. When the system is totally decoupled, the vibration on each degree of freedom (DOF) can be analyzed independently. Therefore, the stiffness and damping coefficient on each DOF could be obtained by limiting the vibration transmissibility, in accordance to actual requirements. The design of a vibration isolation system must be restricted by the size and shape of the payload and the installation space, and the layout constrains are thus also discussed.