Aiming at the problems that the main structure of a mini-optical device for spatial use is too heavy, its ground gravity deformation is severe and the base frequency is too low, an optimization mathematical model was established with the minimum mass and the RMS of random acceleration response as the objectives, and the fundamental frequency and the deformation as the constraint conditions. The topology optimization design of the main structure was carried out. An engineering analysis was made to the optimized main structure. The results showed that the optimized main structure of the mini-optical device has lighter mass, higher fundamental frequency, less deformation and random vibration response.Mechanics experiments were carried out to examine the performance of the main structure. The detection results accord with the overall index, which proved that the optimization method is effective and feasible.