The baffle size and Lagrange invariant of star sensor are two key factors which affect its volume and lens complexity. Through derivation and analysis of the relationship between limiting star magnitude and these two factors, miniaturization of the optical system of star sensor is studied. According to the relations between its optical system parameters and technological requirements, formulae about baffle size and Lagrange invariant under various limiting magnitudes are derived and analyzed. It is shown that with the increase of limiting magnitude, the Lagrange invariant monotonically decreases, but the baffle size has minimum value. On these grounds, limiting magnitude is optimized to get as small baffle size and Lagrange invariant as possible and to facilitate its miniaturization. As an example, the optical system of a star sensor based on STAR1000 complementary metal-oxide-semiconductor transistor (CMOS) detector and suitable for microsatellite is optimally designed, with limiting magnitude, field of view, entrance pupil diameter, optical overall length, and weight of 5.0 MV, 20°×20°, 28.2 mm, 124 mm, and about 300 g respectively.