A new-type off-axis rotary optical fiber transmission system is designed to achieve the connection of the optical signals under the rotary state. The collimating and expanding effect of the fiber collimator with large aperture is analyzed based on the propagation theory of Gaussian beam, and the structure and output beam properties of the fiber collimator is optimized by ZEMAX. The relative optical characteristics of the transmission system are measured. The results show that the fiber collimator outputs Gaussian beam at a working distance of 50 mm. Beam waist radius and far- field divergence degree are 18 mm and 5.48 × 10- 2 mrad, respectively. The maximum connection loss of the transmission system is 23.28 dB when the signal transmission rate is 1.25 Gb/s and the rotation speed is 80 r/min. The overall performance of the transmission system satisfies the requirement of stability for high-speed optical signal transmission between the rotary parts.