The angular velocity input/output characteristics of a spin-exchange relaxation-free(SERF)atomic spin gyroscope under two different pumping conditions were experimentally tested via an optical fiber Sagnac atomic spin precession closed-loop detection technique based on circularly polarized probe lights. The nonlinearity of the SERF atomic spin gyroscope output was discovered. Based on the basic principle of the SERF atomic spin gyroscope, the nonlinear response model was established and the simulation study was carried out, and the simulation results were in agreement with the experimental results. This indicates that the nonlinearity of the SERF atomic spin gyroscope is determined by the internal interaction of atoms and is closely related to the total electron relaxation rate Rtot.