The Infrared Hyperspectral Atmospheric Sounder II (HIRAS-II) mounted on Funyun-3E satellite (FY-3E) can realize vertical atmospheric detection, featuring hyper spectral, high sensitivity and high precision. When the satellite has been on-orbit, nonlinear response would occur due to instrument attenuation and environmental change, which affects the accuracy of on-orbit calibration. For nonlinear response problems, a nonlinear correction method based on in-band spectrum is proposed. Firstly, the nonlinear correction coefficient was solved based on the nonlinear characteristics of the out-of-band in low-frequency spectra, and the coefficient was input into the radiological calibration model as the initial value. The deviation between the black-body spectrum measured on board and the ideal spectrum was taken as the objective function, and the nonlinear correction coefficient was optimized by iterative calculation. Radiation calibration experiments have shown that the brightness temperature deviation of black body is reduced compared to uncorrected result and the out-of-band correction method using the proposed method.Based on the simultaneous nadir overpass method, After matching the observed data of HIRAS-II with IASI and calculating the mean brightness temperature deviation and the absolute value of the deviation, the mean brightness temperature deviation after nonlinear correction by the in-band correction method is -0.13K, which is better than the out-of-band correction method.