Infrared system calibration is an important task in measuring the infrared characteristics of targets, and its accuracy will directly affect the accuracy of the final measurement results. In order to reduce the influence of temperature drift on calibration accuracy, a field calibration method for refrigerated infrared systems considering ambient temperature changes is proposed, and the relation formula between pixel gray response and target radiance considering ambient temperature is derived. The feasibility and accuracy of this method are verified through field calibration experiments on a domestic cooled infrared system. The results show that compared with the traditional calibration method, the proposed method reduces the mean error of the inversion of the temperature points involved in the fitting by 0.57%, and the root mean square error by 0.03. For extended temperature points, the mean error of the inversion is reduced by 1.63%, and the root mean square error is reduced by 0.93. This method offers higher calibration accuracy across the entire frame and significantly improves the accuracy of target characteristic inversion, providing a useful reference for the practical engineering application of infrared system calibration.