In order to achieve the stress defect detection of infrared materials such as Si, Ge, and GaAs, two Photoelastic modulators working at different frequencies are cascaded to form a polarimetry system. The birefringence retardation and fast axis azimuth introduced by the stress defect are loaded into the modulation signals of the polarimetry system. The amplitudes of the fundamental and differential frequency terms were simultaneously obtained by using digital phase-locked technology, and then the two stress parameters are solved. A detailed analysis of the detection principle was conducted and an experimental system was established for verification. The experimental results show that this method has achieved stress defect detection with a standard deviation of 0.31 ° for stress direction angle and 0.72 nm for stress birefringence retardation. The high-speed, high-precision, and high repeatability stress defect detection are realized, and the measurement of stress defect distribution in a Ge samples are demonstrated. An effective method for infrared material quality testing, analysis, and evaluation is proposed.