With the rapid development of detection technologies, the infrared detectors have gained more and more applications. The internal structure of a miniature infrared detector for fast cooling-down process was numerically simulated and analyzed in this paper. The simulating result shows that the thermal contact resistance has significant effect on the detector module cooling-down process. By adjusting the contact resistance, the cooling-down time of the detector chip from ambient temperature to 90 K are 4-10 s. The effects of the initial ambient temperature and the diaphragm surface emissivity show few influence on the cooling-down process of the chip. While the temperature of the throttling cool head has a great impact on the cooling-down process. The faster the temperature drops, the shorter the startup time of the chip. Therefore, the detector can be improved by optimizing the thermal contact resistance and the throttling cool head, to achieve the higher requirements.