To reduce the burden of the working platform, optimize the optical structure and promote the thermal adaption of the imaging spectrometer, an optical design of a compact infrared imaging spectrometer was presented. Due to compactness and aberration correction, the extended polynomial surfaces were utilized in the optical design of the infrared imaging spectrometer with spectral region between 1 μm to 3.4 μm, optical speed of F/2.86, and spectral sampling of 7.5 nm. The optical system was comprised of a free form three mirror telescope and a plane grating based free form spectrometer. The telescope was semi telecentric for pupil matching with the telecentric spectrometer, free of distortion and able to provide enough imaging space for other modules. The good imaging quality of spectrometer, and spectral distortion correction were achieved, and the tilt of image plane for detector arrangement was improved. The system is analyzed from the diffraction efficiency of the grating, the supressed stray light, as well as the opto-mechanical design of the integrated mirrors, which show it has high engineering feasibility.