According to the technical requirements of the large field of view and wide spectral band in polarization spectral imaging detection, a wide spectral band and large field of view polarization imaging spectrometer based on Polarimetric-spectral intensity modulation (PSIM) was designed. For the front telescope group, according to the achromatic analysis of existing domestic glass materials, the achromatic glass from visible to short-wave infrared is selected. By controlling the light angle of the PSIM module in the mirror group, the incident angle demand on the PSIM module in the large field of view is realized. Based on the results of the analysis, optical design software is used to optimize the design. The design results show that the front telescopic system can achieve high-quality imaging with a wavelength range of 400～1 700 nm, a field angle of 72°, a focal length of 20 mm, and an F-number of 4. The transfer function of the detector at the cut-off frequency in the full spectrum is better than 0.4, and the maximum incidence angle on the PSIM module is ±4.99°, effectively ensuring the consistency of polarization modulation in each field of view. The post-spectral spectroscopic system uses a convex grating based on the Offner structure. The optimization results show that the point array of each band is less than one pixel and the MTF of the central wavelength at the Nyquist frequency of the detector reaches 0.6, and all indicators meet the design requirements. This paper has important practical significance for the engineering of polarization spectral imaging instruments based on PSIM wide spectrum and also has certain guiding significance for the achromatic design of wide-spectrum optical systems.