On the basis of characteristics of Passive Millimeter Wave (PMMW) imaging, an Improved Sparse Representation-Circle-Surround Center Difference(ISR-CSCD) algorithm is proposed to improve the weaker distinction between dim small target and background and the smaller target features to be extracted. The algorithm firstly improves the sparse representation to complete the background suppression and target enhancement. Then, according to the features and prior knowledge of the target and the surrounding background, the background suppression algorithm of circle-surround center difference is used to suppress the background of the image. The results by two methods mentioned above are fused to get the final enhanced target image. Finally, the Constant False Alarm Rate (CFAR) is used to complete dim small target extraction. The millimeter wave images in different scenes are detected. The results show that as compared with the mainstream algorithms, Sparse representation (SR), Newton methods for Robust Regularized Kernel Regression(NRRKR), Spatio-temporal Classification Sparse Representation(STCSR) and Accumulated Center-surround Difference Measurement(ACSDM), the ISR-CSCD algorithm has a lower false alarm rate, higher detection accuracy and stronger robustness. For all kinds of false alarm rates and the signal to noise ratios of the millimeter wave small target detection results in statistics, the detection rate of ISR-CSCD is increased by about 15% as compared with other algorithms.