For the infrared real-time dim small target detection and tracking of the image sequences,a parallel processing system based on 4×ADSP TS201S DSP is designed. The loose coupling parallel processing system is constructed by the TS201S link ports. The system possesses Low Voltage Differential Signaling (LVDS) digital video interface, and supports cross interconnecting between DSPs and board level interconnecting. The computational performance of the system can be up to 19.2GMAC@16 bit/s in fixed-point operation and 14.4 GFLOPS in floating-point operation. The infrared dim small target detection and tracking methods have been implemented in the system. The experiment result shows that the parallel processing system has real-time ability, applicability and expansibility. And the real-time infrared dim small target detection and tracking of the large size and high frame frequency image sequences can be realized.