Aiming at the need of video surveillance and other outdoor video equipment for real-time restoration of video image under severe weather conditions such as fog and fog, an field-programmable gate array(FPGA) implementation method for fast image restoration algorithm was proposed. By appropriately reducing the complexity of the algorithm, the purpose of reducing time cost and hardware implementation complexity could be achieved.In this algorithm, a frame of video is decomposed into 3×3 small windows for operation, and a frame of video pixel calculation is replaced by a pipeline calculation of the center pixel of the window, so that each pixel’s all operations are processed in 3 clock cycles, ensuring that the system can be restored in real time, and the hardware cost of the recovery module can be reduced.Experimental results show that the results of FPGA processing coincide with the results of MATLAB processing, the real-time processing resolution is 640×480 , and the video can be displayed smoothly at 29 frames /s under the premise of ensuring the defogging effect.