Aiming at the problem of trapezoidal distortion of the displayed picture due to the fact that the camera is not perpendicular to the monitoring picture in the monitoring system, a hardware structure design of an adaptive real-time video keystone correction system is proposed. Firstly, a series of preprocessing such as gray scale transformation, edge detection, and morphological opening operation are carried out in sequence for each frame. Then, based on the Hough transform, the outline of the trapezoid distortion in the picture is extracted. After that, four pairs of mapping point coordinates are determined according to the trapezoidal distortion contour and the connection point method is used to calculate the correction parameters. Finally, the gray-scale interpolation method is used to obtain the corrected image. The whole process uses pipeline and ping-pong buffer structure to optimize the algorithm, and uses the vertical blanking time in the video sequence to calculate, which greatly reduces the hardware resource consumption and improves the algorithm efficiency. It is implemented on the ALINX AX7103 FPGA hardware development platform. The experimental results show that when the video resolution is 640×480 and the refresh rate is 60 Hz, the system can correct the monitoring picture with the trapezoidal distortion range within ±30°, and the correction accuracy is 1°. Basically, the system can meet the self-adaptability and real-time requirements of the monitoring system.