In order to solve the problem of fully automatic and high-precision alignment of monochrome and color videos, the first frame and subsequent frame alignment methods for video alignment are designed to improve the alignment accuracy while ensuring the speed of alignment. For the first frame, firstly, the scale invariant feature transform (SIFT) is improved based on the characteristics of dual cameras, and then the coarse alignment is performed for monochrome and color videos, which increases the stability and accuracy of the coarse alignment matrix. Subsequently, the alignment matrices of monochrome and color videos are modified according to the coarse alignment matrix coefficients using the scale pyramid model. For the subsequent frames in the video, a translation correction method with a step pyramid is used to reuse the corrected alignment matrix obtained in the first frame, and the translation correction method is used to compensate for the translation errors caused by uncertain time difference fluctuations and camera oscillations, which finally improves the alignment efficiency and alignment accuracy of monochrome and color videos. The experimental results show that for monochrome and color videos of 640×480 size, the algorithm in this paper reduces the root mean square error of the first frame alignment by 0.79% and the marker point error of the first frame from about 1 pixel to undetectable compared with the traditional SIFT algorithm, and the average alignment time of consecutive video frames is only 0.357 s, while still keeping the marker point error undetectable. The algorithm in this paper better meets the requirements of fully automatic, high accuracy, fast and robustness for monochrome and color video alignment.