The patrol train with picture pick-up device has been used for railway environment monitoring. A method to generate the stereo panorama from forward motion train-borne video using only one camera to acquire the panoramic stereo information of the entire scene is presented. On the basis of the geometric priori information of the scene and known camera parameters, a set of trapezoidal stripes is extracted from a video sequence continuously, whose width is automatically adjusted according to the vehicle speed, and the stripes are stitched into a panoramic image after aligning all the stripes by homography transformation. The accurate geometric model is built to generate the panoramic stereo pairs. The experimental results show that the method can effectively and efficiently generate a stereoscopic panorama. Compared with the original video format, the stereo panorama requires less storage space, possesses wider field of view and greater realism, and enables the inspector to perform quick examination and immersive browsing.