Combining multiple pushbroom cameras along the sampling direction can overcome the limitation of the detector scale, and effectively expands its field of view and improves the imaging efficiency. Existing stitching methods are based on strict geometric calibration of each sub-field, and strongly dependent on expensive calibration fields. This paper presents a relative orientation method of sub-field of view based on sparse ground control points. This method establishes a relative orientation model with tie points and digital elevation model. To ensure the stability of the solution, it is supplemented by the absolute constraints of the sparse control points. Relationship between adjacent cameras can be achieved by this method, and simulation data was used to verify the method. Results show that the relative orientation parameters can support seamless stitching of sub images. The accuracy of image stitching is better than 2 pixel, and the plane error of geometric positioning accuracy is better than 3 pixel.