For an omni-directional stereo vision system,two omni-cameras are assumed to have a common optical axes,i.e. co-axis stereo. The epipolar geometry of the ideal configuration is linear along radial direction,which simplifies correspondence search. However,it is often violated in practice due to misalignment between optical axes,which reduces the measurement accuracy. In order to compensate this misalignment,a generalized rectification algorithm for omni-directional stereo system is proposed. The essential matrix is derived and computed for an arbitrary omni-directional stereo configuration based on Taylor model. Using the one-to-one corresponding relationship between epipolar curve and its tangential space,the system is rectified to the direction of the line connecting two cameras′ centers. The corresponding points are constraint to the radial line,and image-resample makes them lie on the same scan column. The proposed algorithm is only performed on image plane. Finally,simulation and experimental results illustrate the effectiveness of the algorithm.