A method of calibrating radial distortion for fish-eye lens based on collinear constraint is proposed. The method only needs a single image which contains at least two sets of parallel lines in 3D world. Under single parameter division model, these lines are imaged as circular arcs in the distorted image. These circular arcs imaged from the same set of parallel lines will intersect at two common points, and thus their centers are collinear. Under this constraint, the calculation of parameters of the single parameter division model is then proposed, which can be solved by using the Levenberg-Marquardt algorithm effectively. Experimental results of both synthetic and real images show that the proposed method is simple, robust and accurate.