A method of calibrating radial distortion and intrinsic parameters based on a single template image is proposed. The method based on the constraint that straight lines in the 3D world are imaging as circular arcs in the image plane under the single parameter division model. With the input temple image, the straight lines in the template and the corresponding circular arcs in the distorted image are first identified and estimated. Then the single distortion parameter and the center of distortion are estimated based on the parameters of circular arcs. After the image is undistorted, the homography matrix is calculated and the focal length is linearly estimated. These initial estimations are refined by minimizing the image reprojection error using Levenberg-Marquardt, and accurate results are finally acquired. Results of experiments with fish-eye camera and high-resolution digital camera show that the proposed method is simple, flexible and accurate.