To accurately reflect the geometric imaging relationship of cameras, a self-calibration method was proposed based on simplified Brown nonlinear camera model and improved BFGS (broyden-fletcher-Shanno) algorithm. In this method, the linear camera model and the distortion model were fitted into a nonlinear model, and the nonlinear model parameters were constrained by fundamental matrices of the linear model to obtain a set of nonlinear constraint equations.Then, based on new quasi-Newtonian equation, an improved BFGS algorithm suitable for nonlinear internal parametric constraint equations were presented and the internal parameters of the equation were solved. By using the proposed model and algorithm, the calibration method improves the accuracy and robustness of the calibration results in fewer iteration times and noise conditions. The convergence analysis and robust analysis in with or without noises show that the reprojection error is less than 0.4 pixel when the noise is not greater than ±3 pixel. A real image experiment was performed by calibrating camera parameters and calculating the projection error, and the results show that the calibration precision error is less than 0.06%, and the re-projection error is 0.35 pixel, which verifies the effectiveness of the proposed method. It concluds that the method is applicable to image processing, mode classification and scene analysis in computer vision field.