The polarization axis direction of the sub focal plane polarization image sensor influences the accuracy of the polarization parameter calculation using the Stokes vector. Therefore， it is necessary to calibrate the sensor before using it. To meet this requirement， a method based on measuring the phase difference of the curve fitted by the average output value of pixels in each polarization axis direction of the sub focal plane polarization image sensor is proposed， and the simulation and actual test are conducted. First， the linearly polarized light vertically irradiating the sensor surface is rotated for several cycles according to a certain step size， and the sensor takes pictures simultaneously. Then， the region of interest of the image is selected， and the pixel values in the same polarization axis direction in the region are averaged to obtain the sequence of the average values of the pixels in each polarization axis direction changing with the rotation angle. The Fourier series fitting method is used to fit each sequence， and the expression of the pixel output value changing with the rotation angle is obtained. Finally， one polarization axis is selected as the reference direction， and the other polarization axis is the phase difference between the expression of this direction and that of the reference direction. The actual test results show that the calibration error is less than 0.1°. This method has high calibration accuracy and good operability. Furthermore， it can complete the task of calibrating the polarization axis direction of the sub focal plane polarization image sensor.