To improve the angular accuracy of the polarization imaging orientation sensor, an error propagation model of the polarization imaging orientation sensor based on Stokes vector was established. The principal point deviation, lens distortion, and CMOS planar inclination were studied. First, the polarization azimuth calculation method was derived according to the polarization measurement principle of the Stokes vector. Then, the effects of principal point deviation, lens distortion, CMOS plane tilt, and gray response inconsistency were explored. The Monte Carlo method was used to analyze the measurement error of the polarization azimuth. The rationality of the model established was verified by simulation and skylight imaging experiments. The experimental results show that the CMOS gray response inconsistency has the greatest impact on the measurement error of orientation. In the range of (-6, 6) pixels, the measurement error is within 0.11°, and the mean has an overall offset of 0.12°. The principle point deviation changes (-1.5, +1.5) pixels, the distortion shifts (-1, +1) pixel, the CMOS plane tilt is (-0.5°, +0.5°), and the CMOS gray response deviation is (-6, +6) pixels; under the joint action, the measurement error of orientation is 0.236 3°(σ). The error propagation model may reflect how the error source impacts the accuracy of orientation measurement, and it provides a basis and reference for optimal design of the polarization imaging orientation sensor.