In order to replace the commonly used normal diameter Polarization Maintaining (PM) fiber with the ultra-fine PM fiber in fiber optic gyroscopes and other key components, the method of determining its polarization direction needs to be improved and optimized. First, theoretical images of side-view-image method were obtained by computer simulation using ray-tracing method. Then, the situations were verified whose simulated results were better, and the experimental curves were Fourier expanded to suppress random noise. The three-bright-line method whose experimental result was better was improved into three-bright-line-four-feature-point method, and this method was verified by experiments. Finally, the repeatability of this method was measured. Experimental results indicate that the three-bright-line-four-feature-point method has greatly increased the anti-jamming capability of the process of ultra-fine PM fiber polarization direction’s determining, and the repeatability achieves 1.6°. This result has reached the repeatability of common PM fiber polarization direction’s determining, and met the requirement of practical applications..