Aiming at the requirement of polarized light navigation for accurate position information of feature points in the sky, an accurate detection method for the solar position of imaging system based on all sky polarization mode is proposed. Compared with the traditional detection method of the solar position based on spot, we use the inherent polarization information in the atmosphere to complete the accurate measurement of the solar position, which has the characteristics of simple, high accuracy and wide application range. The optical acquisition system consists of three micro large-field-of-view camera modules and polarizers, which makes the structure more compact, smaller and lower in height. Starting from the principle, the algorithm of solving the solar position is simulated first, and then the algorithm is verified in three weather environments (sunny, occluded, and aerosol) using the optical acquisition system. It can be seen that when the weather is clear, the sun is detected at different times of the same day, and the accuracy of the measured sun's altitude and azimuth are 0.024° and 0.03° respectively; when the sun is blocked by high-rise buildings, the accuracy of the measured sun's altitude and azimuth are 0.08° and 0.05°; when the sun is blocked by the branches and leaves of trees, the accuracy of the measured sun's altitude and azimuth are 0.3° and 0.1° respectively. Only when the aerosol concentration exceeds a certain amount will the Rayleigh distribution mode of polarized light be destroyed, which will affect the detection accuracy of solar position. The experimental results show that this new detection method can not only meet the needs of polarized light navigation for the solar position, but also provide a new way of exploration for fans who like to explore the mysteries of the universe.