Visual odometry is commonly used in various applications including intelligent robots and self-driving cars. However, traditional visual odometry algorithms based on the pinhole camera with a limited field of view (FOV) are usually fragile to moving objects in the environment and fast rotation of the camera, resulting in insufficient robustness and accuracy in practical use. This paper proposes panoramic annular semantic visual odometry as a solution to this problem. Using the panoramic annular imaging system with ultra-wide FOV into visual odometry and coupling semantic information provided by the panoramic annular semantic segmentation based on deep learning into each module of the algorithm, the effect of moving objects and fast rotation is reduced; then, the performance of visual odometry in dealing with these challenging scenarios can be improved. Compared with traditional visual odometry systems, experimental results show that the proposed algorithm achieves more accurate and robust pose estimation in realistic scenarios.