Auditory perception is an indispensable function of mobile robots. Electroacoustic microphones have the disadvantages of low sensitivity and susceptibility to electromagnetic interference, which greatly limits the use of mobile robots. In this paper, a Fabry-Perot interferometric (FPI) optical fiber microphone is used to replace the electroacoustic microphone, and a cross-type array is constructed on the mobile robot platform. The FPGA-based signal processing and sound source localization algorithm software is designed to realize the functions of robot auditory perception and passive acoustic navigation. Firstly, the working principle of FPI fiber optic microphone is introduced, and its sensitivity with electret microphone is tested and compared. Then, the FPGA-based time difference of arrival (TDOA) localization algorithm was optimized, and the design and integration of the sound source localization system on the mobile robot was completed. Experiments show that when the average distance is 6.60 m, the average error of indoor positioning azimuth is 1.54° and the average error of distance is 0.09 m and the average error of outdoor positioning azimuth is 1.84° and the average error of distance is 0.16 m. The robot can accurately reach the position of the sound source. All these show that the system has strong practicability and broad application prospects.