Low energy impact can induce invisible damage of carbon fiber reinforced polymer (CFRP). The damage can seriously affect the safety of the CFRP structure. Therefore, damage detection is crucial to the CFRP structure. Impact location information is the premise of damage detection. Hence, impact localization is the primary issue. In this paper, an impact localization system, based on the fiber Bragg grating (FBG) sensor network, is proposed for impact detection and localization. For the completed impact signal, the FBG sensor and narrow-band laser demodulation technology are applied. Wavelet packet decomposition is introduced to extract available frequency band signals and attenuate noise. According to the energy of the available frequency band signal, an impact localization model, based on the extreme learning machine (ELM), is established with the faster training speed and less parameters. The above system is verified on the 500 mm × 500 mm × 2 mm CFRP plate. The maximum localization error and the minimum localization error are 30.4 mm and 6.7 mm, respectively. The average localization error is 14.7 mm, and training time is 0.7 s. Compared with the other machine learning methods, the localization system, proposed in this paper, has higher accuracy and faster training speed. This paper provides a practical system for impact localization of the CFRP structure.