Phase-sensitive optical time-domain reflection (Φ-OTDR) technology is a distributed fiber-optic sensing technique with the advantage of high-precision vibration monitoring. It can be used to detect disturbance events in the field of perimeter security. Traditional recognition methods require the manual extraction of vibration signal features and cannot retain a time correlation, leading to information loss. To solve this problem, a disturbance event recognition method based on GAF-HorNet, which does not require a feature extraction step, is developed. A one-dimensional vibration signal is converted into a two-dimensional image through a Gramian angular field (GAF), and HorNet is used to train the model and perform recognition and classification. To verify the performance of the algorithm, four classical algorithms are selected to train the model for comparative experiments. The experimental results demonstrate that the average accuracy of the proposed algorithm is 93.56% for six types of signal: background noise, stone knocking, stone stroking, branch stroking, pulling, and climbing. Compared with previous methods, the method proposed has better recognition rate and false alarm rate performances.