In order to improve the stability of network transmission, an automatic identification algorithm for abnormal traffic in elastic optical networks based on isolated forest algorithm is proposed. Perform spectral density detection based on the abnormal distribution characteristics of traffic and the differences in normal data, construct a spectral feature extraction model for elastic optical network traffic, implement spectral feature filtering for abnormal traffic through low-pass filter convolution vector reorganization, adopt isolated forest algorithm to achieve adaptive optimization control for network traffic anomaly detection, and combine multi-dimensional spatial structure reorganization method to achieve detection and recognition of abnormal traffic in elastic optical network. The results showed that the missed detection rate and the false detection rate were relatively low, 3. 16% and 1. 03%, respectively. The detection takes less time, only 16 seconds. During detection, the external intrusion rate does not exceed 1%, and the immunity is strong.