As an important infrastructure in the information age, data centers provide all types of key information services. Currently, data centers face high levels of network attacks and are the main targets of network attacks. To improve network security, this study focuses on an anomaly detection method for data center network traffic. This study includes feature selection, dataset distribution balance, and abnormal traffic detection. First, a classification method for imbalanced datasets is proposed, and the classification performance is improved using feature engineering and a mixed sampling algorithm. Second, traffic anomaly detection methods based on Random Forest (RF) and Light Gradient Boosting Machine (LightGBM) are introduced to fully utilize their advantages in processing imbalanced data and noise resistance. The experiment uses the CSE-CIC-IDS2018 public dataset for verification. The results show that the proposed algorithm has a high precision and recall; among the 15 traffic types, the classification precision of 9 types is higher than 90%, and the classification precision of 13 types is higher than 74%. This study is significant for improving data center security, service quality, and network traffic anomaly detection. It not only provides an effective means to address escalating network threats but also makes a positive contribution to the stable operation of data centers and the reliability of information services.