The frequent occurrence of brown tide pollution in recent years has brought huge losses to the economy of coastal areas. Therefore, the accurate and efficient identification of brown tide algae is of great significance to the prevention of marine environmental pollution. In this paper, a combination method of three-dimensional fluorescence spectroscopy, gradient boosting decision tree (GBDT), and logistic regression (LR) is used to achieve accurate identification of brown tide algae. In order to solve the problem of weak feature combination ability of LR model for nonlinear data, the GBDT algorithm is introduced to make full use of the advantages of the integrated learning algorithm in processing nonlinear data. The prediction result of GBDT model is used as a new feature instead of the original feature which is input into the LR model, and a brown tide algae recognition model (GBDT-LR)that combines GBDT and LR is established. In response to the interference of other types of algae in the complex marine environment, five different types of algae such as Chlorella and Synechococcus elongatus are introduced for comparison in the experiment, and analyzed the identification of brown tide algae in different growth cycles are analyzed. The proposed model is compared with LR, support vector machine (SVM) and back propagation (BP) neural network under the same conditions. The results show that the GBDT-LR model is superior to the other models in terms of classification accuracy, recall rate, and F1-score. The fluorescence spectrum of algae in the exponential growth period is the most stable, and the identification result of the brown tide algae in this period is the best.