A fluorescence spectroscopy method of classification for phytoplankton populations is developed based on the high-frequency component of wavelet transform. Three-dimensional (3D) fluorescence spectra of 52 species are projected onto the wavelet function and a series of high-frequency components (cd1~cd6) are obtained. The characteristic points are chosen by the standard deviation and used to form new feature vectors. These feature vectors are analyzed by Bayesian discrimination and cd3~cd6 components are selected as the optimal feature vector for differentiation with the discriminant accuracy rate as a standard, based on which, nonnegative least squares (NNLS) method is introduced to establish the discrimination technique. The technique is used to identify algal species at both the division and the genus level and the correct discrimination rates (CDRs) are 95.5% and 85.7%, respectively. For the actual mixture samples (the mixed proportions are 100%, 75%, 25%), the CDRs are 100%, 90.9%, 53.3% with the relative contents of 79.7%, 68.3%, 17.5%, respectively at the division level and the CDRs of the dominant species (75%) is 81.2% at the genus level. For the water samples from mesocosm experiment and the Jiaozhou Bay, the method can be used to realize the identification of phytoplankton population and estimate the relative abundance of different classes at the division level effectively.