Based on the combination of the three-dimensional fluorescence detection technique and the alternating penalty trilinear decomposition algorithm, a method is proposed to detect oil species in mixed oils. First, three-dimensional fluorescence spectra of 20 samples of the mixed oil (jet fuel and lube) with different volume ratios are obtained by the FLS920 fluorescence spectrometer, and calibrated by the Delaunay interpolation method. Then, the number of components required for the alternating penalty trilinear decomposition algorithm to analyze three-dimensional fluorescence spectra is determined by the core consistency function. Finally, the effectiveness of the alternating penalty trilinear decomposition algorithm in analyzing three-dimensional fluorescence spectra is evaluated by the root mean square error and the correlation coefficient matrix. The results show that both of the root mean square error and the non-diagonal elements in the correlation coefficient matrix meet the threshold requirements of 0.05 and 0.95, which are obtained by the alternating penalty trilinear decomposition algorithm. As for the solution of the serious overlapping problem of three-dimensional fluorescence spectra, the alternating penalty trilinear decomposition algorithm is superior to the parallel factor algorithm. The purpose of rapid detection of oil species in mixed oils can be achieved by the alternating penalty trilinear decomposition algorithm.