Frequency modulated (FM) signal is extensively applied in sonar, radar, laser and emerging optical cross-research, its sparsity is a common basic issue for the sampling, denoising and compression of FM signal. This paper mainly studies the sparsity of FM signal in the fractional Fourier transform (FRFT) domain, and a maximum singular value method (MSVM) is proposed to estimate the compact FRFT domain of FM signal. This method uses the maximum singular value of amplitude spectrum of FM signal to measure the compact domain, and WOA is used to search the compact domain, which effectively improves the shortcomings of the existing methods. Compared with MNM and MACF, this method gives a sparser representation of FM signal in the FRFT domain, which has less number of significant amplitudes. Finally, the primary application of this method in the FM signal filtering is given.