There exist various kinds of spatial frequency errors on the ultra-precision machined surfaces, which seriously influence their performances. According to different performances of workpieces, it is necessary to use an effective decomposition method to extract the topography containing the spatial frequency errors at specific frequency bands. The traditional spatial frequency error decomposition method has the serious problem of modal aliasing. In order to solve this problem, an adaptive bidimensional variational mode decomposition (BVMD) algorithm is proposed to decompose a three-dimensional surface topography. First, image continuation and self-convolution Hanning window are introduced to preprocess the truncation errors when collecting 3D topographic data. Then, the particle swarm annealing optimization algorithm is used to optimize the penalty coefficient and the number of decomposition layers in the BVMD algorithm. Among them, the fitness function of the optimization algorithm is constructed by taking KL divergence among modal components as aliasing indicators, introducing the minimum risk Bayesian decision theory, and combining KL divergence with reconstruction errors. Finally, the measured topography of the ultra-precision machined surface is analyzed and compared with those by the discrete wavelet decomposition method and the bidimensional empirical mode decomposition methods. The results show that the KL divergence by the proposed method is several hundred, much higher than those by the other two methods. The proposed method has a good inhibition ability for frequency error modal aliasing, and can effectively decompose the spatial frequency errors of an ultra-precision machined surface.