In consideration of the disadvantages such as boundary distortion and poor adaptability in the method for microstructure surface error separation, the bidimensional empirical mode decomposition (BEMD) with adaptive time-frequency resolution capacity was put forward and applied to three-dimensional engineering surface error separation; at the same time, Riesz transform was utilized to construct monogenic signal, calculated overall frequency characteristics of the signal and improved the termination criterion for bidimensional empirical mode decomposition, thus keeping it in strict accordance with the error specified in ISO4287 for each frequency band of the 3D surface with cutoff wavelength and long separation. The simulation results show that compared with Gaussian filter in ISO and commonly-used wavelet filter, this new method can achieve a far better separation effect than traditional methods in error separation for 3D engineering rough surface, and the 3D evaluation parameter errors corresponding to the error in each frequency band are all lower than 5%. Finally, an instance analysis was conducted in the optical filming elements, and the results show that the method can well separate the spatial information of each surface error, and the reference surface for parameter evaluation is free from problems such boundary distortion compared with traditional methods. Hence, the application of the method in actual engineering surface error evaluation is of great feasibility.