Workspace measuring and positioning system is a type of large-size measurement system based on intersection of optical planes. Optical surface was discovered a non-planar face while measuring, which influenced the measuring accuracy of the system. We presented a method of describing optical surface by using ternary quadratic equation. Ternary quadratic equation could be obtained by fitting the discrete points on the optical surface with the method of linear least square. Based on the ternary quadratic equation and orthogonal linearity transformation, we analyzed the shape of the optical surface. By using the method of quadric surface fitting and plane fitting, respectively, with the same group of data, then comparing the results, we draw the conclusion that it’s more accurate to describe the scanning optical surface with parabolic cylinder equation.