The estimation accuracy of geographic entity boundaries is one of the key concerns in high-precision 3D modeling. The mobile measurement system based on 3D laser scanner sensors can be used as a technical support platform for high precision object appearance measurement, where the accuracy of the plane fitting algorithm directly determines the reliability of extracting the surface feature position of the measured object from the point cloud data, thereby affecting the determination of object boundaries. To solve this problem, as the shortest distance from a point to the fitted plane serves as the fitting constraint condition, by deriving the intrinsic relationship between the coefficients of the plane equation and the deviation vectors of the point positions, a new plane equation fitting method is established. The position and attitude of the plane are calculated using the Newton iteration method and the principle of least squares. Consequently,accuracy metrics for the plane equation are established, including the plane attitude error and positioning error. Simulation and actual measurement data verify that the plane fitting method is highly accurate and reliable.