In order to simplify the system calibration and reduce the influence of the error of system calibration and image noise on measurement accuracy by phase measuring deflectometry (PMD), free frame of axes is introduced in PMD, and bundle adjustment in free frame of axes is performed to improve the measurement accuracy. Commonly, the specular shape is measured by PMD with gradient information. However, bundle adjustment can not be directly performed since there is no absolute height information. The introduction of free frame unifies the incident and reflected ray in the same frame, and the absolute coordinate of tested points can be obtained by intersection of incidence and reflected ray. According to the geometrical constraint that, the 3D points must be located on the corresponding incident ray, constraint bundle adjustment is performed to refine the intrinsic, extrinsic parameters and 3D coordinates of shape simultaneously to minimize the reprojection error. Numerical simulation and experimental results demonstrate that bundle adjustment can indeed improve the measurement accuracy in PMD.