A compensation method was proposed for systematic errors of the tetrahedron magnetic gradiometer on an Unmanned Underwater Vehicle (UUV). With the difference algorithm based on tetrahedron magnetic gradient tensor, the method fused the error of each vector magnetometer and the installation error between the magnetometers to establish the mathematic model of the magnetic gradiometer errors. Based on this error model, the error compensation algorithm was proposed and the compensation coefficient recognition method was presented by mathematic relations of 9 components of the magnetic gradient tensor. The method was verified by simulation experiments. The simulation results show that the proposed method compensates 96.2% systematic errors of the magnetic gradiometer efficiently, and the compensation effect is better than that of the existing method in references. As the method compensates systematic errors of the magnetic gradiometer output directly by the compensation coefficients, it realizes the holistic systematic error compensation of the magnetic gradiometer theoretically.