To overcome the shortcomings of the existing magnetic deviation compensation methods in high costs, low efficiency and insufficient tilting compensation ability, a new method based on geometric transformation was proposed to compensate the attitude magnetic deviation of the digital magnetic compass. Firstly, the effect of the magnetic interference on the output of a magnetic sensor was analyzed and a geometric transformation model for the compass deviation was established to reduce the number of magnetic deviation parameters from 12 to 9. Then, a hybrid algorithm for solving these 9 parameters was proposed, in which the Particle Swarm Optimization(PSO) algorithm was taken to adjust the initial value and the Newton algorithm was used to accelerate the convergence process. Finally, the compensation method was used to compensate the magnetic heading error. The experimental results show that the method can achieve the all attitude magnetic deviation compensation without other assisting magnetic heading information,the compensation time is less than 3 min and the magnetic heading error is 1/8～1/10 that before compensation. The method is prior to traditional method in the speed and accuracy and satisfies the requirement of the system for attitude and heading reference.