In view of this phenomenon that the output signal precision of the absolute type encoder is not stable, which relies on artificial debugging, a set of output signal error automatic compensation system is designed. The system establishes the connection with the server by TCP communications, and acquired the parameters needing to debug. Then analyze the reason for leading to the signal error. With the high-precision encoder estimated value as the goal of learning, it establishes Radial Basis Function neural network mode, which would form the output value of the minimum error as the ideal input value of the fine adjustment. In this way, it might realize errors minimizing. After the calibration data, write to the encoder chip. Experiments show that, the automatic system is used to compensate each code channel of the value of amplitude, phase and offset, which compared with the manual debugging, and the signal error is reduced on average by 49.82%.