The traditional image processing algorithms used for the location of Chinese chess pieces have high complexity and the traditional character recognition methods used for the recognition of chess pieces have low generalization and accuracy. A segmentation method based on chess piece color features and an improved binary image filtering algorithm are proposed to achieve the fast location of chess pieces, and the second correction of positions is not needed. A recognition method of chess pieces based on a convolutional neural network is proposed, which can be used for the recognition of chess pieces with different fonts. In the case of chess piece replacement, this method can still recognize chess pieces quickly and accurately. The experimental results show that as for the proposed method, the location error is 0.51 mm, the average location time is 0.212 s, and the average recognition accuracy of chess pieces with four types of fonts is about 98.59%. The effectiveness and practicability of this method are confirmed.