A crime scene sketch plays an important role in forensic science as an important part of investigation records. However, in real life, unqualified sketches are developed in many cases. Therefore, this study proposes an automated method to classify crime scene sketches based on a convolutional neural network. First, 64098 crime scene sketches and 27162 photos used as negative samples are collected from the national criminal scene investigation information system (crime scene survey system for short), and are manually labeled to build a crime scene sketch dataset. Then, a new convolutional neural network called XCTNet is designed by introducing “Inception” into AlexNet. Finally, the performance of XCTNet is measured with respect to many aspects, and the images misclassified by XCTNet are extracted. The results denote that XCTNet achieves an accuracy of 98.65% on the test set, which is 3.78 percentage points higher than that of AlexNet; meanwhile, it only uses one tenth of the parameters of AlexNet. However, the recognition accuracy of the proposed method for self-drawn location sketches should be improved.