In this study, a tracking algorithm based on channel pruning and weighting is proposed for improving the speed and accuracy of the convolutional correlation filter algorithm. This algorithm selects the single-layer convolutional features that are suitable for tracking an object. Initially, the feature mean ratio is proposed to prune the inconclusive channels; then, a combination of one-dimensional gray features is used for improving the feature representation. Subsequently, we construct the weighted correlation filter algorithm by considering the feature mean ratios as the convolution channel weight for predicting the target position. Further, an accurate location method based on the minimizing mean frame is used to reduce the prediction location error. Finally, the tracking model is updated to improve the tracking speed. The different algorithms are tested using the OTB-100 dataset. Results show that the average distance precision and the average speed of the proposed algorithm are 91.3% and 31.8 frames/s, respectively. Furthermore, the proposed algorithm can track an object under occlusion, scale variation, fast motion, and deformation in real time, effectively improving the speed and accuracy of object tracking.