Aiming at the accuracy and efficiency of manual and traditional automatic detection of polarizer surface defects, and solving the problem of poor manual design features and generalization capabilities of traditional machine vision, a polarizer surface defect detection method based on improved Faster-RCNN is proposed. First, by comparing the four feature extraction networks, finally select ResNet-101 and introduce the Feature Pyramid Network (FPN) to improve the detection ability of small defects; then use ROI Align instead of the original ROI Pooling to solve the pixel error caused by two rounding operations; Finally, the polarizer surface image is acquired through the acquisition scheme, and three types of defect data sets are established, and the k-means++ clustering algorithm is combined to improve the anchor generation scheme. Experiments show that the improved network has a mAP of 93.5% in the polarizer defect dataset, and the average detection time for a single image to be tested is 0.142s, which can meet the needs of actual detection.