The existence of defects in solar cells due to the process or material reasons in the production process. Based on the photoluminescence imaging principle, an image enhancement method for solar cells based on background assessment and a defect recognition method based on morphological feature and HOG feature fusion were proposed. Firstly, the characteristics of shape and location of cell defects were analyzed, and the two-step segmentation method was proposed to extract multi-directional HOG features from the segmented defects, and Laplace feature mapping method was adopted to reduce the dimension of HOG features. Then, the morphological characteristics such as aspect ratio and circularity were fused. Finally, according to the kernel function and penalty factor in support vector machines (SVM), the particle swarm optimization (PSO) algorithm was optimized to improve the defect classification effect. Fifty images were detected by using the proposed method, and the accuracy of classification recognition reached 98.3%. Comparing the proposed algorithm with the traditional SVM algorithm and Le-Net network, it can be seen that the proposed algorithm has the higher recognition accuracy.