Aiming at the problems of slow speed, high rates of false detection and miss detection in denim artificial operation, we propose an automatic defect detection algorithm of denim by using the optimal Gabor filter. Firstly, an arbitrary two-dimensional Gabor filter is constructed for the normal denim image. Meanwhile an improved differential evolution algorithm is used to optimize the parameters of Gabor filter to get the best match with the normal denim texture. Secondly, the Gabor filter is constructed according to the optimal parameters, following which, an operation of convolution is applied to the image to be detected to obtain the corresponding feature image. Then, the initial detection result is obtained by combination with the threshold operation. Finally, we use the rectangular box and the local Otsu method to separate the exact defect area. Experimental results show that the proposed algorithm can better detect the denim defects with short learning time, strong robustness and high accuracy.