Face detection is the first step of face recognition. In order to fix the face region from the complex background rapidly, a method based on LVQ is studied in this paper. By analyzing the color features of skin, images are converted into YCbCr space and HSI space firstly, and then Cb, Cr, H, S color components are extracted to describe the facial image features. An LVQ neural network with a structure of 4-20-2 is constructed, 100 images as training samples and 20 images as test samples are selected for this neural network. The number of iterations is 150 and the error is 0.001. After training, the effective weight are obtained. Using the trained neural network, the performance tests are performed on the LFW, Faces, and AFW datasets. The positive detection rates in the three data sets are 76.82%, 84.42%, 100%, the false detection rates are 17.34%, 12.34%, and 0, and the missed detection rates are 21.55%, 15.63%, and 0, respectively. The experimental results show that the method in our paper shows good superiority in terms of positive detection rate, false detection rate and missed detection rate compared with the traditional face detection method based on skin color features.