Although the face recognition model based on the deep convolutional neural network can achieve high recognition accuracy, there are massive calculations in the model and a large amount of memory resources are required, which cannot meet the resource constraints and real-time requirements. To solve this problem, two lightweight recursive residual neural networks are designed, which can effectively fuse the information between the layers in the feature map, enrich the semantic information of the feature map and improve the recognition accuracy. First, the MTCNN face detection algorithm is used to face alignment and cropping on the original data set. Then, the ArcFace loss function is used as the supervision signal, this loss function can make the data set aggregation and inter-class dispersion, effectively improve the classification effect of the model. Finally, the model is verified on the LFW, AgeDB and CFP-FP datasets. Experimental results show that the designed network model can achieve high face recognition accuracy while reducing a large number of parameters.