In recent years, the research of facial expression recognition has achieved good recognition accuracy, but in the actual environment, due to the influence of posture, occlusion, light and other factors, its detection accuracy has a little weakening effect. To solve these problems, a new FER system based on the dual-stream convolutional networks is proposed. A dual stream CNN from two aspects of appearance and geometric characteristics is established . The network based on appearance features is to extract the local derivative pattern features of the preprocessed image as the input, whereas the geometric feature-based network learns the coordinate change of action units’ landmark, which is a muscle that moves mainly when making facial expressions. In addition, a technique to generate facial images with neutral emotion using the autoencoder technique is be proposed. By this technique, the dynamic facial features between the neutral and emotional can be extracted images without sequence data. The detection accuracy of the algorithm is 98.81% and 96.05% respectively on CK+ and Jaffe datasets, which shows better results compared with other latest methods.