The traditional algorithms for the extraction of sign language features only rely on the low-level features to realize recognition, which makes it difficult to obtain the high-level semantic features and the misunderstanding of sign language is further induced. To solve this problem, the idea of image semantic analysis is introduced into the study of sign language recognition and then an optimized fully convolutional neural network algorithm is proposed. The fully convolutional neural network is used to extract the semantic features of sign language images and the discriminative random fields for semantic annotation is used for the post-smoothing to recover the detailed information among pixels and thus the sign language recognition is completed. The experimental results show that the proposed algorithm has strong robustness and can be used to obtain the semantic features effectively. Compared with the traditional algorithms, this method can be used to identify sign language accurately with an average recognition rate of 97.41%.