In human-robot collaborative sewing, the premise of realizing human-robot interaction is based on the detection and understanding of robot sewing gestures. The traditional algorithm is characterized by a low gesture-recognition rate and poor target-gesture detection. Therefore, a method based on an improved single-shot multibox detector (SSD) model for recognizing sewing gestures is proposed in this work. First, a deeper Resnet50 residual network was introduced to replace the VGG16 basic network in the original SSD model to improve the network feature extraction capability. Subsequently, a feature base pyramid (FPN)-based network structure was used to perform a fusion of high- and low-level features, thereby further improving the detection accuracy. Experiment results reveal that for the constructed sewing gesture dataset, the improved model exhibited higher detection accuracy than the original SSD algorithm and other algorithms. Furthermore, the residual connection in the network improved accuracy without increasing the number of parameters and complexity of the model. In our method, the average detection speed is 52 frame/s, which can fully meet the requirements for real-time detection of sewing gestures.