Aimed at the problems of pig skeleton extraction, such as difficulty, low accuracy and long-time consumption, a pig skeleton extraction method based on encoder-decoder network is proposed. The key point heat map generation model was constructed, ResNet50 residual network and U-Net semantic segmentation network were combined to build an encoder-decoder network structure, and the attention mechanism was introduced to improve the feature extraction accuracy of the key points of small targets such as tail and hoof. The offset of key points was predicted while generating the key point heat map, which made up for the accuracy loss when calculating the original position of the key points. The Hough voting mechanism was used to weighted aggregate the two points, and the pig skeleton was finally mapped. The experimental results show that the skeleton extraction accuracy is 85.27%. Compared with the ResNet50 residual network, the accuracy is increased by 22.67 percentage points with similar time consumption. This study provides a new method for pig skeleton extraction, which can provide a technical reference for further pig behavior research.