The feature pyramid network (FPN) adopts the method of upsampling and addition when fusing different scale feature maps. However, the spatial stratification information of the upsampled feature map is seriously lost, so that direct addition will inevitably make certain errors. At the same time, the deep feature information of the FPN structure is poorly forward-transferred, and its auxiliary effect to the shallower layer basically disappears. This paper uses the advantages of Long Short-Term Memory (LSTM) network in processing context information to improve the FPN structure. A top-down memory chain is established between feature layers of different depths, and a multi-gate structure is constructed to filter and fuse the information on the memory chain to generate a higher semantic feature map with stronger representation ability. Finally, the improved FPN structure is added to the SSD (Single Shot MultiBox Detector) algorithm framework, and a new feature fusion network, MSSD (Memory SSD), is proposed and verified on the Pascal VOC 2007 data set. Experiments show that the improved algorithm has achieved better test results, and it has certain advantages compared with the current advanced detection algorithms.