Object detection in remote sensing images is a fundamental task in image analysis and interpretation. We proposed a Multiscale Dilated Convolution Feature Fusion Detector (MDCF²Det) to achieve precise object detection in remote sensing by addressing the problems of multiscale objects and the complexity of the background. To begin, we improve the original feature pyramid network by replacing the general convolution with the dilated convolution to increase the receptive field. Second, to take full advantage of different levels of semantic and location information, we add a skip connection operation from the input node to the output node. Finally, to suppress the noise and highlight the foreground, we add the multi-dimensional attention model before the regional proposal network, to achieve more accurate object detection in remote sensing images. Experiments are carried out on the DOTA and RSOD datasets, and the proposed algorithm’s mean average precision reaches 92.95% and 73.39% respectively. The results show that the proposed algorithm can significantly improve the object detection accuracy of remote sensing images.