Remote sensing satellites commonly use synthetic aperture radar (SAR) and visible light imaging. SAR and visible image data fusion have become an important research field of remote sensing owing to their high complementarity in imaging information. The accuracy of obtaining ground control points is directly influenced by the performance of a heterogeneous data matching algorithm. There are two methods of matching algorithms: two-stage and one-stage. The existing two-stage method is difficult to adapt to remote sensing images with complex terrain and it cannot meet the actual engineering needs in terms of speed, while the one-stage method meets the requirements in terms of speed but lacks in accuracy. To solve this problem, an end-to-end high-precision heterologous remote sensing image matching algorithm based on a residual pseudo-twin convolution cross-correlation network has been proposed. By constructing a pseudo twin network based on residual layer, the proposed algorithm performs convolution cross-correlation operation on the extracted features of SAR and visible images, so as to realize heterogeneous remote sensing image matching. The results show that this algorithm considerably improves the matching accuracy between SAR and visible images, maintaining a high speed and laying the foundation for the engineering applications of depth learning methods in large-scale heterogeneous remote sensing image matching tasks.