Due to the significant nonlinear radiometric differences between synthetic aperture radar (SAR) and optical images obtained by satellite remote sensing, the current SAR and optical images registration algorithms are weakened by their poor real-time performance and weak rotation and scale invariance. To address the problem that the current algorithms only focus on the appearance information on the local features of images and ignore the geometric structure information, a SAR and optical image matching method combining the convolutional and graph neural network (GNN) is proposed. The method uses the convolutional neural network for feature detection and description, and adopts the GNN for feature matching. In contrast to the nearest neighbor matching algorithm that merely uses local descriptor information, the GNN embeds the location coordinates of feature points into the descriptors, thereby providing the descriptors with geometric location information. Then, the geometric context information of the feature descriptors is further aggregated with the attention mechanism. Finally, the matching results of the feature points are directly output by the differentiable optimal transport algorithm to ensure that the network can be trained in an end-to-end manner. The experimental results show that the proposed method achieves state-of-the-art performance on the registration task featuring rotation and scale variation in a large range. In addition, compared with the current mainstream registration algorithm radiation-invariant feature transform, the proposed method not only improves matching accuracy, but also increases the computational speed by more than 50 times.