To overcome the limitation of distortion and quality deterioration in whiskbroom scanning images, we propose a geometric correction and image enhancement method that combines the resolution inversion with deep convolutional neural network (DCNN) architecture. During the whiskbroom scanning process, the total whiskbroom scanning angle and unit field of view angle of a space camera are invariable, and each pixel of the detector on the image plane corresponds to the ground scene pointed by the camera boresight. Suitably, these help in restoring compressed pixels accurately. Furthermore, we adopt real-scene remote sensing panchromatic images as the sample to train the DCNN for remote sensing panchromatic images. Then, image blurring during the process of inversion is solved, and the visual effect of the corrected image is enhanced. In our experiment, the distortion corrected imagery restores the geometric characteristics of the ground scene to a large extent. The no-reference image quality evaluation indicators are used to evaluate our proposed network architecture, network trained on generic image set and interpolation method. The experimental result indicates that our proposed network realizes the best performance of image enhancement among the three methods with a great restoration effect of the whiskbroom scanning images.