Optical images obtained through remote sensing are widely used in weather forecasting, environmental monitoring, and marine supervision. However, the images captured by optical sensors are adversely affected by the atmospheric conditions and weather; cloud covering also leads to content loss, contrast reduction, and color distortion of the images. In this paper, a cloud removal algorithm for optical remote sensing images is proposed. The algorithm is based on the mechanism of fusion and refinement and is designed to achieve high quality cloud removal for a single remote sensing image. A cloud removal network, based on the mechanism of fusion and refinement, implements a transform from cloudy images to cloud-free images. A multiscale, cloud feature fusion pyramid with a fusion mechanism extracts and fuses the cloud features from different space scales. A multiscale, cloud-edge feature refinement unit with a refinement mechanism refines the edge features of the cloud and reconstructs the clear, cloud-free image. This paper adopts an adversarial learning strategy. The discriminator network adaptively corrects the features and separates out the cloud features for more accurate discrimination, and makes the network generate realistic cloud removal results. The experiments were conducted on an open-source dataset and the results were compared with those of five competing algorithms. A qualitative analysis of the experimental results shows that the proposed algorithm performs better than the other five and removes the cloud without color distortion and artifacts. Further, structural similarity and peak signal-to-noise ratio of the proposed algorithm exceeds those of the second-placed algorithm by 11.9% and 15.0%, respectively, on a thin cloud test set, and by 9.3% and 9.9%, respectively, on a heavy cloud test set.