Pansharpening aims to obtain hyperspectral images with high spatial resolutions by fusing hyperspectral images with low spatial resolutions and panchromatic images with high spatial resolutions together. This paper introduces a remote sensing image fusion method based on a deep convolutional neural network (CNN), which extracts spectral and spatial features step by step from hyperspectral and panchromatic images using two independent branch networks. The proposed fusion network is composed of two branches and a main network. The two independent branch networks are used for extracting the spatial-spectral features from hyperspectral and panchromatic images, while based on the features extracted from the branch network, the main network is used to reconstruct and the final fused hyperspectral images with high spatial resolutions are obtained. The experimental verifications were conducted on both CAVE and Pavia Center datasets. Through comparison, one can see that the proposed fusion algorithm outperforms the prevailing algorithms in terms of spatial detail and spectral fidelity.