The impulse noise is one of the main noise sources during the imaging processes, meanwhile, it is hard to remove high density impulse noise with traditional filters. To solve this problem, an image denoising algorithm for impulse noise based on asymmetric neural networks is proposed. In this algorithm, the steganalysis rich model is used to extract the convoluted noise feature maps of the noisy image. then the feature maps of original image and noisy convoluted version are delivered to two identical convolutional neural networks. The l1 loss and l2 loss are combined as the total loss function of the neural networks, it takes advantages of high visual quality of l1 loss and strong convergence of l2 loss. Experimental results show that the proposed denoising method outperforms the filter based denoising algorithms on different densities of noise, it also performs better on high density impulse noise.