In view of lacking of the texture information and the edge information in the infrared image, the accuracy of depth estimation is hard to be improved. We propose a deep neural network to estimate the depth of infrared images. The network combines a two-dimensional (2D) residual neural network and a three-dimensional(3D) convolution neural network. The traditional methods of estimating the depth of a single infrared image omits the interframe information and is prone to fuzzy or even missing object contour. The 2D and 3D network inputs are added dense optical flow and the frame before and after the image, respectively. Secondly, the feature map extracted from the 3D convolutional network is further connected to the feature maps of the 2D residual network. Unlike the fully connected layer of the traditional neural network, fully convolutional layer breaks through the size constraints of the input. The experimental results show that the accuracy of the proposed infrared image depth estimation method is improved, and the object contour estimated is clear and complete.