The task of scene prediction is studied to improve the decision-making speed of driverless vehicles for reducing the probability of traffic accidents at night. A dual-channel encoding night scene prediction network is proposed based on a convolutional long-short term memory network. First, the temporal features of infrared video sequences and the spatial features of infrared images are extracted by the temporal and spatial sub-networks, respectively. Second, spatial-temporal features obtained by the fusion network are input into the decoding network to predict future frames of infrared video. This is an end-to-end network and can predict multiple frames. The experimental results show that the proposed network is more accurate in night scene prediction and can predict images 1.2 s in the future with a fast prediction speed of0.02s/frame, which fulfills the real-time requirement.