The analysis of the morphological characteristics of retinal vessels is helpful in diagnosing retinal diseases. To segment retinal vessels more accurately, this paper proposes a new method based on a two-stream network. First, the whole vessel and small vessels are segmented using a convolutional neural network with an encoder-decoder structure. Subsequently, the two prediction maps are fused after the artifacts and noises are removed from the fusion image. The final vascular segmentation is then obtained. Because of the separate segmentation of small vessels, the proposed method can more effectively segment small vessel pixels that make it difficult to recognize the edges and low-contrast areas of retinal vessels. Experimental results show that the sensitivity of the proposed method on DRIVE, STARE, and CHASE_DB1 datasets is 0.8062, 0.8308, and 0.8135, respectively. The performance of the proposed method is better than that of other methods.