Diseases affect crop quality seriously and cause economic losses. Disease spot segmentation is an important process of identification and disease severity estimation, whose segmentation results can provide an effective basis for subsequent identification and severity estimation. Due to the irregularity and complexity of lesions, and the visible spectrum image of lesions in the natural environment is susceptible to be change in illumination, traditional image processing methods have low accuracy, low universality and robustness for image segmentation of lesions. In this regard, this article proposed a method for the segmentation of crop leaf diseases based on semantic segmentation and visible spectrum images. Firstly, taking peanut brown spot and tobacco brown spot as the research objects, 165 visible spectrum images were collected using a Nikon D300s SLR camera. The visible spectrum images of the diseases were pixel-labeled through the Matlab Image Labeler APP, and the brown spot and background area were respectively marked. Secondly, the labeled dataset adopted image enhancement methods such as horizontal flipping, vertical flipping, changing brightness, etc., to obtain 1 850 enhanced sample data sets and randomly divided them into the training set, validation set and test set according to the ratio of 8∶1∶1. At the same time, in order to save computational cost, the pixel resolution of the data set was adjusted to 300×300. Finally, four types of disease spot segmentation models were constructed based on the three semantic segmentation networks of FCN, SegNet and U-Net. The effects of data enhancement and disease types on the lesion segmentation model were explored. Four segmentation indicators were used to evaluate the model’s segmentation effect. The test results showed that only for lesion segmentation, image enhancement could improve the segmentation accuracy of the model. The model’s Mean Precision (MP) and Mean Intersection over Union (MIoU) were 95.71% and 93.36%, respectively. The 4 semantic segmentation models were significantly better than the Support Vector Machine (SVM). Compared with the U-Net, SegNet-2 and SegNet-4 segmentation models, FCN can effectively avoid the influence of light changes. The Precision (P) of lesion segmentation and the Intersection over Union(IoU) reached 99.25% and 97.55%, respectively. For the lesion classification and segmentation experiment, the Precision (P) of FCN for the two diseases reached 90.41% and 97.54%, and the Intersection over Union(IoU) of the two diseases reached 95.61% and 70.30%, respectively, which were better than the other three segmentation models. FCN can distinguish disease types well while segmenting disease spots, which has good generalization and robustness and realize the identification and segmentation of disease spots in natural scenes and provide a technical reference for the severity estimation of mixed diseases.