Verticillium wilt poses a severe threat to cotton yield and quality. Rapid and accurate detection of Verticillium wilt is essential for controlling cotton Verticillium wilt (CVW). Existing CVW detection methods mainly focus on the image or spectral level, overlooking the importance of feature fusion, which limits model performance. We propose a CVW grade detection method, spatial-spectral Fusion Network (SSFNet), to address this. First, we enhance the LAB color space, which is sensitive to pixel changes in infected plants, to enrich the feature representation of RGB images and use an improved ResNet network to build an image feature extraction module. Next, we construct a spectral feature extraction module based on the improved ResNet network and compare the performance of two common feature extraction methods: Least Absolute Shrinkage and Selection Operator (LASSO) and Principal Component Analysis (PCA). Finally, we build the feature fusion model SSFNet based on image and spectral level exploration. Experimental results show that SSFNet performs best compared to single data type features, with an F1 score of 95.96%, demonstrating the potential of image-spectral feature fusion methods combined with deep learning for CVW grade detection.