At present, Convolutional Neural Network (CNN) has made a breakthrough in species recognition. As an important part of the agricultural economy, shellfish has a wide variety of species with complex characteristics. Some of the shellfish are highly similar and the distribution of various samples is unbalanced, which causes a low accuracy of CNN classification. In view of this situation, a shellfish recognition method based on visible spectrum and CNN is proposed in this paper, which aims to extract more effective shellfish features to improve the classification accuracy of shellfish. Firstly, a filter information measurement and feature selection method including output entropy measurement and orthogonality measurement is proposed, which reinitializes the pruned filter and makes it orthogonality, captures different directions in the network activation space, so that the neural network model can learn more useful shellfish feature information and improve the classification accuracy of the model; secondly, a shellfish classification objective function including regularization term and focal loss term is proposed, which reduces the weight of easily classified samples by controlling the shared weight of each sample to the total loss, it tilts the attention of the model to the samples with inaccurate prediction, so as to balance the distribution of samples and the difficulty of sample classification, and improve the accuracy of shellfish classification. The shellfish image dataset in this paper consists of 74 shellfish species with 11 803 pictures in total. After obtaining the original dataset, data augmentation which consists of horizontal flipping, vertical flipping, random rotation, rotation within the range of [0, 30°], scaling and moving within the range of [0, 20%] and moving is performed on the images of the dataset, increasing the number of images from 11 803 to 119 964. The whole image dataset is randomly divided into training set with 95 947 pictures, validation set with 11 996 pictures and test set with 12 021 pictures in an 8:1:1 ratio. In this paper, based on the establishment of the shellfish image dataset, the experimental verification has reached the classification accuracy of 93.38%, which increases the accuracy of the benchmark network (Resnest) by 1.18%. Compared with SN_Net, and MutualNet, the accuracy of the proposed method is increased by 4.34% and 0.85%, respectively. And the training time is 22 320 seconds, which shortens the training time of the benchmark network (Resnest) by 960 seconds, the training time of the proposed method is 3 180 seconds and 2 460 seconds shorter than SN_Net and MutualNet, respectively. The experiments results demonstrate the effectiveness of the proposed method.