Automatically analyzing the sentiment of natural images plays a vital role in analyzing user needs and network public opinion monitoring. However, the training processes of deep learning-based classification algorithms are too difficult to be controlled, and their classification results are always lack of interpretation. A deep learning structure optimization algorithm with human cognition was proposed to classify image sentiment. Firstly, the emotional features extracted were visualized by the convolutional neural networks. Then, the network structure was optimized by combining with human’s subjective perception of image emotion, and the network structure was driven by human knowledge to focus on the apparent features of emotional information. Finally, the parameters of the rebuilt network were fine-tuned to make it more suitable for images sentiment classification task. Contrastive experiments on the Twitter dataset systematically demonstrate that the proposed algorithm achieves 88.1% classification accuracy, which has superior performance than other methods. Ablation experiments confirm that our network optimization improves the classification effect by 8.1%. Besides, the process and reason were intuitively explained for the model operation through class activation maps, spatial location visualization and neuron group visualization. The visualization experimental results further demonstrate the ability of proposed algorithm to recognize the sentiment of natural images.