To improve the diagnosis and treatment level of pneumonia in children in primary medical institutions and doctors' efficiency and quality in analyzing clinical medical images, an auxiliary diagnosis model of pneumonia in children, based on the Vision Transformer (ViT), is proposed. First, ResUNet is used to segment the lung region in the chest film of children, and the left and right lung regions are separated from the chest film to mitigate the interference of other tissues during pneumonia diagnosis. Further, the segmented image is input into the improved hybrid ViT model for diagnosis. This model uses the feature map of the traditional convolutional neural network (CNN) as the input of the Transformer and introduces the self-attention mechanism into the CNN to improve convolution to enhance its ability to obtain global correlation. Finally, the backbone network of the CNN and Transformer model are trained end-to-end so that the proposed model can achieve good image classification results. Experiments were conducted on the Chest X-Ray Images pneumonia standard dataset. The experimental results show that the accuracy, precision, and recall of the proposed model for pneumonia recognition reach 97.27%, 97.69%, and 98.60% respectively. In other words, the model has good feasibility and can significantly improve the clinical diagnosis accuracy of pneumonia in children at the grass-root level.