Esophageal squamous cell carcinoma (ESCC) is one of the most common malignant digestive tract tumors in China. Clinically, narrowband imaging combined with magnifying endoscopy (NBI-ME) can be used to investigate the morphological changes of microvessels in the esophageal mucosa and serves as an important means of diagnosing ESCC. To solve the ESCC recognition model's difficulty in considering both the recognition accuracy and reasoning efficiency, a lightweight residual network (CALite-ResNet) with an integrated attention mechanism is proposed to classify esophageal NBI-ME images. The dataset for this study comprises 11468 NBI-ME images of 206 patients collected from multiple hospitals. The experimental results show that the accuracy and sensitivity of the ESCC recognition is 96.39% and 95.70% at the image level, and 95.70% and 94.62% at the patient level, respectively, and the average prediction time of a single esophageal image is 16.42 ms. Therefore, the CALite-ResNet model has a higher recognition accuracy and faster reasoning efficiency for ESCC recognition, and a certain clinical significance and application value, thereby making it effective for use in the auxiliary clinical diagnosis of ESCC.