Defects in photovoltaic cells can reduce their photoelectric conversion efficiency. At present, defects in photovoltaic cells are commonly detected using electroluminescence (EL) technology. This study proposes a three-stage, attention-based cascaded lightweight model called YOLO-FEE for fast and accurate recognition of defects in EL images of photovoltaic cells. First, we constructed a photovoltaic cell EL image dataset containing seven types of defects. We then constructed the proposed three-stage cascaded lightweight object detection network using CSP-FBE, Faster PANet, and EMSHead. During the experimental evaluation, YOLO-FEE reduced the parameter number and computational complexity by 26.34% and 40.74%, respectively, from those of the YOLOv8n model. The mean average precision of YOLO-FEE reached 96.4%, confirming the accurate and rapid detection of defects in photovoltaic cells. This model is easily deployable on mobile and embedded devices, allowing the automatic EL detection of photovoltaic cell defects in practical industrial environments.