To address the issue of autonomous obstacle avoidance during unmanned aerial vehicle (UAV) flight in outdoor environments, a lightweight binocular vision-based UAV obstacle avoidance algorithm was proposed utilizing Ghost module to improve YOLOv5. Firstly, the Ghost module was introduced to enhance the CBL and CSP_X units of YOLOv5, while utilizing CIOUloss as the regression loss function, and optimizing the loss function by modifying the non-maximum suppression from CIOUnms to DIOUnms. Secondly, the stereo cameras were calibrated and corrected,and the ORB feature point extraction and sliding window matching algorithm was utilized to obtain the disparity value of the detected targets, and the distance information of the obstacle was solved based on the disparity value and camera intrinsic parameters. Finally, autonomous obstacle avoidance of the UAV was achieved based on the position and distance of the obstacle. The obstacle avoidance algorithm was implemented on an embedded system, an average FPS of 14.3 was achieved, and the feasibility of the algorithm was verified through UAV flight testing. The improved network had an average detection accuracy of 76.88%, which was 0.37% lower than that of YOLOv5, but the detection time and parameter quantity were reduced by 22% and 25%, respectively. This algorithm has significant value for the autonomous obstacle avoidance of UAVs.