Real-time analysis of vehicle motion state has important practical application value in automatic driving and assistant driving of vehicles. In order to realize the judgment of vehicle behavior, a vehicle behavior recognition algorithm based on lane information fusion is proposed. Firstly, a model based on improved Robinson and LSD is proposed. The improved Robinson operator is used to obtain the optimal gradient amplitude to realize the edge extraction of the lane, and then the lane detection is realized by LSD algorithm. Then, a cubic spline interpolation method based on sliding window is used to fit the lane. Finally, the motion state of the vehicle is analyzed according to the lane parameter information, and the deviation information of the vehicle is obtained combining with the center position of the vehicle. In the test of BDD100K dataset, the accuracy of lane detection in the algorithm is 9561%, the accuracy of vehicle behavior recognition is 93.04%, and the number of transmission frames per second reaches 42.37. The experimental results show that the proposed algorithm can effectively distinguish the motion state of the vehicle and give the vehicle deviation information in different scenarios, which has higher accuracy and robustness.