The proportion of traffic accidents caused by driver factors is high, therefore, it is of great significance to study a recognition method for the correct identification of abnormal driving behavior by analyzing the driver activity state. We propose a recognition method of abnormal driving behavior based on the covariance manifold and two classification of multi-class LogitBoost classifier. First, we extract the basic features, such as texture, color and gradient direction, to overcome the shortage of recognition of driving behavior based on a single feature. Then, we use the covariance manifolds for the multi-feature fusion to eliminate the feature redundancy and reduce the impact of image processing and recognition due to excessive differences in numerical values of different features. Finally, the classification and identification are performed using a multi-class LogitBoost classifier based on two classification. The experimental results show that compared with the traditional multi-class LogitBoost method, the proposed method greatly improves the correct rate of multi-classification, and the correct recognition rate for different targets can reach 81.08%.