The optical flow method cannot fully exploit the facial color information of micro-expressions, resulting in low recognition accuracy. Therefore, this paper proposes a multi-attention dual-flow network method based on color and optical flow. Firstly, the facial color difference maps are obtained in the CIE Luv color space, and the emotional-physiological features are extracted to compensate for the singularity and limitation of the micro-expression optical flow features. Then, the PAM module and ECA block are combined in parallel to obtain the lightweight dual-attention module, which extracts the spatial and channel key features. Finally, a cross-attention mechanism is designed to obtain mixed features of color and optical flow. The mixed features are fused with spatial channel key features for micro-expression classification. The model is evaluated experimentally using leave-one-out cross-validation. The accuracy and F1 scores reach 69.18% and 67.04% on the SAMM dataset, and 72.38% and 70.85% on the CASME Ⅱ dataset. The experimental results are superior to the current mainstream algorithms, further proving the effectiveness of the proposed model and its modules in micro-expression recognition.