In the unconstrained open-space, face detection is still a challenging task due to the facial posture changes, complex background environment, and motion blur. The rotation-invariant algorithm based on cascaded network and pyramid optical flow is proposed. Firstly, the cascading progressive convolutional neural network is adopted to locate the face position and facial landmark of the previous frame in the video stream. Secondly, the in-dependent facial landmark detection network is used to reposition the current frame, and the optical flow mapping displacement of the facial landmark between the two frames is calculated afterwards. Finally, the detected face is corrected by the mapping relationship between the optical flow displacement of the facial landmark and the bounding box, thereby completing the rotation-invariant face detection. The experiment was tested on the FDDB public data-sets, which proved that the method is more accurate. Moreover, the dynamic test on the Boston head tracking da-tabase proves that the face detection algorithm can effectively solve the problem of rotation-invariant face detection. Compared with other detection algorithms, the detection speed of the proposed algorithm has a great advantage, and the window jitter problem in the video is well solved.