Remote photoplethysmography (rPPG) can be used to calculate heart rate from face video. However, most of the proposed rPPG algorithms have many requirements and restrictions on subjects, which is not conducive to heart rate measurement under daily conditions. An improved method based on Plane-Orthogonal-to-Skin algorithm (POS) is proposed to solve the problem of unstable heart rate measurement due to the movement of the tested subjects in the actual environment. Using the frequency domain filter based on color distortion and POS algorithm, the signal is firstly filtered in the frequency domain, and then sent to the POS algorithm for time domain filtering, the influence of different noises on the signal quality is reduced. The result shows that the evaluation indexes of this algorithm are improved compared with other algorithms, and it is suitable for stable heart rate detection under face motion conditions.