Research demonstrates that a Fresnel hologram can be generated and simultaneously encrypted numerically based on a secret symmetric key formed by the maximal length sequence (M-sequence). The method can be directly extended to encrypt a video holographic clip in a frame-by-frame manner. However, given the limited combination of signals in the family of M-sequence, hacking the secret key through trial and error can be time consuming but not difficult. In this letter, we propose a method that is difficult to crack with brute force for encrypting a holographic video sequence. An M-sequence is first randomly assigned to encrypt each frame of the holographic video signal. Subsequently, the index of the selected M-sequence, which is necessary to decrypt the hologram, is encrypted with the RSA algorithm before transmitting to the receiving end. At the receiving end, the decoder is provided with a private key to recover the index for each frame, and the corresponding M-sequence is used to decrypt the encoded hologram.