Aiming at the safety of double random phase encoding system in the Fresnel domain, an improved encryption algorithm is proposed. By means of the phase encoding in advance, the new system overcome the flaws of insensitivity to the first random phase mask and the first diffraction distance in original system. After the double random phase encoding module, the secondary encryption based on the amplitude-phase substitution of complex value image makes the pixel value distribution of encrypted image is more uniform. Random masks in the proposed algorithm are generated from three different chaotic systems. With the dynamics of chaotic systems, such as nonlinearity and sensitivity to initial values, the key volume is reduced, the key space is enlarged and complexity of the system is increased. Simulation is conducted on statistical analysis, correlation analysis and key sensitivity test. The experimental results show that the improved algorithm has higher security.