A novel fully phase color image encryption/decryption scheme based on joint fractional Fourier transform correlator (JFRTC) and phase retrieval algorithm (PRA) is proposed. The security of the system is enhanced by the fractional order as a new added key. This method takes full advantage of the parallel processing features of the optical system and could optically realize single-channel color image encryption. The system and operation procedures are simplified. The simulation results of a color image indicate that the new method provides efficient solutions with a strong sense of security.