By analyzing the reason of big noise and low quality of decryption image in the joint transform correlation (JTC) encryption system, a kind of nonlinear JTC image encryption system is proposed. A new encryption image is obtained when JTC encryption image is divided by key power spectrum. On the one hand, new encryption image can remove noise caused by amplitude non- uniformity of key Fourier spectrum and improve the quality of decryption image. Simulation results show that the correlation coefficient between new Lena decryption image and original image increases from 0.4104 to 0.7190 and root mean square (RMS) value decreases from 0.8154 to 0.7089, and that the correlation coefficient between new binary text decryption image and original image increases from 0.8458 to 0.9785 and RMS value decreases from 0.6887 to 0.4583. On the other hand, new encryption image can resist chiphertext only attack (COA) arithmetic attack. Simulation results show that using the COA arithmetic, high quality original image can be restored from JTC encryption image, but information of original image can not be obtained from new encryption image. This system improve the security of JTC encryption system effectively.