To realize nonlinear encryption of optical images, an optical encryption method based on random Fractional Mellin Transform (FrMT) was established and a corresponding encryption configuration was constructed. In this configuration, two mutually conjugated random phase masks generated by the chaotic map were placed on the input and output planes of an optical Fractional Fourier Transform (FrFT) lens, respectively. The essential role of random phase masks was used to process randomly the kernel function of the FrFT to obtain the random FrFT. The random FrMT was composed of a log-polar transformation and a random FrFT. An optical image was encrypted into one complex-valued ciphertext with the random FrMT, by which the dual encryption of was implemented by the secrecy of pixel value and pixel position simultaneously. The mean square error (MSE) between the decrypted image and the input image for all keys were calculated. It shows that the MSE of the decryption image can be magnified over 200 times when the seed of the chaotic map as the key is increased 10-16, which expands the key space of the encryption algorithm. Moreover, the fractional order of the random FrMT which is taken as a key also has a high sensitivity. Numerical simulation results demonstrate the feasibility and the effectiveness of the proposed method, and the performance analysis for noise addition and occlusion of the encrypted image shows that the algorithm is robust.