Based on the dispersive Drude model in metamaterials, nonlinear coupled Schrdinger equations are derived for two copropagating optical waves with cubic-quintic nonlinearity and modulation instabilities induced by the cross-phase modulation (XMI) are studied. The impact of quintic nonlinearity on the gain spectra of XMI is analyzed. It shows that the quintic nonlinearity strengthens the XMI with broader XMI spectra and higher peak gain. It is found that the XMI gain is obviously larger in the region with abnormal group velocity dispersion (GVD) than that in the region with normal GVD. Two cases with or without the group velocity mismatch (GVM) in XMI are compared and the GVM is found to play an important role in the occurrence of XMI. It also shows that the serious XMI occurs when two optical waves propagate simultaneously in the positive refractive index region or in different refractive index regions.