The influence of nonlinear dispersion effects on modulation instability (MI) in nonlinear metamaterials (MMs) is investigated and some new instability phenomena are revealed. Firstly, a general expression for MI gain, taking account of the effects of arbitrary high-order nonlinear dispersions, is derived. It is shown that all even-order nonlinear dispersion not only deforms the original instability regions, but also may lead to the appearance of new instability regions; while all odd-order nonlinear dispersion always suppresses MI irrespective of their signs. Then, the influence of second-order nonlinear dispersion (SND) induced by dispersive magnetic permeability is further analyzed. It is found that SND not only leads to the occurrence of MI inaccessible in ordinary materials, but also opens up a new instability window with an infinite modulation frequency range and the gain value increases monotonically with increasing modulation frequencies.