The sensing theoretical model of long-period gratings (LPGs) with high-refractive-index nanometer film coating is built. The influence of overlay thickness and external refractive index on the resonant wavelength of LPG is investigated theoretically. The phenomenon of reorganization of cladding modes at different coating film thicknesses is shown and explained in detail. The theoretical analysis shows that a significant modification of the distribution of cladding modes is seen as an azimuthally symmetric nanoscale high-refractive-index coating deposited on the cladding of LPG. If the parameters of layers (the refractive index and thickness) and the external refractive index are chosen accurately,the lowest cladding mode HE1,2 will become the overlay mode and other cladding modes will experience mode transition. For the lower cladding modes (such as HE1,6),there is two-step modes transition. For higher cladding modes(such as HE1,14),there is only the one-step modes transition. The mode transition on the sensitivity of the external refractive index is also analyzed:Modes transition shifts to lower refractive index region with the increase of the overlay thickness.