Based on the coupled-mode theory, according to the variation of effective index of cladding modes in coated long-period fiber grating (LPFG) with the film parameters, the partition and characteristics of mode transition region are pointed out. The response characteristics of refractive index in coated LPFG transmission spectrum in mode transition region and its vicinity are investigated. Further, the two kinds of sensitivity definitions are given for the different detection types of wavelength shift and amplitude change from the aspect of sensor design. The changes of sensitivity with film parameters and grating parameters are discussed. By using optimization method, the optimum design parameters and regions with high sensitivity are obtained. The results show that the sensor resolution of film refractive index can be also available to 10-7, if detected by wavelength shift and transmittance change. When the suitable film parameters and grating parameters are selected, mode transition region is fit for the sensor detected by wavelength shift, while the vicinity of mode transition region is fit for the sensor detected by transmittance.