Optical fiber refractive index sensor is widely used for monitoring in various complex environments. A high-sensitivity refractive index sensor based on the structure of few-mode fiber (FMF)-coreless fiber (CLF)-FMF was designed, and the experimental verification was carried out. The sensor consisted of a thinned section of CLF fused between two small sections of FMF to form a Mach-Zehnder interferometer (MZI) for measuring external refractive index, and the fiber Bragg grating (FBG) was used for temperature compensation. The resonance trough of interference spectrum generated by MZI structure was affected by both refractive index and temperature, while FBG was only affected by temperature. The sensitivity matrix was constructed by using the refractive index and temperature sensitivity coefficients of MZI and FBG to realize the simultaneous measurement of refractive index and temperature. Experimental results show that the refractive index sensitivity of MZI is 345.66 nm/RIU, and the temperature sensitivity is 0.0134 nm/℃. Meanwhile, the temperature sensitivity of FBG is 0.0104 nm/℃.