An ultracompact refractive index sensor based on an integrated optical waveguide trench coupler is proposed herein. The structure includes incident, reflection, and transmission waveguides as well as a submicron-wide trench which is located at the intersection of the incident and the reflection waveguides. The guided wave in the incident waveguide occurs, and the frustrated total internal reflection at the sidewall of the trench makes it so that the light can be partly converted into a transmission wave through evanescent field coupling, while the other partial light will be reflected into the reflection waveguide. The intensity of the transmission light depends on the coupling efficiency of evanescent field, and the reflection light intensity depends on the refractive index of the trench, implying that this integrated optical waveguide trench coupler can be used as a micro/nanosensor for the real-time detection of liquid refractive index or solution concentration. Herein, the new refractive index sensor based on the aluminum nitride rib waveguide is designed and verified via simulation. After the structural optimization, the refractive index sensor has a sensitivity of 207.05%/RIU.