An elliptic cylindrical fiber optic gyroscope (FOG) for petroleum well trajectory inertial measurement is designed for the special condition of the limited borehole diameter and the large gradient of temperature in inclinometer while drilling (IWD). Due to the thermally induced nonreciprocal phase shift in fiber coil, the three-dimensional (3D) temperature transient response mathematical model of elliptic cylindrical fiber coil is established by quadrupole (QAD) winding method, which is based on the Shupe effect of fiber coil. Combined with the actual working environment of FOG, we use finite element method to numerically simulate elliptic cylindrical fiber coil and analyze the Shupe error at working temperature gradient quantitatively. The experimental results verify the correctness of the model. In order to compare the thermal error rates of two fiber coils, we simulate the cylindrical and elliptic cylindrical fiber coils under the same temperature excitation when the fiber length, the layers of fiber coil and the radius of fiber coil are fixed. The results show that the thermal error rate of the elliptic cylindrical fiber coil is 35% to 39% smaller than that of cylindrical fiber coil and the precision of the well trajectory inertial measurement can be improved when the underground measuring instrument is limited in volume.