The existing ellipse fitting algorithm uses the Lissajous figure to solve the demodulation error caused by the non-ideal 3×3 coupler. However, it is difficult to demodulate the signal when it is weak and the corresponding Lissajous figure deviates from the ellipse. This paper proposes a modified ellipse fitting algorithm on the basis of a fixed Lissajous figure for the same system. The algorithm uses the reference signal to determine ellipse parameters and thereby accurately demodulate the weak signal with small phase changes. The experimental results show that compared with the conventional ellipse fitting algorithm, the modified algorithm can accurately demodulate the signals with weak vibration. The detection resolution is increased by 400 times and the demodulated phase has a good linear relationship with the strain. The results prove that the modified algorithm solves the problem that the conventional ellipse fitting algorithm cannot demodulate weak signals and improves the measurement resolution of the system.