A transmission ellipsometer with the configuration of heterodyne interferometer and two acousto-optical modulators was investigated. A single layer of indium tin oxide on a glass substrate was measured, and the measurement errors of the sample thickness and refractive index range up to 8 nm and 7%, respectively. The influence of the elliptic polarization due to metal-coated mirrors on measurement accuracy was analyzed with Jones vector method. The error only produced by the depolarization effect of metal-coated mirrors can be eliminated just by removing the sample out of the configuration. However, the calibration does not work to the error resulted from the misorientation and the depolarization effect. The calculation shows the error of sample thickness measurement is 4 nm, and the error is approximately linear with the orientation error and independent of the film parameters.