Polarization-sensitive full-field optical coherence tomography (PS-FFOCT) system can produce the depth-resolved en face images of the birefringence and diattenuation of biological tissue samples. One of the important problems in the birefringence measurement of biological tissue using PS-FFOCT system is the analysis of the effects of relative phase delay error of quarter-wave plate on the measurement accuracy. A PS-FFOCT system was set up in this study. First, the Jones matrix model of the system was derived, and the effects of relative phase delay errors of quarter-wave plate on the measurement accuracy of birefringence were then analyzed. Finally, the results simulated using Matlab software were presented to reveal the specific behaviors. The research results can help the design of PS-FFOCT system and the accurate measurement of birefringence characteristics of biological tissues.