To reduce the nonlinearity of nanometer measurement in laser heterodyne interferometric, the influence mechanics of nonideal splitting performance of polarization beam splitter (PBS) upon nonlinearity must be confirmed. In the paper, nonlinearity model is proposed when polarization transmissivity and reflectivity are not ideal. Model is also proposed to describe how the nonideal splitting performance of PBS influences the second-harmonic nonlinearity. The simulation results show that influence of nonideal polarization transmissivity and reflectivity on nonlinearity is the first-harmonic and the influence will be serious while the splitting performance of PBS gets worse. When reflectivity is 0.90 and transmissivity of PBS reduces from 1 to 0.90, nonlinearity increases from 0.62 nm to 1.24 nm. When the rotation angle error of PBS exists, the second-harmonic nonlinearity caused by the nonideal splitting performance of PBS will be ineffective, but the first-harmonic nonlinearity will increase and the nonlinearity is influenced greatly. It is assumed that the rotation angle error of PBS is 5° and transmissivity of PBS reduces from 1 to 0.90, nonlinearity increases from 0.39 nm to 1.41 nm.