We study the impact of stimulated Raman scattering (SRS) and polarization effect on the gain of fiber optical parametric amplifier. We find that the gain spectrum is asymmetrical because of the Raman scattering, contemporaneously, the gain peak value in SRS case is larger than that in no SRS case. When the difference of two pumps is increased, the gain peak value decreases and the gain flatness is worse. When the Raman response delay time decreases is, the gain peak value increases. The greater birefringence coefficient is, the smaller gain peak value is. Moreover, the polarization effect is considered, the signal gain and the signal power are dependent on the signal polarized angle, namely, the signal gain and the signal power reach maximum if the signal and the pumps are horizontally polarized, but they reach minimum if the signal and the pumps are orthogonally polarized.