A new algorithm of error vector magnitude (EVM) for quadrature phase shift keying (QPSK) signals impaired by narrow filtering is proposed theoretically in this paper. The new algorithm can remove the influence of intersymbol interference (ISI) induced by filtering, so the calculated EVM can be utilized for accurate estimation on optical signal to noise ratio and bit error rate. The numerical results also confirmed that the proposed scheme has better estimation accuracy on optical signal to noise ratio, and the modified algorithm taking into account the ISI-induced distortion can improve the estimation accuracy of bit error rate. Meanwhile, the interference-related symbol number is discussed for trade-off between computation efficiency and estimation performance, and the estimation performance is analyzed under different system conditions, including optical signal to noise and filter bandwidth.