The composite second order distortion (CSO) induced by self-phase modulation (SPM) in a vestigial sideband amplitude modulation (AM-VSB) externally modulated CATV system employing optical fiber Raman amplifier is studied theoretically. Firstly, the SPM-induced CSO expression is obtained analytically by solving Schrdinger equation with the help of perturbation method. Subsequently the corresponding numerical results are presented and discussed. According to the calculation results, it is shown that, compared with using EDFA, the CSO performance can be improved by more than 10 dB by using distributed Raman amplification in the system.