A theoretical model of passively mode-locked fiber laser was constructed, and the influences of high-order dispersion on output pulse characteristics were studied by numerical simulation in both time domain and frequency domain. In terms of time domain, the third-order dispersion resulted in distortion of pulse shape, asymmetry of pulse time-domain profile and temporal shift. In terms of frequency domain, the third-order dispersion caused changes in position and intensity of sidebands. At the same time, it was found that the fourth-order dispersion could broaden the pulse and change sidebands intensity of the pulse. The results are instructive for the practical applications of the passively mode-locked ytterbium-doped fiber lasers.