Aiming at the effect of surface scattering on the imaging performance for astronomical telescope optical system, we propose a method of calculating the encircled energy in image plane based on the power spectrum density (PSD) of the mirror surface. The relationship between the intensity distribution of the image surface and the power spectral density of the mirror surface is proposed based on Harvey-Shack scattering model. The propagation of the scattering in the off-axis three mirror optical system on each reflecting surface is described. The relationship between irradiance distribution and the ratio of the effective root-mean-square (RMS) to wavelength is discussed. The surface power spectral density in different spatial frequency bands of 1.5 m-diameter optical surface is obtained by different measurements, and the full-band one-dimensional and two-dimensional power spectral density are fitted by k-correlation model. It shows that the effective RMS of surface shape on the processed sample is 13.7 nm. Then the encircled energy distribution of the image plane at different wavelengths is compared. The processing requirements of a large space astronomical telescope in our country including scattering conditions are given: the effective RMS of mirror surface must be less than 10.3 nm, and the RMS of low frequency error needs to be less than 8 nm.