It is important to obtain the dielectric constant of materials in terahertz band for applications of 6G communication, radar remote sensing, detection and imaging. In this paper, the complex permittivity of samples is extracted based on the Newton Raphson(NR) iterative method, of which the influence of the initial value is analyzed. A set of 8f quasi optical system consisting of Vector Network Analyzer(VNA), spread spectrum module and four parabolic mirrors is built in 325~500 GHz(Yband) to realize the free space measurement of scattering parameter S 21. The relationship between complex permittivity and S 21 is derived from the electromagnetic wave transmission model. The complex permittivity spectra of Teflon, ABS and PMMA samples are extracted by iterative method. The results are consistent with other literatures, which verify the effectiveness of the proposed system and the method.