A particle swarm optimization algorithm (PDW-PSO), of which the inertia weight is modulated by particle position, is proposed for step-by-step optimization. The diffraction efficiency is calculated using rigorous coupled wave analysis (RCWA), and structural parameters of gratings are optimized. The comparison among PDW-PSO, traditional particle swarm optimization of which the inertial weight is unchanged (PSO), and particle swarm optimization of which the inertia weight is iteration-determined (IDW-PSO) shows that PDW-PSO has a faster convergence rate. Compared with PSO and IDW-PSO, the average number of iterations of PDW-PSO decreases from 89.83 and 74 to 21.2, and the number of calling RCWA drops from 3144.05 and 2590 to 224. The influence of wavelength matching number on the algorithm is analyzed. The magnification of RCWA calling numbers of PSO and IDW-PSO is equal to that of wavelength fitting number, while the magnification of RCWA calling numbers of PDW-PSO is less than that of wavelength fitting number. Experiments on algorithm accuracy are carried out. In 30 runs, PDW-PSO, PSO, and IDW-PSO have similar times of correct convergence to the optimal value, and the error is less than 6.6%. With the increasing particle number, the accuracy of the three methods improves, and the algorithm can be guaranteed to converge to the right optimal value after the particle number increasing to 27.