As an essential resolution enhancement technique, source optimization can improve the quality of advanced lithography. In the field of advanced lithography, the convergence efficiency and optimization ability of the source optimization are very important. Particle swarm optimization (PSO) is a global optimization algorithm. The adaptive control strategy can improve the global search ability of particles, and the nonlinear control strategy can expand the search range of particles. In this paper, a PSO algorithm based on adaptive nonlinear control strategy (ANCS) is proposed to solve the problem of source optimization by transforming it into a multivariable evaluation function. The image optimization simulation is carried out with a brief periodic grating image and an irregular image, and the source shape is optimized by the global iteration property of the proposed method. By using the pattern errors (PEs) as a multivariate merit function, the results of 300 iterations are evaluated, and the PEs of the two kinds of simulation patterns are reduced by 52.2% and 35%, respectively. Compared with the traditional PSO algorithm and genetic algorithm, the proposed method not only improves the imaging quality, but also has higher convergence efficiency.