In practical engineering problems such as the optimal design of filter antennas, the optimization of large-scale variables is usually involved, and there are complex associations between variables, which will lead to slow convergence speed, low solution accuracy, and poor population diversity. To solve this problem, an improved non-dominant sorting gray wolf optimization algorithm is proposed. In order to achieve effective dimensionality reduction and decoupling of large-scale variables, a multi-objective differential grouping method is proposed, which is applied to the original non-dominant sorting gray wolf optimization algorithm, which divides the search space into multiple subspaces and finds the optimal solution in each subspace. In addition, in order to improve the uniformity of the distribution of the initial population in the search space, the Tent chaotic map and K-means clustering algorithm are used for population initialization. In order to improve the exploration ability of the algorithm, the golden sine strategy is used to update the position of the gray wolf, and an adaptive nonlinear control parameter is proposed. The performance of the algorithm is verified on the ZDT test set, and the proposed algorithm is better than the comparison algorithm in both inverse generative distance (IGD) and hypervolume (HV). The proposed algorithm is applied to the optimization design of the filter antenna, and the experimental results show that the optimization solution of the filter antenna can be found quickly, and compared with the original NSGWO algorithm, the average value of the S11 parameters of the optimized filter antenna is reduced by 36.4%, and the average gain is increased by 41.4%; Compared with the NSWOA* algorithm, the average value of the S11 parameters is reduced by 24.0%, and the average gain is increased by 33.5%.