In this paper, the Zernike polynomials are used to describe deformed optical surfaces. Using the adjoint method, the sensitivities of Zernike polynomials to design variables can be derived. This procedure effectively overcomes the bottleneck of computational cost in sensitivity analysis when using the finite difference method. Therefore, the topology optimization models, which usually have thousand or tens of thousands of design variables, can be implemented by using the objectives and design constraints directly based on Zernike coefficients. Meanwhile, within the frame of numerical finite element discretization, adaptive finite element basis functions and element numerical integrals can be implemented to solve structural deformation and Zernike coefficients accurately and efficiently. This algorithm is flexible to be applied to general structural topology optimization models with objectives or constraints being a reasonable linear combination of Zernike coefficients. The numerical examples illustrate that the algorithm can optimize Zernike coefficients effectively.